RE: iSCSI:flow control, acknowledgement, and a deterministic recovery

["Douglas Otis" <dotis@sanlight.net>]

David,

A limitation in the number of tasks for each or all targets would be a
control method or as a type of Xon/Xoff as with SEP, but not an iSCSI one as
it is now.  Santosh said all commands could use a null CmdSN in his first
statement.  Perhaps iSCSI should explicitly exclude this use.  This does
imply there is no acknowledgements, no flow control, and no sequential
delivery within iSCSI.

I pointed out in the second paragraph there is still a need to ensure timely
delivery or NO delivery.  That is not a property of IP.  With multiple
connections, if you are not going to use a valid CmdSN, or in his case a
null CmdSN for all commands, then there would be a requirement to include a
timestamp to meet a timely delivery requirement in the same manner as used
in the FC encapsulation.  He had made it clear he was not concerned about
sequential delivery and perhaps I made the rather absurd but valid point as
a result.

You have now offered the possibility of different flow control schemes.  I
started this effort to offer a solution for the existing scheme.  The flora
gets thick in this area it would appear.

Doug


> Doug,
>
> There are a couple of confusions in here:
>
> TASK_SET_FULL *is* a flow control mechanism, albeit a
> rather primitive one - the Initiator issues commands until
> it gets a FULL response and hence knows the capacity
> of the Target's queue.  The statement that the Target loses
> resource control if it only uses TASK_SET_FULL is wrong,
> although it is relying on code at or above the SCSI driver
> on the Initiator to handle the flow control.
>
> Santosh did not propose to remove CmdSN, although he
> did point out an issue that needs attention - we need
> to say something about the interaction of CmdSN with
> TASK_SET_FULL since both mechanisms may be implemented.
> In particular, it's possible to use CmdSN for delivery
> ACKs and TASK_SET_FULL for flow control, although that's
> a "peculiar" combination (and perhaps a SHOULD NOT?).
>
> Beyond this, your entire second paragraph is well off
> into the weeds.  Please be careful about taking the
> discussion off onto wild tangents.
>
> Thanks,
> --David
> ---------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 42 South St., Hopkinton, MA  01748
> +1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
> black_david@emc.com       Mobile: +1 (978) 394-7754
> ---------------------------------------------------
>
> From:	Douglas Otis [dotis@sanlight.net]
> Sent:	Tuesday, April 10, 2001 2:09 AM
> To:	santoshr@cup.hp.com
> Cc:	Ips
> Subject:	RE: iSCSI:flow control, acknowledgement, and a deterministic
> recovery
>
> Santosh,
>
> In a change recommending a flag and a valid CmdSN and not a null value, I
> included a flag named "Casual" to provide this mode of operation you
> describe which still allows flow control.  I would not wish to permit this
> Casual mode, but as David feels such lists or threads are easy to manage,
> this should not represent any greater effort for feedback.  But
> then again,
> if you are not using flow control, you are also not using any
> acknowledgement other than an eventual SCSI status.  If there is no flow
> control, this removes resource control and reception acknowledgement.  If
> this is not needed, then such flow control should be removed for
> simplicity
> as there then must always be an abundance by design of adequate recourses.
> For cases where command sequence is important, CmdSN together with
> acknowledgement must remain or only one command at a time can be
> issued.  I
> can envision three modes, a Sequential mode, a Casual mode, and an Exigent
> mode where the command is given head of sequence status (with
> rejection for
> feedback simplicity ;).
>
> You seem to be placing many things on the table for reconsideration.  Flow
> control, acknowledgement, sequential delivery, and perhaps even a means of
> moving a command up the queue feeding SCSI targets.  One command at a time
> for sequential assurance is not impossible.  Is flow control
> needed?  Do you
> care if there is any reception acknowledgement if all feedback is in the
> SCSI layer?  It would seem in this barebones case, iSCSI adds little to
> ensure integrity.  The FC encapsulation places a time stamp
> within the PDU.
> Perhaps a time stamp should be employed as an alternative to sequence
> control?  At least after a timeout, you are assured a missing command is
> forever gone.  Otherwise, the check was to see that CmdSN is not too small
> to ensure it is not a stray.
>
> Doug
>
>
> > Doug [& All],
> >
> > Some comments on this thread :
> >
> > 1) The immediate command feature can be exploited by initiators which
> > are not required to provide strict command ordering for their SCSI
> > sub-system. (The majority of today's scsi stacks). IOW, all commands can
> > be sent with CmdSN=0 to indicate no ordering required.
> >
> > Any scheme to restrict the number of "immediate commands" that can be
> > sent prohibits such a feature and is un-desirable.
> >
> > 2) The (ExpCmdSN, MaxCmdSN) based acknowledgement and flow control is a
> > freebie that targets can use to implement additional flow control.
> > There's nothing in the spec that mandates that a target MUST use this
> > feature to throttle its window and apply flow control.
> >
> > Hence, any dependence and assumptions on the existence of CmdSN based
> > flow control may be incorrect, since the target may be using QUEUE FULL
> > (TASK SET FULL) to apply flow control.
> >
> > > I also suspect it is a mistake to allow commands
> > > to remain trapped within the sequencer during emergency or
> > abnormal events
> > > signified by the use of these immediate commands.
> >
> > 3) If the initiator suspects that the CmdSN queue at the target is
> > stuck, it can always use a task mgmt command or even an iSCSI login with
> > the X (restart) bit to perform a cleanup of stuck commands at the
> > target. There's no need to build in implicit assumptions that a CmdSN
> > with the immediate flag result in a clean-up of previously pending
> > commands at the CmdSN queue of the target.
> >
> > - Santosh
> >
> >
> > Douglas Otis wrote:
> > >
> > > David,
> > >
> > > > > To not support rejects, you are then relying on the target to
> > > > > handle these now out of sequence commands.
> > > >
> > > > Immediate commands are "out of sequence" by definition - immediate
> > > > means deliver immediately without regard to CmdSN order.  Only the
> > > > target can do this, so I don't see any problem with relying
> > on the Target
> > > > to do so.
> > >
> > > The commands that are NOW out of sequence are those commands
> > bypassed by the
> > > immediate command.  If this immediate command was Rewind, then
> > a long list
> > > of write commands become invalid as a new tape is now needed for those
> > > operations.  You are expecting that the target to handle a
> long list of
> > > invalid commands made possible by the iSCSI sequencer.
> > Commands invalidated
> > > by immediate commands become out of sequence commands.
> > >
> > > I'll reserve further comment until Julian has made some progress in
> >
> > >
> > > Forgive my advocacy, but if one does not attempt to support a
> > position, then
> > > the subject is not explored.
> > >
> > > Doug
>
>
>