RE: iSCSI Reqts: In-Order Delivery

[Charles Monia <cmonia@NishanSystems.com>]

Hi:

The point of my original posting was to suggest ways in which the semantics
of all the task management functions could be preserved in multi-connection,
command striping implementations without a lot of complicated bookeeping.

In that regard, the proposed solution imposes no additional tracking
equirements on initiators aside from those that would be needed anyhow to
issue the ABORT TASK request. For the most part, that amounts to keeping
track of each pending I/O request including a handle by which the task can
be referenced and a pointer to the connection the SCSI command was issued
on.

I did neglect one restiction however: Specifically, that the initiator have
no more than one task management request pending at a time to a specific
target. 

In other respects, as long as ordered delivery to the SCSI layer is
preserved for individual connections, I don't see a problem.

> ....Those commands executed out of 
> sequence by means of a bypass flag, those commands that are Task
Management .....

I apparently don't understand how the bypass flag is supposed to work.  I'd
assumed its function was to maximize the benefits of command striping by
allowing commands on other connections in the session to be bypassed. I'd
assumed that commands on the same connection are never bypassed (since there
appears to be no benefit in doing so).

Hence my statement:

> > .....I've made the tacit assumption that commands 
> on a given
> > connection are presented to the SCSI layer in order they were sent,
> > regardless of whether or nor cmdSN was set to 0.  I assume 
> the framing
> > mechanisms that have been discussed for buffer offloading do not
> > affect this
> > behavior.  I.e., a fully formed PDU slated for immediate 
> delivery won't be
> > passed to the SCSI layer before a partially complete PDU 
> that was received
> > earlier.

Is this assumption incorrect?

Charles

> -----Original Message-----
> From: Douglas Otis [mailto:dotis@sanlight.net]
> Sent: Monday, April 23, 2001 12:10 PM
> To: Charles Monia; Santosh Rao (E-mail)
> Cc: Ips (E-mail)
> Subject: RE: iSCSI Reqts: In-Order Delivery
> 
> 
> Charles,
> 
> Your solution requires a fair amount of tracking of commands 
> based solely on
> their Client Tags.  These Tags are randomly generated but will need to
> retain sequential order for your scheme.  The transport must 
> remember the
> type of command sent together with their relative placement 
> based only on
> the Client Tag.  In addition, these commands will need to be 
> placed into
> different categories.  Those commands executed out of 
> sequence by means of a
> bypass flag, those commands that are Task Management 
> commands, and commands
> affected by these other types of commands.  It seems that in 
> large part,
> these concerns can be met with proper handing of the 
> transport without such
> laborious sorting of the Client Tags.  The out-of-sequence or 
> bypass flag
> also depends on the transport sorting the Client Tag.  In addition to
> disabling flow-control, this technique of not incrementing 
> the serialization
> of these commands, requires all commands with the same 
> serialization value
> to be sent on the same connection without acknowledgment, if 
> these commands
> are also to be kept in sequence.  This connection requirement 
> is yet to be
> specified.
> 
> Ver 6, Pg 12:
>    "iSCSI may avoid delivering some command to the
>    SCSI layer if so required by some prior SCSI or iSCSI action (e.g.,
>    clear task set Task Management request received before all the
>    commands it was supposed to act on)."
> 
> Here, there seems to be expectations of the iSCSI transport 
> interpreting the
> content of the SCSI commands.  How this is done is not 
> obvious.  Is the
> transport expected to generate SCSI responses?
> 
> In addition, although iSCSI presently relies on ACA, there are few
> applications that implement ACA.  It would appear for iSCSI 
> to work with the
> present protocol, significant application changes are 
> required.  With the
> proposal I am suggesting, this is not a problem as all 
> bypassed commands are
> rejected back to the Initiator.  The drivers that implement 
> iSCSI will be
> required to provide handling for these commands that bypass 
> other commands.
> The amount of information contained in a rejected command 
> list should be
> relatively small and these occasions for such Management 
> rare.  Without
> proper handling of these events, there will be 2:00 AM alarm 
> pagers going
> off.
> 
> Here in the proposal, sorting CmdSN based on LUN values takes 
> place within a
> "Barrier List."  I can not tell what is implied by these recovery
> instructions.  What is meant by Remove, Release, Drop, 
> Cleanup, Placeholder,
> and ALL.  What is the intended feedback to the initiator for 
> this Clean-up?
> It would appear the transport works on behalf of the target.  In the
> proposal that I am suggesting, there is no actions within the 
> transport on
> behalf of the target.  All decisions are done either by the 
> Target or the
> Initiator.  None by the transport.
> 
> The concept is simple.  Keep the transport simple.  Do not expect the
> transport to decipher SCSI commands.  Do not expect the 
> transport to respond
> on behalf of the Target.  Do not expect the transport to sort pending
> commands based on LUN value.  Do not expect the transport to 
> require SCSI
> and iSCSI ACA.
> 
> In the case of session wide serialization, what is good for 
> the goose is
> also good for the gander.  It is important from the prospect 
> of quickly
> detecting an error and knowing the server state to also use 
> session wide
> serialization from the server.  The technique of replicating 
> Management
> commands down each connection in addition to changing global 
> commands into
> specific commands already over burdens the set-aside that 
> must be made to
> handle these non-serialized management commands.  My proposal 
> eliminates the
> problem of set-aside resources and loss of server state.  Rather than
> silently rejecting commands out-of-sequence, these rejections 
> are reported.
> Once done, this feature can be used to extract pending 
> commands in a simple
> and direct manner without burdening the transport.
> 
> As attempts are made to support the SCSI architecture, rather than
> increasing the intelligence of the transport, efforts should 
> be made to
> simplify the transport.  The number of fields that the transport must
> manipulate will be met with complexity and non-uniform implementation.
> 
> See:
> http://www.ietf.org/internet-drafts/draft-otis-iscsi-fullack-00.txt
> 
> Ver 6, Pg 92:
>      "N.B. As an alternative to Logout and reissue commands, the
>       initiator MAY instead reset the target and terminate all
>       outstanding commands with a service response indicating
>       Delivery Subsystem Failure. The initiator MUST perform one of
>       the two actions."
> 
> ...
> 

> Ver 6, Pg 93:
>    "The following general mechanism can be used to achieve 
> the effect of
>    ordered delivery for task management commands while enabling the
>    "urgent" delivery that some of them imply and immediate 
> execution of
>    the task management commands without:
> 
>       At Initiator when a relevant task management command is issued:
> 
>          a) if ExpCmdSN is equal to CmdSN skip to step c
>          b) mark all pending commands with a CmdSN field between
>          ExpCmdSN and the current CmdSN and a relevant LUN as
>          candidates for cleanup and retain CmdSN in a "barrier list".
>          c) send the task management command for immediate delivery
>          to the target
> 
>       At initiator when updating ExpCmdSN:
> 
>          a) if the "barrier list" is empty or ExpCmdSN is less than
>          the first entry in the barrier list then skip to step d
>          b) remove the barrier list entry and remove and drop all
>          entries marked for cleanup having a CmdSN field less than
>          ExpCmdSN
>          c) go to step a
>          d) release all queued entries between the old and new
>          ExpCmdSN from the queue
> 
>       At target when receiving a relevant task management command for
>       immediate delivery:
> 
>          a) if ExpCmdSN is equal to CmdSN skip to step c
>          b) mark all pending entries (commands received and
>          placeholders) with a CmdSN field between ExpCmdSN and the
>          current CmdSN as candidates for cleanup and retain CmdSN in
>          a "barrier list" including the referenced LUN (or an ALL
>          marker)
>          c) send the task management command to SCSI for immediate
>          execution
> 
>       At target when updating ExpCmdSN (releasing ordered commands to
>       SCSI):
> 
>          a) if the "barrier list" is empty or ExpCmdSN is less than
>          the first entry in the barrier list then skip to step d
>          b) remove the barrier list entry and remove and drop all
>          entries marked for cleanup and having the same LUN as the
>          barrier entry (any if the barrier is marked ALL) and a CmdSN
>          field less than ExpCmdSN
>          c) go to step a
>          d) release all queued entries between the old and new
>          ExpCmdSN from the queue
> 
>    Note that this scheme will withstand connection recovery."
> 
> Doug

< remainder deleleted>