RE: iSCSI: Out Of Sequence due to null sequence with multiple connections.

["Douglas Otis" <dotis@sanlight.net>]

Robert,

The modification of using a flag to indicate "Exigent" rather than use of a
null sequence is a means to handle the event of a PDU bypassing many queued
commands within the iSCSI delivery system.  This queue at the iSCSI
sequencer is not the same as described in SCSI for a target device.  I do
not know of any corollary behavior within existing SCSI delivery schemes
where potentially a substantial number of commands are held beyond the
control of the initiator and the target.  In addition, due to the null
serialization, this command may be applied after subsequent commands have
been sent due to the unpredictable nature of IP networks with false
assumptions made as to the assured expediency of these networks.  Yes, place
in enough time-outs and this problem becomes a bit more predictable in some
cases.  Suggesting only a single null sequence PDU be used at any point in
time does not resolve principle issues of dealing with the sequencer queue
nor is this command acknowledged which is sure to lead to this rule being
violated upon a retry placing additional wild cards into the systems.

Although SAM may allow some flexibility in how management is handled, the
situation at the iSCSI sequencer goes beyond the intent of this flexibility
as I do not think they envisioned this trapped queue of commands.  Although
additional status interlocks within the target is welcome, those commands
held in the iSCSI sequencer will not be handled according to SCSI rules
unless a new abort behavior is in place at the SCSI target and the SCSI
management command signals this behavior.  The initiator should be given the
option of how these bypassed commands are to be handled.  Rejecting those
commands within the sequencer at the time of the bypass or "Exigent" command
allows the initiator this option without adding additional behavior models
to the target.

This flag solves three problems, one) late arrivals of a management command,
two) predictable handling of commands without SCSI target behavior
modifications, three) acknowledgement of these critical management commands.
It seems the means of handling this event as described requires less effort
than as now envisioned and relies less on SCSI trapping iSCSI errors.

Doug

> This apparently did not get out on the ips reflector.  Trying again.
>
> -----Original Message-----
> From: Robert Snively
> Sent: Tuesday, April 03, 2001 3:11 PM
> To: 'Black_David@emc.com'; CBinford@pirus.com; ips@ece.cmu.edu
> Subject: RE: iSCSI: Out Of Sequence due to null sequence with multiple
> connections.
>
>
> I support David's conclusions that task management should be sent
> once.  Fortunately, the ordered/unordered question need not even
> be asked, because SAM-2 compliant SCSI applications already know
> what will happen.
>
> At present, SCSI applications do not have a clear guarantee of
> the order between task management functions and the processing or
> delivery of any particular task.  In fact, the concept of
> an ORDERED attribute applied to a task management function is
> unknown.  As a result, SCSI drivers have to be aware of
> the implications.  Those implications include the possibility that
> a command may be in any state of delivery or completion as a
> task management function is executed and therefore may or may not
> be included in the scope of the task management function.  This
> has been partially clarified by the recently created status value
> developed by Charles Binford, "TASK ABORTED", which at least indicates
> which tasks for other initiators have been cleared by a task
> management function.
>
> In general, the approach is as follows:
>
> 	When a task management operation is received by a target,
> 	tasks are treated in one of three ways, depending on where
> 	they are with respect to the timing of the execution of
> 	the command.
>
>    a)	If the command has been completed, it is perfectly possible that
> 	completion status for the command is transmitted in spite of the
> 	fact that the task management operation has been received.
>
>    b)	If the command is in progress or enqueued, it is likely that
> 	the command will be managed according to the rules of the
> 	task management command unless a) or c) happen.
>
>    c) If the command has not been received by the time the task
> 	management function is completed, the command will be received
> 	and treated as a command that has occurred after the task
> 	management function, even if it were sent before that.  That
> 	typically involves presentation of a UNIT ATTENTION condition
> 	or some other notification.
>
> 	When a task management operation passes through an initiator,
> 	the initiator has the option of acting on the commands
> 	whose state it knows, including the possibility of discarding
> 	or presenting both expected and unexpected returned status,
> 	in a vendor specific manner.
>
> It is possible that all this work of creating a synchronous behavior
> for actions that are designed to be asynchronous may be unnecessary.
> The present drivers know what to do to clean up the mess.  All you
> have done is statistically increase the number of tasks that may
> be involved in behaviors a) and c) above.
>
> References in support of this include:
>
> >From SAM-2, Rev 15, 4.6.2:
>
> 	For convenience, the SCSI architecture model assumes
> 	in-order delivery to be a property of the service delivery
> 	subsystem. This assumption is made to simplify the
> 	description of behavior and does not constitute a requirement.
> 	In addition, this specification makes no assumption about,
> 	or places any requirement on the ordering of requests or
> 	responses between one sending device and several receiving devices.
>
> >From SAM-2, Rev 15, 4.7.4
>
> 	The order in which task management requests are
> 	executed is not specified by this standard. In particular, this
> 	standard does not require in-order delivery of such
> 	requests, as defined in 4.6.2, or execution by the task manager
> 	in the order received. To guarantee the execution order
> 	of task management requests referencing a specific logical
> 	unit, an initiator should, therefore, not have more than
> 	one such request pending to that logical unit.
>
> >  -----Original Message-----
> >  From: Black_David@emc.com [mailto:Black_David@emc.com]
> >  Sent: Tuesday, April 03, 2001 12:47 PM
> >  To: CBinford@pirus.com; ips@ece.cmu.edu
> >  Subject: RE: iSCSI: Out Of Sequence due to null sequence
> >  with multiple
> >  con nections.
> >
> >
> >  > I would state this much stronger.  Applications had better
> >  not have to
> >  know
> >  > that it is iSCSI underneath vs. FCP or parallel SCSI else
> >  I believe we
> >  > missed the objective (granted, some things such as target
> >  address space
> >  are
> >  > unavoidably different, but I believe task management
> >  functions should be
> >  the
> >  > same).  The transport needs to handle the transport issues without
> >  exposing
> >  > quirks to the SCSI or application layer.
> >
> >  Unfortunately, I think we have an impossible situation.  It
> >  appears to me
> >  that
> >  we have to pick at most two of the following three goals, as
> >  I have yet to
> >  see
> >  any way to achieve all three for a single task management
> >  command on a
> >  multiple connection session:
> >
> >  (1) The command takes effect immediately and its status/response
> >  	is available immediately.
> >  (2) The command affects all commands in flight, and its
> >  status/response
> >  	is delayed until all such effects are complete.
> >  (3) There is no significant visible departure from existing SCSI task
> >  	management behavior.
> >
> >  The problem is that trying to do both (1) and (2) either
> >  requires SCSI to
> >  "execute" the task management command twice or requires that iSCSI do
> >  some task management (e.g., on the in-flight commands) on
> >  SCSI's behalf
> >  (or worse like having SCSI prolong the execution of the task
> >  management
> >  command until everything in flight in iSCSI arrives).  All
> >  of these appear
> >  to lead to problems with (3) in one form or another - two executions
> >  result in two SCSI status/responses that have to be merged, and iSCSI
> >  task management will sooner or later do something different from SCSI
> >  (e.g., I sincerely doubt that a Target in a bridge will ever
> >  get this 100%
> >  identical to the devices that are being bridged).
> >
> >  The current iSCSI draft provides the choice of  [(1)] XOR [(2), (3)];
> >  the reason for not getting (3) with (1) is the possibility
> >  of the task
> >  management command bypassing commands that it's supposed to
> >  affect.  Charles' original proposal is [(2), (3)] because it
> >  has to time out
> >  a stuck connection before executing the command, and is roughly
> >  equivalent to sending the command for ordered delivery and having
> >  the implementation treat any queue between iSCSI and SCSI as
> >  being on the SCSI side of the line.  Doug Otis's counter-proposal
> >  falls into the category of iSCSI doing task management on SCSI's
> >  behalf and provides an example of how this results in visible changes
> >  in behavior -- for the CLEAR ACA task management command,
> >  aborting all tasks that are queued or in flight is generally
> >  incorrect.
> >
> >  I would note that this issue does not arise on single
> >  connection sessions,
> >  because sending the command for immediate delivery plus some care not
> >  to reorder things in the iSCSI Target (i.e., consider the
> >  iSCSI to SCSI
> >  queue
> >  to be in "SCSI" and hence subject to the task management command)
> >  obtains all of (1) through (3).
> >
> >  Going out on a limb, I suspect applications will generally
> >  want [(2), (3)]
> >  -- send for ordered delivery and wait for the dust to settle
> >  because that
> >  provides the best odds of having some weird device get into a known
> >  state from which further progress is possible.  This allows
> >  the application
> >  to not know whether parallel SCSI, FCP or iSCSI is underneath and
> >  relies on other iSCSI recovery procedures to make sure that the task
> >  management command is delivered and executed (e.g., unstick and/or
> >  close "stuck" connections).  There will be cases in which (1) is
> >  needed (e.g., observe tape robot doing something obviously wrong,
> >  and get it to stop immediately), but those may involve fairly blunt
> >  instruments (e.g., LUN RESET) and the need to clean up any collateral
> >  damage.
> >
> >  Sandeep's proposal to create state in the target either
> >  fails to achieve
> >  (1) [if the response is delayed until the state is removed]
> >  or violates SAM2
> >  [returns the response to the task management command before the task
> >  management command is complete].  Having state linger after
> >  a completed
> >  LUN or TARGET RESET is almost certainly wrong.
> >
> >  So, I think I'm down to sending task management functions
> >  once, usually
> >  for ordered delivery with the application making the ordered
> >  vs. immediate
> >  delivery choice (and sending the task management function twice if it
> >  so chooses).  I think apps will generally choose ordered
> >  delivery, choosing
> >  predictable behavior over immediacy concerns.  Aside from a longer
> >  discussion of this issue, I still don't see the need for additional
> >  mechanism(s) to task management - what have I missed in the above
> >  discussion?
> >
> >  --David
> >
> >  ---------------------------------------------------
> >  David L. Black, Senior Technologist
> >  EMC Corporation, 42 South St., Hopkinton, MA  01748
> >  +1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
> >  black_david@emc.com       Mobile: +1 (978) 394-7754
> >  ---------------------------------------------------
> >
> >
>