Re: iSCSI: Out Of Sequence due to null sequence with multiple con nections.

[Sandeep Joshi <sandeepj@research.bell-labs.com>]

Doug,

thanks.  If you (or anyone) could correct the psuedo-code below to 
illustrate your solution, it might help achieve quicker consensus 
and avoid some discussion.

I see what I missed, in addition to Julian's point about the 
refTaskTag usage preventing ITT reuse.  But dont you still need 
the cmdSN of the original task to find out if task_mgmt command 
is early or late?
 (a..assuming you are still sending the task_mgmt 
command with immediate delivery)

**Event=task_mgmt at initiator:
    purge PDUs in queue at initiator
    send task_mgmt to target (cmdSN=0)

**Event=task_mgmt at target:
    compare refCmdSN with executing <min,max>CmdSN queue
    if (refCmdSN < minCmdSN)
        /*task_mgmt cmd is early */
        must wait & drop the orig_task PDU when it arrives 
    else if (refCmdSN > maxCmdSN)
        /*task_mgmt cmd is late, original task has completed at target*/
        return task_response (response code=Task was not in task set)
    else
        /*task is executing*/
        give task_mgmt command to SCSI layer


-Sandeep 


Douglas Otis wrote:
> 
> David,
> 
> Sandeep missed a point found within serial math, you have a window that
> rotates with respect to prior commands based on the magnitude of the
> difference.  There is no need to maintain any state other than the sequence
> of the flagged command where prior pending to be sent commands are rejected.
> Obviously before this window rotates more than 2 billion PDUs, this prior
> value will need to be retired.  This is not a difficult or high overhead
> operation with respect to rejecting prior commands.  There would not be any
> decisions within the sequencer regarding content of any rejected PDU.  You
> still should want to purge PDUs waiting in a queue pending to be sent to the
> target should an "immediate" command be flagged.  Your concept creates an
> odd event with both sequential and non-sequential delivery of a task
> management command.  You are then left with a time interval where a
> non-sequential command reception must modify behavior waiting for a possible
> counter-part.  Causing all pending PDUs to be rejected immediately there is
> no waiting for status information or any further activity to occur.  You
> would see reject-reject-status.  If the initiator needs these rejected
> commands replayed, this becomes an option of the initiator.
> 
> Doug
> 
> > > I would state this much stronger.  Applications had better not have to
> > know
> > > that it is iSCSI underneath vs. FCP or parallel SCSI else I believe we
> > > missed the objective (granted, some things such as target address space
> > are
> > > unavoidably different, but I believe task management functions should be
> > the
> > > same).  The transport needs to handle the transport issues without
> > exposing
> > > quirks to the SCSI or application layer.
> >
> > Unfortunately, I think we have an impossible situation.  It appears to me
> > that
> > we have to pick at most two of the following three goals, as I have yet to
> > see
> > any way to achieve all three for a single task management command on a
> > multiple connection session:
> >
> > (1) The command takes effect immediately and its status/response
> >       is available immediately.
> > (2) The command affects all commands in flight, and its status/response
> >       is delayed until all such effects are complete.
> > (3) There is no significant visible departure from existing SCSI task
> >       management behavior.
> >
> > The problem is that trying to do both (1) and (2) either requires SCSI to
> > "execute" the task management command twice or requires that iSCSI do
> > some task management (e.g., on the in-flight commands) on SCSI's behalf
> > (or worse like having SCSI prolong the execution of the task management
> > command until everything in flight in iSCSI arrives).  All of these appear
> > to lead to problems with (3) in one form or another - two executions
> > result in two SCSI status/responses that have to be merged, and iSCSI
> > task management will sooner or later do something different from SCSI
> > (e.g., I sincerely doubt that a Target in a bridge will ever get this 100%
> > identical to the devices that are being bridged).
> >
> > The current iSCSI draft provides the choice of  [(1)] XOR [(2), (3)];
> > the reason for not getting (3) with (1) is the possibility of the task
> > management command bypassing commands that it's supposed to
> > affect.  Charles' original proposal is [(2), (3)] because it has
> > to time out
> > a stuck connection before executing the command, and is roughly
> > equivalent to sending the command for ordered delivery and having
> > the implementation treat any queue between iSCSI and SCSI as
> > being on the SCSI side of the line.  Doug Otis's counter-proposal
> > falls into the category of iSCSI doing task management on SCSI's
> > behalf and provides an example of how this results in visible changes
> > in behavior -- for the CLEAR ACA task management command,
> > aborting all tasks that are queued or in flight is generally incorrect.
> >
> > I would note that this issue does not arise on single connection sessions,
> > because sending the command for immediate delivery plus some care not
> > to reorder things in the iSCSI Target (i.e., consider the iSCSI to SCSI
> > queue
> > to be in "SCSI" and hence subject to the task management command)
> > obtains all of (1) through (3).
> >
> > Going out on a limb, I suspect applications will generally want [(2), (3)]
> > -- send for ordered delivery and wait for the dust to settle because that
> > provides the best odds of having some weird device get into a known
> > state from which further progress is possible.  This allows the
> > application
> > to not know whether parallel SCSI, FCP or iSCSI is underneath and
> > relies on other iSCSI recovery procedures to make sure that the task
> > management command is delivered and executed (e.g., unstick and/or
> > close "stuck" connections).  There will be cases in which (1) is
> > needed (e.g., observe tape robot doing something obviously wrong,
> > and get it to stop immediately), but those may involve fairly blunt
> > instruments (e.g., LUN RESET) and the need to clean up any collateral
> > damage.
> >
> > Sandeep's proposal to create state in the target either fails to achieve
> > (1) [if the response is delayed until the state is removed] or
> > violates SAM2
> > [returns the response to the task management command before the task
> > management command is complete].  Having state linger after a completed
> > LUN or TARGET RESET is almost certainly wrong.
> >
> > So, I think I'm down to sending task management functions once, usually
> > for ordered delivery with the application making the ordered vs. immediate
> > delivery choice (and sending the task management function twice if it
> > so chooses).  I think apps will generally choose ordered
> > delivery, choosing
> > predictable behavior over immediacy concerns.  Aside from a longer
> > discussion of this issue, I still don't see the need for additional
> > mechanism(s) to task management - what have I missed in the above
> > discussion?
> >
> > --David
> >
> > ---------------------------------------------------
> > David L. Black, Senior Technologist
> > EMC Corporation, 42 South St., Hopkinton, MA  01748
> > +1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
> > black_david@emc.com       Mobile: +1 (978) 394-7754
> > ---------------------------------------------------
> >
> >