RE: iSCSI Reqts: In-Order Delivery

[Charles Monia <cmonia@NishanSystems.com>]

Hi Santosh:

Please see below.

> -----Original Message-----
> From: Santosh Rao [mailto:santoshr@cup.hp.com]
> Sent: Friday, April 20, 2001 11:46 AM
> To: Charles Monia
> Cc: ips@ece.cmu.edu
> Subject: Re: iSCSI Reqts: In-Order Delivery
> 
> 
> Charles Monia wrote:
> 
> > > (1) MUST provide ordered delivery of SCSI commands from
> > >       the initiator to the target in the absence of transport
> > >       errors visible to iSCSI (e.g., iSCSI CRC failure,
> > >       unexpected TCP connection closure).
> > 
> > Does the term "SCSI commands" include task management 
> functions as well?  If
> > not, it should.
> 
> 
> Charles,
> 
> Could iSCSI use a variant of the approach FCP-2 takes to solve the
> ordering issue for task mgmt error recovery ?
> 
> The FCP-2 task management error recovery scheme is :
> - task mgmt function uses CRN 0
> - task mgmt function is executed immediately with no ordering 
> latencies
> - both initiator & target clear all resources that can be cleared
> un-ambiguously.
> - any ambiguous exchanges shall be aborted by the port that 
> detects the
> ambiguous state.
> 
> In the case of iSCSI, an analogous approach could be :
> - task mgmt function uses immediate delivery flag for the 
> task mgmt PDU.
> - task mgmt fn executed immediately avoiding any ordering latencies.
> - initiator & target clear all resources that can be cleared
> un-ambiguously.
> - initiator uses Abort Task to explicitly abort all active outstanding
> I/Os at the time the task mgmt fn was issued to avoid any ambiguous
> stale PDUs of an exchange from appearing at the target. 
> 
> Such an approach would avoid latencies on the execution of 
> the task mgmt
> fn while still flushing out all the stale PDUs upon completion of the
> initiator actions for that task mgmt fn.
> 

The problem is to avoid scenarios where the initiator and target's view of
the task set are out of step.  Specifically, we must avoid the case where an
initiator receives a PDU from a task it believes has been terminated. 

In that respect, the technique you describe above should work for an ABORT
TASK operation.

In the case of ABORT TASK SET, the function could be emulated by issuing a
series of ABORT TASK requests. For CLEAR TASK SET, an initiator would
probably want to do the individual ABORT TASK operations, followed by a
CLEAR TASK SET to terminate tasks from other initiators.  I assume TARGET
RESET and LUN RESET would be emulated in a manner similar to CLEAR TASK SET.
In all of these cases there may be some "atomicity" side effects caused by
doing things one at a time instead of all at once.

The only sticky problem is insuring that the CLEAR ACA function works right.
By that I mean that you don't want to issue the function until all prior
SCSI commands that were in flight when the ACA occurred have been terminated
with the ACA ACTIVE status.  You can't simply replicate the command on each
connection since you might inadvertently clear a subsequent ACA. (Yes -- I
know these are all edge cases, but we may as well try to get it right.)
Maybe the thing to do is implement the function such that the ACA interlock
is not cleared until the CLEAR ACA function is sent on all the connections
comprising the session.

One minor distinction worth noting is that CRN is enforced in the SCSI
layer, whereas cmdSN is enforced in the iSCSI transport.  So, a CRN of 0
doesn't take effect until the transport presents the command to the SCSI
layer for processing.  In that case, leapfrogging of PDU ordering never
occurs.

Incidentally, I've made the tacit assumption that commands on a given
connection are presented to the SCSI layer in order they were sent,
regardless of whether or nor cmdSN was set to 0.  I assume the framing
mechanisms that have been discussed for buffer offloading do not affect this
behavior.  I.e., a fully formed PDU slated for immediate delivery won't be
passed to the SCSI layer before a partially complete PDU that was received
earlier.

If that's true, immediate delivery seems to have no meaning in a
single-connection scenario.  What's more, in all cases, the iSCSI layer
doesn't really have to be aware of task management semantics -- unless
someone decides to intermix immediate and sequential commands in a
multi-connection session.  Then all bets are off.

Charles