RE: iSCSI Requirements Draft - Informal WG Last Call

["Douglas Otis" <dotis@sanlight.net>]

Santosh,

CRN does not resolve possibilities of trapped commands beyond the ULP
timeouts.  To resolve the trapped command problem, each command MUST pass
through the sequencer in order.  Stray commands are then identified by a
serialization beyond the current sequence point.  Any alternative scheme
would be complex.  This problem is not related to SCSI, it is an artifact of
IP networks with the potential for greatly delayed delivery.  The example
that I used was a technician temporarily unplugging the cable.

For security reasons, a command rejected for this reason should not be done
silently.  The iSCSI proposal should be changed to indicate that commands
serialized outside the current window will be reported.  The paragraph on
page 11 should be changed with an error code assigned to indicate the
direction of the error, before or beyond the command window.

Ver 5-92 Pg 11
  "The target MUST NOT transmit a MaxCmdSN that is more than 2**31 - 1
   above the last ExpCmdSN.  For non-immediate commands, the CmdSN field
   can take any value from ExpCmdSN to MaxCmdSN. For immediate commands,
   the CmdSN field can take any value from ExpCmdSN to MaxCmdSN+1. The
   target MUST silently ignore any command outside this range or
   duplicates within the range that have not been flagged with the retry
   bit (the X bit in the opcode)."

If iSCSI were just a single connection protocol, you would have some basis
for your complaint.  The topic still not addressed by the "immediate"
command designation is with respect to limits placed on the number of
successive "immediate" commands to ensure enough resources are held in
reserve.

Doug


> Julian,
>
> Ordered delivery can be achieved to better effect using the SAM-2 CRN
> based ordering due to the following reasons :
>
> 1) CRN provides ordering on a per-lun basis and can be turned on and off
> for a subset of I/Os to that LUN. This allows for flexible ordering
> since ordering is a function of the I/O type from the application.
> Applications that are doing READ only operations (like a search engine)
> do not require any ordering. Ordering is required on metadata updates,
> any form of synchronization I/Os, WRITEs interspersed with READS, etc.
> Thus, an ordering solution should be flexible enough to be applied at
> the scope of a subset of I/Os destined to a LUN.
>
> Such an ordering scheme would also allow ordering to be turned on for
> only tape applications if disk applications did not require ordering.
>
> iSCSI's ordering solution does not provide this flexibility, whereas
> usage of CRN would.
>
> 2) Such a fine granularity scope of ordering also minimizes the impact
> of error recovery actions taken when loss of order occurs. The impact
> with CRN would be a target-initiator handshake based on ACA + some
> checkpoints to error back all the pending CRN enabled commands on that
> LUN.
>
> Comparing the equivalent error recovery in a CmdSN based iSCSI ordering
> solution, the action taken would be to error back all the pending I/Os
> destined to the entire session following loss of order. With high end
> disk arrays having 1000+ LUN configurations, such error recovery is
> extreme, [especially when ordering may have been desired by the appln
> only on a small subset of I/Os to 1 LUN, and loss of ordering for the
> remaining 999 LUNs was a don't care].
>
> 3) A CRN based ordering scheme works for all underlying SCSI transports
> as opposed to CmdSN based ordering.
>
> 4) The generation of a stream of commands that expect strong ordering
> will need to be accompanied by corresponding generation of a sequence
> number at the same layer. (CRN would provide such a sequencing). Failure
> to do so can result in silent loss of order that slips un-detected due
> to potential points of failure in the stack b/n the SCSI ULP and the
> physical bus/link. (ex : I/O failures within the HBA driver due to
> resource allocation failures or other such conditions can cause loss of
> order.).
>
> Attempts to enforce ordering at multiple layers of the stack (CRN at the
> ULP and CmdSN at the LLP), especially when CmdSN does not provide all
> the benefits that CRN would provide is over-engineering the solution to
> the ordering problem. It also impacts iSCSI performance.
>
> - Santosh
>
> >
> > julian_satran@il.ibm.com wrote:
> > >
> > > Ordered delivery of commands to ANY TYPE of devices will increase in
> > > importance as network speeds increase and the need to hide latency
> > > increases.
> > >
> > > Today databases don't use queuing and rely and trickle the commands to
> > > devices 1 by 1 to ensure atomicity and order.
> > > As latency will become the determining factor in performance
> this is bound
> > > to change.
> > >
> > > SCSI has done an excellent job in defining the queueing
> mechanism. We have
> > > to make it work with good performance in our environment.
> > >
> > > Julo
> > >
> > > Santosh Rao <santoshr@cup.hp.com> on 13/04/2001 04:33:45
> > >
> > > Please respond to Santosh Rao <santoshr@cup.hp.com>
> > >
> > > To:   ips@ece.cmu.edu
> > > cc:   Black_David@emc.com
> > > Subject:  Re: iSCSI Requirements Draft - Informal WG Last Call
> > >
> > > David & All,
> > >
> > > I object to the following requirement :
> > >
> > > " MUST support ordered delivery of SCSI commands from the initiator to
> > > the
> > >   target, to support SCSI Task Queuing. "
> > >
> > > Ordered delivery is not a requirement for disk based applications and
> > > non tagged queueing tape applications, which form the majority of
> > > today's data traffic.
> > >
> > > To impose strict ordering (even in the presence of errors ?) as a MUST
> > > is penalizing the majority of today's data traffic that does
> not expect
> > > ordering from the SCSI subsystem.
> > >
> > > I am particularly concerned about the effect of the above
> requirement in
> > > the presence of errors. Does iSCSI expect strict ordering to be
> > > maintained even when individual I/O errors like ULP timeout occur ?
> > >
> > > On a ULP timeout (caused by, say, a hole in CmdSN), the initiator may
> > > choose not to retry the command, but instead, error it back
> to the ULP.
> > > In such a case, it can plug the hole in CmdSN with a NOP-OUT.
> > >
> > > The above requirement is not feasible to be met under such
> circumstances
> > > and others similar to this. Mandating strict ordering on ULP timeouts
> > > implies a session level error recovery on any individual I/O being
> > > failed back from iSCSI to SCSI ULP. This is a very heavy hammer to use
> > > as error recovery and should not be imposed.
> > >
> > > The above requirement must be changed to :
> > > " SHOULD support ordered delivery of SCSI commands from the
> initiator to
> > > the
> > >   target, to support SCSI Task Queuing. "
> > >
> > > - Santosh
> > >
> > > Black_David@emc.com wrote:
> > > >
> > > > It is intended to submit draft-ietf-ips-iscsi-reqmts-02.txt
> > > > as an Informational RFC. There is no formal requirement for
> > > > a WG Last Call, but if you have any further substantive comments
> > > > on the document please raise them on this list within the next
> > > > two weeks, i.e. by April 27th at the latest.
> > > >
> > > > If you have typographical/editorial comments please send them
> > > > direct to the document's author, Marjorie Krueger
> > > > <marjorie_krueger@hp.com>.
> > > >
> > > > Thanks,
> > > > --David and Elizabeth, IPS WG co-chairs
> > >  - santoshr.vcf