RE: iSCSI:flow control, acknowledgement, and a deterministic reco very

[julian_satran@il.ibm.com]

Doug & David,

Commands can be recovered based on their ITT (that is the main identifier).

Nevertheless the idea of a separate immediate delivery flag is appealing.

The idea was raised many months ago and I will have to review some old
notes to find why we dropped it in the first place (except that FCP at the
time and SCSI now use the same marking - i.e. 0 for immediate).

Julo

Black_David@emc.com on 06/04/2001 15:11:26

Please respond to Black_David@emc.com

To:   dotis@sanlight.net, ips@ece.cmu.edu
cc:
Subject:  RE: iSCSI:flow control, acknowledgement, and a deterministic reco
      very




Let me see if I can distill out the issue in Doug's long
message on this subject ...

When a CmdSN of zero is used to mark a command for immediate
delivery, the CmdSN-based acknowledgement and windowing
mechanisms don't apply to that command.  This means that
any command sent for immediate delivery (CmdSN = 0):
(A) Cannot recover from a CRC (digest) error via CmdSN-based
     retransmit.  The Initiator can still time this out,
     but that seems like a poor way to initiate recovery
     of something that was supposed to be done immediately.
(B) Does not have its use of resources (e.g. command buffer)
     on the target controlled by the CmdSN windowing
     mechanism, complicating target resource management.
     Targets have better control over their resources if
     all inbound commands use one set of resources,
     described/controlled by the CmdSN window, rather
     than having to set aside separate resources just
     in case the initiator sends an immediate command.
The alternate proposal that Doug envisions is to apply
CmdSN (and hence its error recovery and resource management)
to immediate commands, and use a flag  bit elsewhere in
the header to indicate that the command is to be delivered
to SCSI immediately by iSCSI rather than waiting for
"missing" commands to show up.  That seems reasonable,
and comments are invited.  One of the things that this
does is transfer the responsibility to keep some space
in the CmdSN window open for immediate commands (and
determine how much is appropriate) from the Target to
the Initiator - all other things being equal, this
is the right direction to move functionality in a SCSI
system.

Note Well: This concept of "immediate" delivery is an
iSCSI concept that affects only the iSCSI CmdSN sequence
- this does not affect TCP's "deliver in order" behavior.

Thanks,
--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
black_david@emc.com       Mobile: +1 (978) 394-7754
---------------------------------------------------

> -----Original Message-----
> From:   Douglas Otis [SMTP:dotis@sanlight.net]
> Sent:   Thursday, April 05, 2001 11:34 PM
> To:     Ips
> Cc:     David Black; Elizabeth G Rodriguez (Elizabeth)
> Subject:     iSCSI:flow control, acknowledgement, and a deterministic
> recovery
>
> Ver 5 Pg. 10
>
>    "Command numbering is session-wide and is used for ordered command
>    delivery over multiple connections.  It can also be used as a
>    mechanism for command flow control over a session."
>
> "1.2.2.1 Command Numbering and Acknowledging
>
>    iSCSI supports ordered command delivery within a session.  All
>    commands (initiator-to-target) are numbered."
>
>    "Commands in transit from the initiator SCSI layer to the SCSI target
>    layer are numbered by iSCSI; the number is carried by the iSCSI PDU
>    as CmdSN (Command-Sequence-Number).  The numbering is session-wide.
>    All iSCSI PDUs that have a task association carry this number. CmdSNs
>    are allocated by the initiator iSCSI within a 32-bit unsigned counter
>    (modulo 2**32).  The value 0 is reserved and used to mean immediate
>    delivery. Comparisons and arithmetic on CmdSN SHOULD use Serial
>    Number Arithmetic as defined in [RFC1982] where SERIAL_BITS = 32.
>
>    Not covered in this document are he means by which the SCSI layer may
>    request immediate delivery for a command or by which iSCSI will
>    decide by itself to mark a PDU for immediate delivery."
>
>
> Not all commands are serialized as a result of serial number 0
> representing
> a special case for immediate delivery.  This has an impact on flow
> control,
> acknowledgement, and a deterministic recovery in the face of an error
> situation.  With the exception of those commands with null serialization,
> all commands MUST be sequenced at the point of network aggregation
> described
> here as a PDU sequencer that issues commands to the SCSI target.  This
> focal
> point is likely to find situations where its normal operation is
> curtailed.
>
>    1) Prolonged device operation resulting in a resource constraint.
>    2) Digest Error causing a missing sequence.
>    3) Connection loss causing a missing sequence.
>
> The technique of using a null sequence to bypass the sequencer has some
> inherent problems.  In first case, those queued commands MAY become
> invalid
> following management that terminates prolonged operation with a command
> that
> has bypassed the sequencer.  Those invalidated commands queued within the
> sequencer can not be cancelled in an orderly manner within the existing
> scheme.  This sequencer MUST be used as defined in the iSCSI proposal
and,
> as a result, these queued commands are beyond SCSI and iSCSI controls.
> Issuing a null sequence command followed by a replicate serialized
command
> will have differing results depending on the target but will not result
in
> a
> deterministic treatment of these pending commands.
>
> The use of the sequencer bypass technique (null serialization) should
> signal
> an extreme measure where logically, those commands being bypassed become
> suspect.  The conservative approach to this situation would be to reject
> all
> bypassed commands.  As a result of this conservative behavior, a
technique
> that does not use a null sequence would be to institute a flag such as
> "Exigent" that signals an extreme condition exists and that all pending
> commands within the sequencer are to be rejected.
>
> Note:
> In addition to checking for the next PDU sequence, the sequencer should
be
> checking for PDUs with a serialization number that are prior to the
> desired
> next sequence.  This examination would look something like this:
>
> if ( (sequencer CmdSN - next sequence CmdSN ) > 2^(SERIAL_BITS - 1))
>    {
>    reject_pdu(CmdSN, SEQUENCER_INVALIDATION);
>    }
>
> if (sequencer CmdSN == next sequence CmdSN)
>    {
>    send_pdu(CmdSN);
>    next sequence++;
>    }
>
> Upon receipt of a PDU flagged as "Exigent", the 'next sequence' value
> immediately becomes the serial number of this command as well as ExpCmd
> advancing to this value plus one.  The effect of this is to have the
> sequencer reject all pending commands up to the "Exigent" command.  This
> has
> the benefit of removing these now suspect commands as well as allowing
> this
> highly urgent command to be sent to the target immediately without
> accidentally affecting subsequent commands as is now possible.
>
> As it is now, one other possible use of a null sequence would be to
always
> bypass the sequencer as perhaps the sequencer is viewed as unnecessary.
> The
> use of zero to allow commands to bypass the sequencer then represents a
> problem with respect to resource management as now the flow control
scheme
> no longer works.  If bypassing the sequencer is the desired behavior and
> this command will not impact validity of those commands serialized prior
> to
> this command, then this PDU flagged "Casual" would allow this command to
> be
> issued directly to the target.  These commands should still include a
> serial
> number to allow flow control and acknowledgement to remain functional.
> The
> task of acknowledgement would be to comprise a min-max list of those
> commands sent and to look for the highest sequential value.
>
> In the case of a lost connection, waiting to time-out on those holes
> created
> within the sequencer would be one method of handling this situation.  If
> there was a means of rejecting those commands within the sequencer using
> an
> "Exigent" command would mean there are no holes left to fill.  Those
> commands received would be rejected and the initiator could then resend
> these commands on a different connection without stumbling through a
> process
> of repeated timeouts.  Should the method used to recover from a digestion
> error mean terminating the connection, then those commands can also be
> quickly shifted over to a new connection.
>
> This does not resolve all issues created in a error event, but it does
> provide a simpler solution for most of those events concerning the
> sequencer
> and gets rid of the special case handing of the command serial number.
> Not
> having flow control, acknowledgement, and a method of dealing with queued
> commands appear as a serious flaw in the present protocol.  I hope I have
> presented this in a clear manner and I am interested in finding how
others
> deal with these situations.
>
> Doug