FW: iSCSI:flow control, acknowledgement, and a deterministic reco very

["BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" <matthew_burbridge@hp.com>]

Alternatively, only allow one command with CmdSN=0 outstanding at anyone
time.  This has the advantage of only having to reserve one buffer at the
target,  the retry bit could still be used if the command or response has a
digest error, and finally its simple!.  The obvious disadvantage is that
there can only be one message is progress at a time - is this an issue?

Matthew Burbridge
NIS-Bristol
Hewlett Packard
Telnet: 312 7010
E-mail: matthewb@bri.hp.com


> -----Original Message-----
> From: Black_David@emc.com [mailto:Black_David@emc.com]
> Sent: 06 April 2001 14:11
> To: dotis@sanlight.net; ips@ece.cmu.edu
> Subject: RE: iSCSI:flow control, acknowledgement, and a deterministic
> reco very
> 
> 
> Let me see if I can distill out the issue in Doug's long
> message on this subject ...
> 
> When a CmdSN of zero is used to mark a command for immediate
> delivery, the CmdSN-based acknowledgement and windowing
> mechanisms don't apply to that command.  This means that
> any command sent for immediate delivery (CmdSN = 0):
> (A) Cannot recover from a CRC (digest) error via CmdSN-based
> 	retransmit.  The Initiator can still time this out,
> 	but that seems like a poor way to initiate recovery
> 	of something that was supposed to be done immediately.
> (B) Does not have its use of resources (e.g. command buffer)
> 	on the target controlled by the CmdSN windowing
> 	mechanism, complicating target resource management.
> 	Targets have better control over their resources if
> 	all inbound commands use one set of resources,
> 	described/controlled by the CmdSN window, rather
> 	than having to set aside separate resources just
> 	in case the initiator sends an immediate command. 
> The alternate proposal that Doug envisions is to apply
> CmdSN (and hence its error recovery and resource management)
> to immediate commands, and use a flag  bit elsewhere in
> the header to indicate that the command is to be delivered
> to SCSI immediately by iSCSI rather than waiting for
> "missing" commands to show up.  That seems reasonable,
> and comments are invited.  One of the things that this
> does is transfer the responsibility to keep some space
> in the CmdSN window open for immediate commands (and
> determine how much is appropriate) from the Target to
> the Initiator - all other things being equal, this
> is the right direction to move functionality in a SCSI
> system.
> 
> Note Well: This concept of "immediate" delivery is an
> iSCSI concept that affects only the iSCSI CmdSN sequence
> - this does not affect TCP's "deliver in order" behavior.
> 
> Thanks,
> --David
> 
> ---------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 42 South St., Hopkinton, MA  01748
> +1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
> black_david@emc.com       Mobile: +1 (978) 394-7754
> ---------------------------------------------------
> 
> > -----Original Message-----
> > From:	Douglas Otis [SMTP:dotis@sanlight.net]
> > Sent:	Thursday, April 05, 2001 11:34 PM
> > To:	Ips
> > Cc:	David Black; Elizabeth G Rodriguez (Elizabeth)
> > Subject:	iSCSI:flow control, acknowledgement, and a deterministic
> > recovery
> > 
> > Ver 5 Pg. 10
> > 
> >    "Command numbering is session-wide and is used for 
> ordered command
> >    delivery over multiple connections.  It can also be used as a
> >    mechanism for command flow control over a session."
> > 
> > "1.2.2.1 Command Numbering and Acknowledging
> > 
> >    iSCSI supports ordered command delivery within a session.  All
> >    commands (initiator-to-target) are numbered."
> > 
> >    "Commands in transit from the initiator SCSI layer to 
> the SCSI target
> >    layer are numbered by iSCSI; the number is carried by 
> the iSCSI PDU
> >    as CmdSN (Command-Sequence-Number).  The numbering is 
> session-wide.
> >    All iSCSI PDUs that have a task association carry this 
> number. CmdSNs
> >    are allocated by the initiator iSCSI within a 32-bit 
> unsigned counter
> >    (modulo 2**32).  The value 0 is reserved and used to 
> mean immediate
> >    delivery. Comparisons and arithmetic on CmdSN SHOULD use Serial
> >    Number Arithmetic as defined in [RFC1982] where SERIAL_BITS = 32.
> > 
> >    Not covered in this document are he means by which the 
> SCSI layer may
> >    request immediate delivery for a command or by which iSCSI will
> >    decide by itself to mark a PDU for immediate delivery."
> > 
> > 
> > Not all commands are serialized as a result of serial number 0
> > representing
> > a special case for immediate delivery.  This has an impact on flow
> > control,
> > acknowledgement, and a deterministic recovery in the face 
> of an error
> > situation.  With the exception of those commands with null 
> serialization,
> > all commands MUST be sequenced at the point of network aggregation
> > described
> > here as a PDU sequencer that issues commands to the SCSI 
> target.  This
> > focal
> > point is likely to find situations where its normal operation is
> > curtailed.
> > 
> > 	1) Prolonged device operation resulting in a resource 
> constraint.
> > 	2) Digest Error causing a missing sequence.
> > 	3) Connection loss causing a missing sequence.
> > 
> > The technique of using a null sequence to bypass the 
> sequencer has some
> > inherent problems.  In first case, those queued commands MAY become
> > invalid
> > following management that terminates prolonged operation 
> with a command
> > that
> > has bypassed the sequencer.  Those invalidated commands 
> queued within the
> > sequencer can not be cancelled in an orderly manner within 
> the existing
> > scheme.  This sequencer MUST be used as defined in the 
> iSCSI proposal and,
> > as a result, these queued commands are beyond SCSI and 
> iSCSI controls.
> > Issuing a null sequence command followed by a replicate 
> serialized command
> > will have differing results depending on the target but 
> will not result in
> > a
> > deterministic treatment of these pending commands.
> > 
> > The use of the sequencer bypass technique (null 
> serialization) should
> > signal
> > an extreme measure where logically, those commands being 
> bypassed become
> > suspect.  The conservative approach to this situation would 
> be to reject
> > all
> > bypassed commands.  As a result of this conservative 
> behavior, a technique
> > that does not use a null sequence would be to institute a 
> flag such as
> > "Exigent" that signals an extreme condition exists and that 
> all pending
> > commands within the sequencer are to be rejected.
> > 
> > Note:
> > In addition to checking for the next PDU sequence, the 
> sequencer should be
> > checking for PDUs with a serialization number that are prior to the
> > desired
> > next sequence.  This examination would look something like this:
> > 
> > if ( (sequencer CmdSN - next sequence CmdSN ) > 2^(SERIAL_BITS - 1))
> > 	{
> > 	reject_pdu(CmdSN, SEQUENCER_INVALIDATION);
> > 	}
> > 
> > if (sequencer CmdSN == next sequence CmdSN)
> > 	{
> > 	send_pdu(CmdSN);
> > 	next sequence++;
> > 	}
> > 
> > Upon receipt of a PDU flagged as "Exigent", the 'next 
> sequence' value
> > immediately becomes the serial number of this command as 
> well as ExpCmd
> > advancing to this value plus one.  The effect of this is to have the
> > sequencer reject all pending commands up to the "Exigent" 
> command.  This
> > has
> > the benefit of removing these now suspect commands as well 
> as allowing
> > this
> > highly urgent command to be sent to the target immediately without
> > accidentally affecting subsequent commands as is now possible.
> > 
> > As it is now, one other possible use of a null sequence 
> would be to always
> > bypass the sequencer as perhaps the sequencer is viewed as 
> unnecessary.
> > The
> > use of zero to allow commands to bypass the sequencer then 
> represents a
> > problem with respect to resource management as now the flow 
> control scheme
> > no longer works.  If bypassing the sequencer is the desired 
> behavior and
> > this command will not impact validity of those commands 
> serialized prior
> > to
> > this command, then this PDU flagged "Casual" would allow 
> this command to
> > be
> > issued directly to the target.  These commands should still 
> include a
> > serial
> > number to allow flow control and acknowledgement to remain 
> functional.
> > The
> > task of acknowledgement would be to comprise a min-max list of those
> > commands sent and to look for the highest sequential value.
> > 
> > In the case of a lost connection, waiting to time-out on those holes
> > created
> > within the sequencer would be one method of handling this 
> situation.  If
> > there was a means of rejecting those commands within the 
> sequencer using
> > an
> > "Exigent" command would mean there are no holes left to fill.  Those
> > commands received would be rejected and the initiator could 
> then resend
> > these commands on a different connection without stumbling through a
> > process
> > of repeated timeouts.  Should the method used to recover 
> from a digestion
> > error mean terminating the connection, then those commands 
> can also be
> > quickly shifted over to a new connection.
> > 
> > This does not resolve all issues created in a error event, 
> but it does
> > provide a simpler solution for most of those events concerning the
> > sequencer
> > and gets rid of the special case handing of the command 
> serial number.
> > Not
> > having flow control, acknowledgement, and a method of 
> dealing with queued
> > commands appear as a serious flaw in the present protocol.  
> I hope I have
> > presented this in a clear manner and I am interested in 
> finding how others
> > deal with these situations.
> > 
> > Doug
>