Re: ISCSI: Urgent Flag requirement violates TCP.

[Matt Wakeley <matt_wakeley@agilent.com>]

Black_David@emc.com wrote:

> > >  So, it's ok
> > > to parse/frame/etc. the data to set it up for execution
> > > of the SCSI command, but DON'T DO ANYTHING until TCP
> > > has "delivered" the command and all the data prior to
> > > it by sending or being prepared to send the appropriate
> > > ACKs.
> >
> > This I disagree with.  If a tightly coupled TCP/iSCSI stack where able to
> regain
> > framing after a lost TCP segment, it is perfectly OK for iSCSI to process
> the
> > iSCSI messages and deliver them to the SCSI layer (with the exception that
> > commands must always be delivered in order), even before the lost segments
> > arrive.  This works perfectly well in Fibre Channel today.
>
> Fibre Channel is a bad analogy in this regard because Fibre Channel does not
> in general have the ordered delivery requirement that TCP does.  If a frame
> (say frame N) goes missing in Fibre Channel, it's usually the case that
> subsequent frames (i.e., N+1, etc.) can be delivered to the ULP (e.g., SCSI)
> and
> processed.
> In this case, it's the responsibility of the ULP to figure out what
> happened, including doing any retransmits.

Not exactly.  The FC-2 and FC-4 layers in FC determine how and what to do
when FC frames are dropped (FCP-2).  Not the ULP communicating over FC (SCSI).


> In contrast, if a TCP segment goes missing, subsequent
> segments can be buffered at the receiver (and ACKed if SACK is being used),
> but MUST NOT be delivered to the ULP (e.g., iSCSI or SCSI).

David, I agree that *if* I was simply building a TCP, that this TCP must deliver
the *byte stream* to the ULP in order.

> TCP takes responsibility for sorting out what happened, doing the retransmits,
> and
> hiding the fact that things went wrong (aside from the fact that they took
> longer)
> from the ULP.  This is fundamentally different from Fibre Channel.

Since TCP is already defined, we should not discuss what TCP can or can't
deliver to iSCSI.
The original issue you raised is the behavior of the interface between iSCSI and
it's ULP (SCSI).  *If* iSCSI was implemented over UDP or SCTP, there would be no
discussion on the orderness of iSCSI message delivery to SCSI, since the
interface between iSCSI and it's ULP (SCSI) hides the lower layers. So, why are
we having this discussion just because the lower level transport is TCP?.

>
>
> > For example, if the following iSCSI messages are sent:
> >
> > Command A sent to target  ->
> > Command B sent to target  ->
> > <- target sends data/status for command A
> > <- target sends data/status for command B
> >
> > If the TCP segment containing the data/status for A was lost, but framing
> was
> > regained to receive the data/status for B, there is nothing wrong with
> > completing command B even though a message for command A was lost and
> needs retransmitting.  Indeed, the initiator SCSI layer would not know the
> difference than if the disk drive completed command B first instead of A.
>
> From a SCSI standpoint this is correct, but from a TCP standpoint it is wrong.
>
> For example, if command B is an iSCSI ping, and SACK is not in use, this is
> exactly the disaster of returning SCSI status for a command for which
> delivery of the CDB has not been ACKed by TCP.

You still haven't described *WHY* this is a "disaster".  What is so disasterous
about it?

> Note that SCSI is allowed to complete command B (e.g., an iSCSI ping) before
> command A

> (e.g., A write with R2T used

> for data), but the presentation of commands to SCSI/iSCSI MUST respect TCP's
> in-order delivery requirements

Hold it! Now you're mixing transports with mapping protocols again!  Just
because TCP is "in-order" has nothing to do with how the SCSI/iSCSI interface is
defined. If iSCSI was implemented over UDP or SCTP, there would not be this
"in-order" mentality. iSCSI already specifies that SCSI commands are to be
delivered to SCSI in the same order they were presented to iSCSI (and it
achieves this by useing the Command reference numbers).  Therefore, there should
be no requirements on the "orderness" on the SCSI/iSCSI interface based on what
the lower level transport is.

> (e.g., at the point where status from B comes back,
> it had better be the case that TCP has ACKed delivery of A's CDB,

Why? What is the "disaster" if TCP has not ACKed delivery of A's CDB? Nothing.

> but I still think the requirement is better described in terms of initiation
> of command
> execution at the target).
>
> I have to agree with Doug on the higher level point here -- if one wants a
> message
> oriented transport in which the messages can be delivered out of order on a
> single connection, that's not TCP, and something like SCTP will need to be
> used.

If one wants to do a message oriented transport, and figures out a way to do it
with the existing definition of TCP, this "mapping" specification should not
disallow it.  All this "mapping" specification needs to do is define what runs
up and down the wire so that the endpoints understand each other.  This
"mapping" already specifies that commands are delivered in order.  No other
messages matter if they are out of order or not.

>
>
> Also ... given Matt's recent note that the use of Urgent is not applicable
> to all implementations of iSCSI, it appears to me that the use of "MUST" to
> describe this requirement in the draft is incorrect and needs to be changed.

Look, the whole purpose of doing iSCSI is to take advantage of the high speed
links coming along (1Gbps and 10Gbps and...).  The only way link speed can be
achived with these wire rates is to offload the TCP/IP processing from the main
cpu and OS.  This means that it will probably be moved to HBAs or hardware
implementations.  These high speed implementations will require framing in order
to prevent a massive amount of buffer resources to "buffer up" TCP segments that
arrive after a dropped TCP segment.  It's true that a software implementation
will not be able to take advantage of the "framing" features.  But the next
generation devices will, and if they cannot depend on the feature being there,
it will make these next generation devices more complex, more expensive and take
longer to develop.  The requirement must remain a "must".

-Matt Wakeley
Agilent Technologies

>
>
> --David
>
> ---------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 42 South St., Hopkinton, MA  01748
> +1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
> black_david@emc.com       Mobile: +1 (978) 394-7754
> ---------------------------------------------------