Re: ISCSI: Urgent Flag requirement violates TCP.

[Matt Wakeley <matt_wakeley@agilent.com>]

Black_David@emc.com wrote:

> Some form of Matt's longer explanation (see below) about
> the effects of coalescence needs to be substituted in
> the draft for the sentence:
>
>   "The result is the TCP urgent pointer will point to the first byte of
>         the iSCSI message in the TCP segment."
>
> because it's very easy to construct cases in which the
> TCP urgent pointer points out of the segment, making the
> above sentence false.  In particular, someone constructing
> a hardware framer that looks for iSCSI CDBs should not be
> mislead into relying on the urgent pointer to identify
> a useful (or even worse, the first useful) boundary
> in every inbound segment.
>
> Words are also needed to make it absolutely clear that
> TCP is not permitted to "deliver" out of order to the
> iSCSI layer - as Julian explained, the only point of
> this urgent mechanism is to help out framing logic that's
> processing the inbound stream, especially if SACK is being
> used. Among the disasters that this prevents is iSCSI
> completing and sending a response to a command when
> the delivery of that command's CDB to the target has
> not been ACKed by TCP (e.g., a TCP not using SACK).

You make a point here that things can't be delivered to iSCSI out of order, then
your reasoning below shows how this is not a requirement.

>
>
> Let me expound on this a bit more because it's come up
> in the past.  TCP is an ordered delivery mechanism;
> data must be "delivered" to the target in order.  In
> contrast, SCSI is mostly an unordered mechanism - unless
> order is explicitly specified, commands can complete out
> of order, and often do on disks and disk arrays.  I
> put "delivered" in quotes because the only way to know
> that a SCSI command has been "delivered" is to observe
> that execution of the command has started (e.g., the
> disaster in the previous paragraph where execution completed
> on data that was never ACKed by TCP is clear evidence that
> someone is not playing by the rules).

Right - it's impossible for a "status" to be received before the "command" for
that status was delivered.  So your "disaster" can't occur.

>  So, it's ok
> to parse/frame/etc. the data to set it up for execution
> of the SCSI command, but DON'T DO ANYTHING until TCP
> has "delivered" the command and all the data prior to
> it by sending or being prepared to send the appropriate
> ACKs.

This I disagree with.  If a tightly coupled TCP/iSCSI stack where able to regain
framing after a lost TCP segment, it is perfectly OK for iSCSI to process the
iSCSI messages and deliver them to the SCSI layer (with the exception that
commands must always be delivered in order), even before the lost segments
arrive.  This works perfectly well in Fibre Channel today.

For example, if the following iSCSI messages are sent:

Command A sent to target  ->
Command B sent to target  ->
<- target sends data/status for command A
<- target sends data/status for command B

If the TCP segment containing the data/status for A was lost, but framing was
regained to receive the data/status for B, there is nothing wrong with
completing command B even though a message for command A was lost and needs
retransmitting.  Indeed, the initiator SCSI layer would not know the difference
than if the disk drive completed command B first instead of A.

-Matt Wakeley
Agilent Technologies


>
>
> Meanwhile, I should note that a TCP stack may respond to having
> urgent data presented to it by sending immediately (i.e., and
> not waiting for the rest of the CDB).  For alignment reasons,
> using 1 byte of urgent data may not be the best choice.
>
> --David
>
> ---------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 42 South St., Hopkinton, MA  01748
> +1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
> black_david@emc.com       Mobile: +1 (978) 394-7754
> ---------------------------------------------------
>
> > -----Original Message-----
> > From: Matt Wakeley [SMTP:matt_wakeley@agilent.com]
> > Sent: Tuesday, November 07, 2000 4:44 PM
> > To:   Douglas Otis
> > Cc:   julian_satran@il.ibm.com; ips@ece.cmu.edu
> > Subject:      Re: ISCSI: Urgent Flag requirement violates TCP.
> >
> > Douglas Otis wrote:
> >
> > > ISCSI: Urgent Flag requirement violates TCP.
> >
> > No it doesn't.
> >
> > > Julian,
> > >
> > > The result indicated within the iSCSI draft on page 14 will require TCP
> > to
> > > be modified for this statement to be true.  By requiring every message
> > to be
> > > marked, a continued transparent update of the pointer will obscure the
> > > urgent pointer by continued coalescence.  This obscuring coalescence
> > will
> > > occur at both the sender and receiver.  Only a state between normal and
> > > urgent is signaled at the receiver and not the actual pointer.  Unless
> > there
> > > is an intent to modify TCP, this scheme offers no benefit even if
> > offending
> > > portions of the urgent proposal is removed.
> >
> > We don't care if sometimes coalescing happens.  The point is, that
> > somewhere
> > down the stream, an urgent pointer will point to the beginning of a new
> > iSCSI
> > message and messages after this point can be processed.  All messages
> > between
> > the point of the dropped segment and this point will of course have to be
> > retransmitted.
> >
> >
> > >    The iSCSI protocol uses the urgent bit in the TCP header to delineate
> > >    iSCSI messages. The first byte of every iSCSI message MUST be marked
> > >    "urgent".  The result is the TCP urgent pointer will point to the
> > >    first byte of the iSCSI message in the TCP segment.
> > >
> > > This pointer will be skewed by the size of the send buffer (beyond the
> > TCP
> > > segment) and prevent normal use unless again TCP is modified to comply
> > with
> > > stated expectations.  I would advise complete removal of this Urgent
> > flag
> > > requirement if to remain within the WG charter as only a modified TCP
> > could
> > > use this feature in the intended manner.  This intended manner is
> > plainly
> > > spelled out with concerns about BSD vs. RFC 1122 compliance. There is no
> > > benefit from an occasional toggle between normal and urgent mode.  If
> > you
> > > wish to use a datagram or record based protocol, you are advised to
> > review
> > > RFC 2960.
> >
> > The intended manner that is to be implemented is this:
> >
> > 1. iSCSI sends the first byte of the iSCSI message to TCP, with the byte
> > marked
> > "urgent".  You can do this today using any off the shelf TCP stack,
> > including MS
> > windows.
> > 2. iSCSI sends the rest of the bytes of the iSCSI message to TCP with
> > normal
> > delivery.
> >
> > If TCP coalesces, fine.  But eventually, a TCP segment will be sent that
> > will
> > contain a byte referenced by the urgent pointer.  At this point, the
> > remote side
> > (and any LAN analyzers on the network) will be able to "sync up" on iSCSI
> > messages.
> >
> > Remember, this "urgent pointer" stuff is *only* to be used in the cases
> > where a
> > TCP segment is dropped, not for normal delivery.
> >
> > So, there is *no* violation of any RFC or WG charter.
> >
> > >
> > >
> > > Doug
> >
> > -Matt Wakeley
> > Agilent Technologies