Re: new iSCSI draft - 02.txt

[Kevin Quick <kquick@iphase.com>]

: From: Matt Wakeley <matt_wakeley@agilent.com>
: 
: David Peterson wrote:
: > 
: > I believe there was agreement to remove the Urgent-Pointer framing mechanism
: > but don't recall any agreement to replace it with an in-stream marker. For a
: > software implementation it would be hard to support this type of framing
: > mechanism. I believe a TCP option indicating the message boundry or a
: > fixed-length PDU at a granularity to minimize overhead are much better
: > solutions and are workable for both software and hardware implementations. I
: > have not seen an agenda but I would hope this issue would be discussed at
: > the upcoming meeting in Orlando.
: > Dave
: 
: What is so difficult for software to insert a few extra bytes in the byte
: stream?  It's simply a layering problem:
: 
: iSCSI layer <-> marker layer <-> TCP
: 
: Normally, the marker layer simply transfers the bytes from the iSCSI layer to
: the TCP. Every X number of bytes, the marker layer inserts the marker into the
: byte stream.
: 
: Since your software will probably not benefit from the receipt of the markers,
: it would negotiate not to receive the markers.  It would only send markers
: *IF* the remote node requested them to be sent.
: 
: -Matt Wakeley
: Agilent Technologies


I would think that there would have to be a number of changes to existing
TCP implementations to utilize TCP-option-based framing:

* Modification of the transmit request interface to indicate frame
  boundaries
* Modification of the TCP transmit segmentation algorithm to recognize
  and preserve frame boundaries
* Modification of TCP slow-start window sizing to recognize and preserve
  frame boundaries
* Modification of MTU path discovery, with possible rejection of some
  frame transmit requests at the transmit request interface if the frame
  exceeds allowable boundaries.
* Possible delivery failures if the path is silently changed or
  multiplexed after initial discovery?
* Possible failures if "legacy" nodes either preserve option data but
  re-segment the traffic, or discard the "new" option information
  entirely.
  ** Firewalls, data-caching nodes, layer-3 routers, and other
     store-and-forward nodes would be susceptible.
* Modification of the receive path socket management to preserve frame
  boundaries.
* If any node re-transmits any portion of the data stream (due to NAK or
  ACK-timeout) then the retransmit logic must be modified to identify the
  frame boundaries in the retransmitted portion, recalculate the TCP
  options for that retransmitted portion (since the headers may occur at a
  new position relative to the datastream), and then perhaps resynchronize
  at the original location when the remote ACKs the rest of the previously
  transmitted window.

Some or all of the above may not be typical situations, or even generally
"seen", but since we are defining a standard, it's important to define
that standard relative to what the existing standards *allow*, not
relative to how current sw/hw is *implemented*.  [Note: I'm not familiar
enough with the details of the standards myself to know if the concerns
above are warranted... I'm trying to point out potential issues here for
consideration by more knowledgeable experts.]

As Matt points out above, in-band markers are unequivocably perserved as a
consequence of being part of the data stream, and can be handled easily by
a layer above the existing TCP management at the endpoints only.

Additionally, an in-band marker can forward reference the id of the next
command in the stream, thereby allowing receive hardware to prefetch DMA
buffer information for that next command.  This usage is congruent with
the high-performance considerations of this general effort.  Note that
this enhances both loss-less and loss-error transfers, since a marker
allows resynchronization only at the *next* message anyhow.

Regards,
  Kevin