RE: iSCSI: Flow Control

["Douglas Otis" <dotis@sanlight.net>]

Pierre,

You will be happy to know your technique is supported within SCTP as a
standard means with feedback on send buffer size in chunks to allow pacing.
By using SCTP, you can dictate priority of any stream (queue) in addition to
allowing out-of-sequence delivery to also prevent head of queue blocking at
the receive side as well.  You will note in my proposal, the streams carry
both priority as well as serving as the Service Delivery Port.

Doug

>
> Douglas Otis wrote:
>
> > David,
> >
> > I think that Pierre is explaining how to re-engineer a TCP
> stack to allow
> > just-in-time delivery of iSCSI packets to be sent to allow last moment
> > re-ordering.  As he points out on the receive side, as long as the TCP
> > segments are in order, the receive side should be shallow.
>
> David,
>
> I agree with Doug.
>
> Regards,
>
> Pierre
>
> > His concept of
> > head of queue blocking is only from the perspective of the sending side.
> > One other means might include some heuristic indicating the
> size of the send
> > buffer but that would be part of the pacing he is employing.  I would
> > imagine two send queues would be employed with different
> priorities.  These
> > would not be a standard TCP implementations but of no concern to the
> > proposed draft.
> >
> > Doug
> >
> > > Pierre,
> > > I don't know if I completely understand what you are proposing,
> > > but is seems that you are proposing to process TCP segments out
> > > of order.  As I have said in a previous message, this is extremely
> > > dangerous as the TCP layer will not ACK any segment until all previous
> > > segments are processed. Without an ACK the segment may be
> retransmitted
> > > many times and that will require iSCSI to track what has been
> processed
> > > and what has not by adding a segment number.  Essentially duplicating
> > > TCP segment numbers, SACK doesn't help either.
> > >
> > > If you are proposing that iSCSI simply keep receiving data without
> > > doing the SCSI layer processing and thus at the SCSI layer processing
> > > them out of order.  This is feasible but it is still subject to buffer
> > > restrictions that would cause data to be discarded which is the whole
> > > point of these flow control discussions, to minimize data being
> > > discarded.
> > >
> > > What I strongly object to is any feature in the iSCSI layer that
> > > requires
> > > any direct manipulation of the TCP layer features like the window
> > > pointers.
> > > Implementations are free to violate layering as an optimization but
> > > it MUST be possible to have a functional implementation without
> > > knowing any details of the TCP implemenation.
> > >
> > >       -David
> > >
> > > Pierre Labat wrote:
> > > >
> > > > julian_satran@il.ibm.com wrote:
> > > >
> > > > > Pierre,
> > > > >
> > > > > You are wrong again. When the target reopens the window -
> > > i.e., reads some
> > > > > data from the
> > > > > pipe at his end you get to put your Read command - but it
> > > goes after the
> > > > > rest of the window and
> > > > > window can be several megabytes.
> > > >
> > > > Julian,
> > > >
> > > > The TCP window is not a buffer on the receive side.
> > > > On the receive side, in our case (the target) and as far as
> TCP segments
> > > > arrive
> > > > in order, there is not an opaque  FIFO containing a full
> window size of
> > > > command/data
> > > > waiting to be processed. You can avoid that.
> > > > What the target does is: receive bytes through the TCP
> > > connection, does the
> > > > TCP work
> > > > and forms a iSCSI PDU. The maximum you have to store is a few
> > > TCP segments
> > > > to re-build the PDU. As soon as the PDU is built it is processed.
> > > > When the target wants to close the TCP window it updates
> accordingly the
> > > > window and CONTINUEs to process the incoming PDUs.
> > > > At that point you assume that the incoming PDUs are put in an
> > > opaque FIFO, but
> > > >
> > > > rather than that,  the target can process them and put the data
> > > a the right
> > > > location in the target cache.
> > > > Then, when the window is opened again and the read PDU comes,
> > > it is processed
> > > > immediately.
> > > >
> > > > In fact as Y P Cheng described in a previous mail in this
> > > thread, the model
> > > > that
> > > > can be used for iSCSI traffic is different of the common model
> > > we have for
> > > > regular
> > > > TCP/IP networking although a  TCP fully complient with the
> > > RFCs can be used
> > > > for iSCSI.
> > > > In regular TCP/IP networking the application (on the transmit
> > > side) fills
> > > > a FIFO that the adapter empties. In our case as explained
> by Y P Cheng
> > > > you replace the FIFO by an "exchange table" what i called a flat
> > > > array. It allows you to avoid the head of queue blocking at
> this level.
> > > >
> > > > On the receive side (the target in our case) in regular networking,
> > > > the incoming data are tossed in a FIFO by TCP. The application
> > > > empties this FIFO and can block (in this case the FIFO grows)
> > > > and yes, when the application unblock, it has a large amount
> > > > of PDUs to process.
> > > > But in the model described the application never blocks. Hence
> > > there is no
> > > > big receive opaque FIFO on the target. In our case the
> > > application is the
> > > > module that
> > > > process the iSCSI pdus. The application never blocks because it
> > > is able to
> > > > pace down
> > > > the flow coming from the initiator with the TCP window and
> the command
> > > > flow control (MaxCmdRN).
> > > >
> > > > Regards,
> > > >
> > > > Pierre
>