Re: iSCSI: Flow Control

[Pierre Labat <pierre_labat@hp.com>]

Douglas Otis wrote:

> David,
>
> I think that Pierre is explaining how to re-engineer a TCP stack to allow
> just-in-time delivery of iSCSI packets to be sent to allow last moment
> re-ordering.  As he points out on the receive side, as long as the TCP
> segments are in order, the receive side should be shallow.

David,

I agree with Doug.

Regards,

Pierre

> His concept of
> head of queue blocking is only from the perspective of the sending side.
> One other means might include some heuristic indicating the size of the send
> buffer but that would be part of the pacing he is employing.  I would
> imagine two send queues would be employed with different priorities.  These
> would not be a standard TCP implementations but of no concern to the
> proposed draft.
>
> Doug
>
> > Pierre,
> > I don't know if I completely understand what you are proposing,
> > but is seems that you are proposing to process TCP segments out
> > of order.  As I have said in a previous message, this is extremely
> > dangerous as the TCP layer will not ACK any segment until all previous
> > segments are processed. Without an ACK the segment may be retransmitted
> > many times and that will require iSCSI to track what has been processed
> > and what has not by adding a segment number.  Essentially duplicating
> > TCP segment numbers, SACK doesn't help either.
> >
> > If you are proposing that iSCSI simply keep receiving data without
> > doing the SCSI layer processing and thus at the SCSI layer processing
> > them out of order.  This is feasible but it is still subject to buffer
> > restrictions that would cause data to be discarded which is the whole
> > point of these flow control discussions, to minimize data being
> > discarded.
> >
> > What I strongly object to is any feature in the iSCSI layer that
> > requires
> > any direct manipulation of the TCP layer features like the window
> > pointers.
> > Implementations are free to violate layering as an optimization but
> > it MUST be possible to have a functional implementation without
> > knowing any details of the TCP implemenation.
> >
> >       -David
> >
> > Pierre Labat wrote:
> > >
> > > julian_satran@il.ibm.com wrote:
> > >
> > > > Pierre,
> > > >
> > > > You are wrong again. When the target reopens the window -
> > i.e., reads some
> > > > data from the
> > > > pipe at his end you get to put your Read command - but it
> > goes after the
> > > > rest of the window and
> > > > window can be several megabytes.
> > >
> > > Julian,
> > >
> > > The TCP window is not a buffer on the receive side.
> > > On the receive side, in our case (the target) and as far as TCP segments
> > > arrive
> > > in order, there is not an opaque  FIFO containing a full window size of
> > > command/data
> > > waiting to be processed. You can avoid that.
> > > What the target does is: receive bytes through the TCP
> > connection, does the
> > > TCP work
> > > and forms a iSCSI PDU. The maximum you have to store is a few
> > TCP segments
> > > to re-build the PDU. As soon as the PDU is built it is processed.
> > > When the target wants to close the TCP window it updates accordingly the
> > > window and CONTINUEs to process the incoming PDUs.
> > > At that point you assume that the incoming PDUs are put in an
> > opaque FIFO, but
> > >
> > > rather than that,  the target can process them and put the data
> > a the right
> > > location in the target cache.
> > > Then, when the window is opened again and the read PDU comes,
> > it is processed
> > > immediately.
> > >
> > > In fact as Y P Cheng described in a previous mail in this
> > thread, the model
> > > that
> > > can be used for iSCSI traffic is different of the common model
> > we have for
> > > regular
> > > TCP/IP networking although a  TCP fully complient with the
> > RFCs can be used
> > > for iSCSI.
> > > In regular TCP/IP networking the application (on the transmit
> > side) fills
> > > a FIFO that the adapter empties. In our case as explained by Y P Cheng
> > > you replace the FIFO by an "exchange table" what i called a flat
> > > array. It allows you to avoid the head of queue blocking at this level.
> > >
> > > On the receive side (the target in our case) in regular networking,
> > > the incoming data are tossed in a FIFO by TCP. The application
> > > empties this FIFO and can block (in this case the FIFO grows)
> > > and yes, when the application unblock, it has a large amount
> > > of PDUs to process.
> > > But in the model described the application never blocks. Hence
> > there is no
> > > big receive opaque FIFO on the target. In our case the
> > application is the
> > > module that
> > > process the iSCSI pdus. The application never blocks because it
> > is able to
> > > pace down
> > > the flow coming from the initiator with the TCP window and the command
> > > flow control (MaxCmdRN).
> > >
> > > Regards,
> > >
> > > Pierre