RE: iSCSI: Flow Control

["Douglas Otis" <dotis@sanlight.net>]

Joshua,

If the WAN is 15k miles at 1G-bit, then in-flight data would be 15M bytes.
Unless you expect to re-read data if lost, then at least twice and perhaps 3
times this amount would be saved for retransmission at 45 MB.  I assume
construction of the data being such it would be expensive to toss
unacknowledged data.  With this distance, the response to a lost packet may
be greater than 1/4 of a second and so while the wait to recover from the
lost packet to continue receiving at the far end, a store of 30 MB using a
SACK option would be held in lieu of the missing segment. In general, you
would need about 75 MB of buffer to handle a relatively clean but distant
WAN.  If the WAN is lossy, then the data-rate would suffer to the point of
needing less.  If you wish to allow full rate, then your internal buffers
past the TCP buffers should accommodate the sudden 30 MB surge from newly
recovered segments (about 3 seconds worth of FC data from this 1/4 second
gap).  If unable to keep up, at least 30 MB of delay exists before reducing
the flow should something change unexpectedly.  If this is at 10G-bit,
roughly multiply by ten.  Even a credit based system will still require much
the same in communication buffers but the FIFO buffers may be kept a bit
leaner however.  Credit helps most on the MAN.

Doug

> Mike,
>
> >It really comes down to advertising a byte window which can be
> sliced into
> >commands and data.  The credits and operations being tracked do
> not require
>
> >complex implementations to track or exchange.  Why can't one use
> / leverage
>
> >the same sliding window type of buffer advertisement that TCP
> uses and the
> >sender who is initiating the command will know whether it can send the
> >command with immediate data or must resort to a command with the remote
> >fetching the data?
> >
>
> There is an issue I'd like to bring up with using TCP to flow control
> the commands.  I don't know how serious it is, but I would like to
> point this issue out for anyone knowledgeable to comment.
>
> As was discussed earlier, TCP may need to buffer 16MB or more of
> data in order to work at gigabit speeds and 100+ms of latency.  With
> this much data being buffered in the initiator's transmit buffer,
> how does this affect the initiator's ability to send task management
> functions?  I understand most task management functions involve
> reset or reboot type of operations, but can these messages afford
> to wait behind a large queue of data in the TCP buffer?
>
> The alternative would be a credit-based command flow control mechanism,
> allowing the initiator to put a TMF at the head of the queue of data
> to be transmitted to the target.
>
> Josh
>