RE: iSCSI: No Framing

["Douglas Otis" <dotis@sanlight.net>]

Pat,

The macroscopic behavior of the TCP congestion avoidance algorithm by
Mathis, Semke, Mahdavi & Ott in Computer Communication Review, July 97,
provides a simple formula for an upper bound on the TCP transfer rate:

Rate <= (MSS/RTT)*(1/sqrt{p})

A .004% loss rate with 1460 MSS, and 100 ms RTT would be less than 2.3 MB/s
or 18 Mb/s.

In your senario, there would be a requirement to buffer 230 KB.  The system
will also see a reduction in send traffic accommodating receiver side
recovery following 157 packets after a loss seen about every 25,000 packets.
There should be little concern about dealing with a single connection across
great distances working at phenomenal data rates unless packet loss becomes
diminishingly small.  The cost for such a connection will also make related
memory concerns diminishingly small as well.  Reliability and security
should easily override benefits seen by placing the application below TCP as
advocated.

If you wish to implement framing, then may I suggest such a scheme is
possible using SCTP without placing the application beneath the transport.
SCTP offers may other benefits needed in any mission critical application.
You would save your customer grief by not introducing obsolete equipment as
there is already this emerging standard that provides many improvements over
such a kludge.

Doug

> Jim,
>
> In answer to your questions:
>
> Agilent is planning to implement framing. We view both FIM and
> COWS as having about the same utility so we would implement
> whichever one went into the standard.
>
> A smoothing buffer on the chip is feasible wrt your point 2.
> There are some configurations that would use external memory.
>
> Also, one concern is that the very situation where one would
> need large window size for efficiency - high bandwidth long
> distance communication - is where out of order receipt becomes
> increasingly likely so the amount of memory for this situation
> could push up cost to an excessive extent.
>
> Reducing the amount of traffic that has to be shunted to an
> external memory affects the bandwidth that needs to be provided
> for that memory. If we can handle most PDUs internally then the
> external memory doesn't have to be as wide and as fast. For
> instance, if a drop means that the iSCSI PDU with the drop
> is put into external memory then that takes less memory bandwidth
> than if a drop means the whole data stream for that session will
> be going into the buffer until the hole is filled.
>
> This decision also effects latency after a drop and required
> backplane bandwidth.
>
> Let's say one is on a 10 Gbit/s extreme long distance WAN
> connection and that there is a drop. If the round trip delay
> for getting the whole filled is 100 ms, then without framing,
> one could have about 1 Gbit stored in the local buffer memory
> by the time the hole is filled. One will have to process all
> of this through iSCSI and across the backplane before one can
> deal with the new traffic coming in. If the backplane data rate
> is closely sized to the external data rate, an extra 100 ms latency
> has just been inserted into the path until traffic slacks off.
> (Congestion control might mitigate this to the extent that one
> is talking to just one source, but with multiple sessions, one
> will still have to keep up.) To get the buffer emptied to make
> space for further drops and to get the latency back down, one
> will have to size the backplane bandwidth wider and have the
> iSCSI engine process at a higher data rate.
>
> With FIM or COWS, one can end the incident with much less data
> in the buffer as one processes PDUs after the hole while waiting
> for it to be filled. Then, when the hole is filled one just has
> to process a small amount of data to catch up.
>
> Regards,
> Pat Thaler
>
> -----Original Message-----
> From: WENDT,JIM (HP-Roseville,ex1) [mailto:jim_wendt@hp.com]
> Sent: Tuesday, January 29, 2002 10:47 PM
> To: ips@ece.cmu.edu
> Subject: iSCSI: No Framing
>
>
>
> So, it would be good to hear from several iSCSI
> NIC/chip implementors who:
> - have or plan to implement FIM or COWS (or some
> other framing mechanism) and take advantage of it to
> minimize demands on on-NIC buffer memory
> bandwidth/quantity.
> - believe that all-buffers-on-chip solutions are
> feasible and valid (wrt the points above, including
> #2)
> - can quantify the memory/pin/power/space cost
> savings for all-buffers-on-chip solutions
>
> Jim
>