RE: iSCSI: No Framing

["Prasenjit Sarkar" <psarkar@almaden.ibm.com>]

Hi Pat,

I had a few questions about your 10 Gbps example. If we assume that this is
a single TCP stream, I would like to know which application in the near and
forseeable future can sustain a 10 Gbps data rate for a reasonable amount
of time? Or if they are multiple TCP streams, are you assuming packet drops
in all streams simultaneously?

Thanks,


   Prasenjit Sarkar
   Research Staff Member
   IBM Almaden Research
   San Jose



                                                                                                           
                      "THALER,PAT                                                                          
                      (A-Roseville,ex1)        To:       "WENDT,JIM (HP-Roseville,ex1)"                    
                      "                         <jim_wendt@hp.com>, ips@ece.cmu.edu                        
                      <pat_thaler@agile        cc:                                                         
                      nt.com>                  Subject:  RE: iSCSI: No Framing                             
                      Sent by:                                                                             
                      owner-ips@ece.cmu                                                                    
                      .edu                                                                                 
                                                                                                           
                                                                                                           
                      02/03/2002 03:01                                                                     
                      PM                                                                                   
                                                                                                           
                                                                                                           



Jim,

In answer to your questions:

Agilent is planning to implement framing. We view both FIM and
COWS as having about the same utility so we would implement
whichever one went into the standard.

A smoothing buffer on the chip is feasible wrt your point 2.
There are some configurations that would use external memory.

Also, one concern is that the very situation where one would
need large window size for efficiency - high bandwidth long
distance communication - is where out of order receipt becomes
increasingly likely so the amount of memory for this situation
could push up cost to an excessive extent.

Reducing the amount of traffic that has to be shunted to an
external memory affects the bandwidth that needs to be provided
for that memory. If we can handle most PDUs internally then the
external memory doesn't have to be as wide and as fast. For
instance, if a drop means that the iSCSI PDU with the drop
is put into external memory then that takes less memory bandwidth
than if a drop means the whole data stream for that session will
be going into the buffer until the hole is filled.

This decision also effects latency after a drop and required
backplane bandwidth.

Let's say one is on a 10 Gbit/s extreme long distance WAN
connection and that there is a drop. If the round trip delay
for getting the whole filled is 100 ms, then without framing,
one could have about 1 Gbit stored in the local buffer memory
by the time the hole is filled. One will have to process all
of this through iSCSI and across the backplane before one can
deal with the new traffic coming in. If the backplane data rate
is closely sized to the external data rate, an extra 100 ms latency
has just been inserted into the path until traffic slacks off.
(Congestion control might mitigate this to the extent that one
is talking to just one source, but with multiple sessions, one
will still have to keep up.) To get the buffer emptied to make
space for further drops and to get the latency back down, one
will have to size the backplane bandwidth wider and have the
iSCSI engine process at a higher data rate.

With FIM or COWS, one can end the incident with much less data
in the buffer as one processes PDUs after the hole while waiting
for it to be filled. Then, when the hole is filled one just has
to process a small amount of data to catch up.

Regards,
Pat Thaler

-----Original Message-----
From: WENDT,JIM (HP-Roseville,ex1) [mailto:jim_wendt@hp.com]
Sent: Tuesday, January 29, 2002 10:47 PM
To: ips@ece.cmu.edu
Subject: iSCSI: No Framing



So, it would be good to hear from several iSCSI
NIC/chip implementors who:
- have or plan to implement FIM or COWS (or some
other framing mechanism) and take advantage of it to
minimize demands on on-NIC buffer memory
bandwidth/quantity.
- believe that all-buffers-on-chip solutions are
feasible and valid (wrt the points above, including
#2)
- can quantify the memory/pin/power/space cost
savings for all-buffers-on-chip solutions

Jim