iSCSI: No Framing

["WENDT,JIM (HP-Roseville,ex1)" <jim_wendt@hp.com>]

Perhaps we should discuss the possibility of not 
specifying any framing mechanism (FIM or COWS) in the 
first version of iSCSI.

The arguments for not specifying framing include 
(there may be others as well):
1) The brute force approach of putting TCP receiver 
reassembly memory on the iSCSI NIC will work for both 
1Gbps and 10Gbps. Cost is incurred for memory chips, 
ASIC pins, power, and board space. But, it is a 
feasible approach.

2) I/O bus latency (or bandwidth limitations) could 
mandate inbound buffering (as a 'smoothing' buffer) 
from the iSCSI NIC to the host system. If this buffer 
memory is large enough to have to be off-chip, then 
it will require some minimum number of memory chips 
to provide the necessary bandwidth, and the 
memory/pins/power/space costs will be incurred 
anyway.

3) Very large receive buffering on the NIC is only 
required for high-bandwidth links traversing large 
geographic distances (which may not be the 
predominate case). These specialized implementations 
can cost more (e.g. more memory/power/pins/etc) and 
customers will likely be willing to pay accordingly.

4) Many NICs will likely aspire to be combo 
iSCSI+TOE+Ethernet NICs allowing the host system to 
use a single Ethernet port for all of its network 
communications. The TOE function on this combo NIC 
will already require TCP receive buffering and off-
chip buffer memory to support the existing sockets 
interface receive model.  More importantly, to allow 
senders to assume ownership of the buffer upon return 
from a socket send call, the TOE NIC would need to 
copy application send buffers into NIC memory as 
well.

5) The framing/marker mechanism will be an iSCSI-only 
solution. It is quite possible that the problem will 
be solved via a different, and hopefully more 
general, mechanism for other ULPs.

6) Both FIM and COWS are poor solutions for software 
implementation. COWS requires, at a minimum, the 
sender to read every word in the buffer. FIM either 
imposes additional sender gather processing (iovecs) 
or requires a data buffer copy (on systems that don't 
support HW gather on sends).

7) Unless all senders thoroughly test framing/markers 
now (i.e. unless the framing method is a MUST 
implement), there is potential for future 
interoperability problems as framing/marker receivers 
are deployed in the future.

8) Neither FIM nor COWS is an elegant solution. Are 
we comfortable with the legacy we are creating under 
the umbrella of state-of-the-art IP networked 
storage?

9) Is it essential to build in forward compatibility 
now, or will there be a different solution in the 
10Gbps or 40Gbps timeframe - perhaps involving iSCSI-
2?  Will it be reasonable to update or bridge initial 
1Gbps deployments?


So, it would be good to hear from several iSCSI 
NIC/chip implementors who:
- have or plan to implement FIM or COWS (or some 
other framing mechanism) and take advantage of it to 
minimize demands on on-NIC buffer memory 
bandwidth/quantity.
- believe that all-buffers-on-chip solutions are 
feasible and valid (wrt the points above, including 
#2)
- can quantify the memory/pin/power/space cost 
savings for all-buffers-on-chip solutions

Jim