Re: TCP framing

[Mike Brown <mbrown@cs.uml.edu>]

Hi all,

I am against using a tcp framing scheme for reasons previously stated by
Matt Wakely and John Hufferd.

I believe the assertion that iSCSI will be implmented in the kernel in the
UNIX environment is too simplistic a view.  I think robust implementations
will likely consist of some userspace/kernel hybrid.  (I am of the
opinion security associations using strong cryptography should not be
implemented in the kernel).  While this could also avoid an external API
change, requiring os vendors to change their existing TCP/IP stacks might
be too idealistic a view.

The insertion and deletion of markers from an implmentation prespective
would not be too painful an accomodation for software based solutions.
In fact, they would be simpler to implement since existing TCP/IP
stacks could be used.  John Hufferd's idea that "Markers should be 'Must
Implement', but 'Optional to use'" would enable software solutions to
ignore using them and still be efficient if their size is kept small
enough.

I am of the opinion that a new standard's probability of universal
acceptance is inversely proportional to the number of changes it requires
in other standards and existing implementations.  


On Tue, 22 May 2001, Somesh Gupta wrote:

>I think if the framing solution finds favor with the TCP "gods",
>it is a much better alternative than the markers (which I was
>a proponent of - but only because I did not believe that
>a framing mechanism would be approved).
>
>Even if the framing mechanism is not approved, an alternate
>mechanism which provides for alignment of iSCSI headers at
>regular intervals (at negotiate progression in TCP sequence
>#s e.g. 0x12345, 0x22345, 0x32345 and so on) in the TCP stream
>is a much better solution. Unfortunately this would require
>some change to the header format (for padding) which probably
>no one would like to change at this time.
>
>The assumption that people will implement some change as big
>as iSCSI will be to the OS (in an integrated way - not as
>a kludgy add-on for demos) without being able to make the
>relatively minor change required to the API seems to be not
>a very convincing argument.
>
>Furthermore, at least in the Unix environment, iSCSI will be
>implemented in the kernel, and the accomodation for framing
>won't require an external API change. And as Steph says,
>the framing proposal has Microsoft participation.
>
>The insertion and deletion of markers seems to be painful
>accomodation for software based iSCSI solutions. The Unix
>folks might want to speak to this, but the markers will
>probably require more of a kludge in the kernel (to make
>it efficient) as compared to the changes required for framing.
>
>Somesh
>
>> -----Original Message-----
>> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
>> Stephen Bailey
>> Sent: Monday, May 21, 2001 6:51 AM
>> To: ips@ece.cmu.edu
>> Subject: Re: TCP Framing (considered helpful?)
>> 
>> 
>> John,
>> 
>> > I think we must depend on Markers to insure that everything can 
>> operate at
>> > top speed, and at the lowest cost.
>> 
>> A key question is whether markers actually ensure that everything
>> operates at `top speed, and at the lowest cost'.
>> 
>> Matt thinks so.  I (and, presumably those who wrote the framing
>> document) think not.
>> 
>> My issue is not even with `lowest cost'.  I don't believe markers will
>> allow you to run at top speed.  Specifically:
>>   1) I doubt the feasibility of implementing the control required for
>>      an eddy buffer (where you store data you can't place) at 10G.
>>      Admittedly, the validity of this claim can't really be assessed
>>      without actually working the implementation, so for 99% of the
>>      list participants (myself included) this is a `yes it is, no it
>>      isn't' point.
>>   2) an eddy buffer solution requires some substantial speed-up in
>>      both the NIC data path, and MOST IMPORTANTLY: the host bus.  In
>>      order to unload the eddy buffer while still handling incoming
>>      traffic at line rate, clearly the host bus bandwidth must be >
>>      line rate.  
>> 
>> I know of at least one general purpose framed solution operating at
>> 10G which has been available for >3 years (SGI's GSN/ST/XIO NIC).  I'm
>> sure there are others.
>> 
>> I can't imagine there's any argument that a framed solution would be
>> voted `most likely to run fast and be cheap'.  Every storage network
>> and cluster interconnect has been designed that way since antiquity.
>> 
>> The key tradeoff involves the OS vendors, and I'm wondering why we're
>> speaking for them.  The question IS, how much more work is it to
>> introduce TCP framing over and above what is required to insert iSCSI
>> into their network framework.  My experience from writing NIC and
>> storage drivers for many commercial UNIX-family OSes is:
>>   1) it's an easy and well defined process to insert a new SCSI
>>      transport driver into the SCSI stack.
>>   2) it's hard and poorly defined process to insert ANYTHING into the
>>      network stack.
>> 
>> Networking has historically been a user-mode activity.  Architected
>> services are only provided to user mode programs.  Kernel clients have
>> been few and far between and so are handled on a case-by-case basis.
>> For example NFS.  Every OS has hacks to make NFS run fast, but they
>> are not stable interfaces for general purpose use.
>> 
>> Even Solaris' SysV-derived STREAMS stack, which is intended precisely
>> to provide flexible, crisp interfaces for kernel network clients, does
>> not document the relevant (IP stack) intermodule interfaces.
>> 
>> I know that there are more and more kernel network clients, but they
>> are coming either on fluid platforms (e.g. linux), in which case the
>> argument of `it'll take too long to get OS support' doesn't apply, or
>> they are vendor-supplied, in which case a performance iSCSI solution
>> in ANY form may take a while, and the choice of framing or markers
>> isn't going to make a difference.
>> 
>> I can't say squat about the architecture of Winsock, but the fact that
>> there is a Microsoft author of the framing proposal who seems very
>> serious about supporting framing and RDMA as quickly as possible
>> suggests that framing support should be available on Windows very
>> soon.
>> 
>> Steph
>
>_________________________________________________________
>Do You Yahoo!?
>Get your free @yahoo.com address at http://mail.yahoo.com
>


-Michael F. Brown, UMass Lowell Computer Science

phone:  (978) 934-5354
email:  mbrown@cs.uml.edu


"In theory, there is no difference between theory and practice,
 but in practice, there is."       - Jan L.A. van de Snepscheut