Re: Zero-copy TCP stacks (Was: Avoiding deadlock in iSCSI)

[Randall Stewart <rrs@cisco.com>]

Stephen:

comments in line... (even though I too am not a expert :)

Stephen Byan wrote:

> I may be merely increasing the noise floor; if so, I beg forgiveness in
> advance...
>
> Randall R. Stewart [mailto:randall@stewart.chicago.il.us] wrote:
> > Prasenjit Sarkar/Almaden/IBM wrote:
> > > BTW, zero-copy TCP/IP stacks have a lot of caveats (e.g.
> > memory alignment
> > > etc) which
> > > is why they have never made it to any operating system.
> > There are rough
> > > implementations
> > > in Solaris, BSD and Linux but none of them are particularly
> > close to being
> > > robust
> > > (I have tried them all).
> >
> > Prasenjit:
> >
> > I have used a zero-copy TCP/IP stack (if I recall correctly)
> > in VxWorks
> > or was it VRTX... I will grant you these are NOT *NiX systems and
> > are just recently beginning to become PosiX compliant... but I
> > seem to remember the zero-copy symenatics and I don't think there
> > was a memory alignment requriement.. it has been a while so I
> > may be mis-remembering.. :)
>
> I'm no TCP expert, but I think the difference between Randall's experience
> and Prasenjit's experience relates to the extent of the memory protection
> and security guarantees offered by the operating system.
>
> Embedded OS's such as VxWorks and VRTX do not provide separate memory
> address spaces for each task (aka process). Consequently there's no security
> problem in handing a pointer to an arbitrary memory buffer to a task - it
> can already see all of memory anyway.
>
> "Real OS's (TM)" do provide separate memory address spaces for each process
> (aka task). Consequently there is a security problem in handing a pointer to
> an arbitrary memory buffer to a task - the process could potentially view
> packets which happen to share the same memory page but which belong to other
> processes. To solve this problem, the TCP buffers must be aligned to a page
> boundary, and at most one buffer can be allocated per page.
>

I think you have it correct here. I have always considered VxWorks and VRTX
more of
a monitor than a O/S :)  ... Now one point here though is that at least one of
the
ends of the communication that is running I-SCSI may well be in this vain as
well i.e the disk side...

Oh, I also remember once building something for lynx-o/s .. (this DOES have the
MMU
turned on,so it meets my defintion of a O/S :>) that I was able to pass a page
of
speech to it via a system call and it would send it back to me when it was
done...

This was real specific to a particular application i.e. speech processing but I
do NOT see
that TCP  APIcould not also be "adjusted" to allow this. We have talked a
LOT on this
list about "special" interfaces to TCP. Not that I am any way advocating making
adjustments
to TCP :) I think this is a bad idea and I think the working group would do
better going with SCTP (of course I am biased )

R


>
> Regards,
> -Steve
>
> P.S. to the TCP experts - did I get this right?
>
> Steve Byan
> <stephen.byan@quantum.com>
> Design Engineer
> MS 1-3/E23
> 333 South Street
> Shrewsbury, MA 01545
> (508)770-3414
> fax: (508)770-2604