Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc.

["David R. Cheriton" <cheriton@cisco.com>]

At 04:09 PM 2/27/00 +0000, Alan Cox wrote:
>> understand all the protocols that are amased over TCP. To do it in silicon
>> it will probably make sense for a subset.
>
>You dont want to do it in silicon. Forget doing all this in silicon. We have
>this funky stuff called software. The silicon needs no RDMA support to do
>sensible work in the API and the underlying OS are sensibly designed.
>

I think it would be useful to get more quantitative.  I like to
think all of us in this discussion are well aware that some level
of performance can be achieved in software.  I'm not seeing where
"sensibly designed" leads us WRT to OSs, because, if they arent now
by whatever definition you are using, it seems pretty academic, as
they say.
  Clearly, data is being received from hardware and software does not
get to touch it until it has been stored to some memory.  My 
assumption is that the storage system memory is arranged in fixed
size pages of disk/file pages.  Without hardware RDMA to the storage
level, I believe one requires an extra copy, from whatever the
hardware delivers to what the storage system expects.  Either
you use twice the bandwidth in the storage system memory system or
or else you have a separate memory system for the network, and
have software/processor power adequate to copy between at wire
speed (with all the associated support facilities for this processor.)
 Unless there is something wrong with this reasoning,
it seems like a cost issue of providing the above hardware resources
vs. providing a NIC chip that can RDMA.  

My guessitimate is that the software-only approach would be easily
10 times more expensive here at the higher speed rates, of 10 Gbps.
If there is serious doubt about the merits of real hardware support,
we should try to quantify costs further at these speed ranges, IMHO.

>> You can look at it as a simple way to enable the protocl stack and the
>> application to completely separate the protocol state machine (defined by
>> headers and/or trailers) from the payload.
>
>The two are tied together. You have to parse the TCP option stream to get
>the ident in the first place. You can't act on the RDMAID until you
>have checked the packet is syntactically valid and you've processed
>the options including handling the SACK data mixed in with it.
>
>It might also be fragmented of course.
What you need to do >>in the common case<< before processing the RDMA
option is relatively simple and a very small portion of the
overall protocol state machines, so I presume your comment is
asking for more careful wording? Or do you really disagree with the
fundamental point?

>
>
>> As for the vulnerability to attacks with a good size RDMAID and some
>> imagination you can get the same level of protection as with the TCP
>> sequence number (even a bit better because sequence numbers can be guessed
>> from context).
>
>The tcp sequence number protects against ordering errors not against DMAing
>crap into the wrong buffer.
>
>Different game, different cost if you lose.
>
It would help me to have a more careful definition of the types of
attacks you have in mind.  In an unsecure network with intruders,
presumably I can end up with bad data in the right buffer
or right data in the wrong buffer without using RDMA.
Do you view we have made things worse, and if so, how?
or are you objecting to us not making things better?


>Alan
>
David Cheriton