NFS Header/data parsing and RDMA

[Costa Sapuntzakis <csapuntz@cisco.com>]

Ok, so doing NFSv2/v3 header/data splitting is easy on an in-order
TCP stream because NFS has fixed-length trailers. Here's a little
technique:

1) Assume that all READ/WRITE transfers are powers of 2
2) Assume all RPCs larger than 4k are WRITE RPCs and all responses larger
than 4k are READ responses
3) Take the message size from the first 4 bytes of the RPC/TCP
encapsulation
4) Round the message size down to the nearest power of 2 (call this
quantity data_size)
5) The data is the last data_size bytes of the message. Put
the last data_size bytes in a separate aligned buffer.

Note, to do this with NFS/TCP, your NIC has to do some primitive
level of TCP processing (at least keep track of flows). It also
needs to understand RPC/TCP message boundaries.

Are there significantly simpler approaches than this? 

NFSv4 doesn't seem to have fixed length trailers and neither
does CIFS in all cases. And it looks like it will be costly to parse 
NFSv4 headers. 

RDMA still has the following features:

- Per-packet (Works with arbitrary out-of-order reception of TCP
segments)
- Fixed header that's generic across all protocols (NFSv4, v5, AFS,
DFS, CIFS, etc..) 
- No page flipping necessary on solicited transfers
- Message boundary bit (which is admittedly orthogonal to RDMA) allows
out-of-order processing on TCP receive buffer. Decreases parsing latency,
esp. in the face of packet drops.

The following measures should improve security/safety:

- NIC should ascertain that TCP segment is in receive window

- NIC needs to check that the RID is valid for a given TCP conn
  for safety/security reasons
  
  If the NIC does header/data splitting, it needs to keep track of
  per-flow information because most file block transfesr will span
  multiple TCP segments. So the NIC will probably have a notion of
  a TCP flow #.

  The brute force approach way to check RID validity is to use a CAM to
  map from (RID, TCP flow #) -> buffer address.

  If that's too expensive, then RIDs can be hashed and the flow # and
  buffer address stored in the bucket. The flow # is verified before
  using the buffer address.

-Costa