RE: RDMA over TCP (Was Re: VI (Was: Avoiding deadlock in iSCSI))

["Douglas Otis" <dotis@sanlight.net>]

Jim,

<snip>
>                          RDMA / TCP
> 1.  Abstract
>
> This document describes a format for encapsulating RDMA (remote direct
> memory access) information within a TCP data stream.  No changes or
> modification to TCP of any sort are required.

You suggest a frame aligned formatting scheme is possible using an
unmodified TCP.

> This is not intended to
> be a protocol, but rather a common format that may be shared by
> multiple client protocols, for instance VI/TCP and iSCSI.  By using a
> common format it is hoped that design of NICs supporting these multiple
> protocols can be simplified.
>
> Sufficient information is included in the RDMA message format to allow
> determination of the protocol message units, as will as the ability to
> process an incoming RDMA request even if previous packets are missing
> and awaiting retransmission.  In addition a CRC-32 is included in each
> segment to enhance the checksum coverage included in TCP.

By allowing out of sequence delivery of TCP, are you opening the window for
spoofing?  One need not guess the sequence, just get in front of it a bit.
After that, you will never recover state.  Do you expect the RDMA option to
be universally supported, and if not?

Do you see this level of CRC done in software?

> 2.  Overview
>
> Data transfers consist of a sequence of messages.  Each message is of
> one of four types: Send, RDMA_write, RDMA_Read_Request, and
> RDMA_Read_Response.  The maximum size of a message is approximately
> 2^32.  Each message is divided into one or more segments.  It is
> RECOMMENDED that each TCP segment contain exactly one RDMA segment.

How do you achieve this recommendation?

Sending 4GB exchanges, you still have queue blocking if not head of queue
blocking.

> The receive end of the connection cannot assume any alignment between
> the RDMA segments and TCP segments, however a receiver SHOULD optimize
> performance for the case where each TCP segment contains exactly one
> RDMA segment.

Can one suggest frame alignment within TCP?  Have you investigated the use
of SCTP?  SCTP does solve all the related problems without violating
protocol or protocol API.

Doug