Re: New List: rdma@cisco.com: to discuss RDMA

[Stephen Bailey <steph@cs.uchicago.edu>]

> We had a similar debate within InfiniBand on just this subject 
> and it really came down to implementation specific requirements.

I'm not completely up on IB, but what you are describing doesn't
appear to be precisely analogous to the negotiable use of RDMA in
iSCSI.

It seems like you are talking about the subsetting of the capabilities
of a general RDMA mechanism for application specific purposes.  For
example, it makes no sense to require an implementation of RDMA READ
on an endpoint is only ever a data sink.  Similarly, whether any
particular data is transferred as SEND or WRITE also depends upon a
choice in the design of the ULP.

A general RDMA protocol provides a pallete of options that ULPs can
use to accomplish their required data transfer.  Asymmetry of
requirements is an implication asymmetry of ULP operation.  `Host'
(general) endpoints are required to implement the whole pallete
because it allows ULPs to be designed using any element of the pallete
freely as appropriate.  Target (application specific) endpoints need
only implement the operations required by the ULP, or even just a
subset of the ULP they chose implement.  They are specialized
precisely for their function.

Whether iSCSI uses RDMA or an iSCSI specific tagged transfer model
doesn't matter from the standpoint of making iSCSI work.  The problem
with using an iSCSI specific transfer model is that you can't use it
for anything else.  A general RDMA mechanism can be used for other
purposes.

> I don't view RDMA or any of these implementations as panacea
> solutions for this overall resource usage problem.

Was this on the table?  Somebody mentioned in an earlier message that
SAM-2 requires that the buffers be pinned.  However, any transfer
mechanism that has bidirectional buffer flow control probably has the
potential to support a `rolling pin'.

> The benefits of RDMA are associated with the ability to reduce the
> amount of buffering required within a solution at the various points
> along the way and deliver zero-processor copy placement and control.
> These can be implemented in other ways as others have pointed out
> and thus the cost / benefit needs will vary.

I don't believe that a zero copy placement implemented with a the
tagged transfer model is any more costly than one implemented with
RDMA.  If you are suggesting that the adapter must be thick in one
case and thin in the other, I don't see it.

I said in my previous message that using RDMA would make iSCSI
implementation easier.  I take it back.  It's not clear that this is
actually the case.  The real advantage of a general RDMA is that it
can bring the benefits traditionally associated with storage adapters
(zero copy, and low CPU overhead) to protocols other than SCSI.
However, it also seems like an advantage that a high performance iSCSI
on RDMA implementation can be built with no iSCSI specific hardware at
all.

> Let the market decide whether that one implementation should survive or 
> not.  Most likely it will die as others will have been smarter in their 
> designs and functional selection.

Exactly.  If the tagged transfer mode is well designed, and RDMA is
not required, nobody will do RDMA for iSCSI.

Whether to use RDMA or not is a choice that the iSCSI standard is
going to have to make.  Either way works.  However, I do not believe
there is a sensible way to allow for both possibilities.

Steph