RE: iSCSI: No Framing

["Douglas Otis" <dotis@sanlight.net>]

Julian,

Implementing a scheme for doing direct data placement by means of an
application specific, undocumented or proprietary layer below TCP will
fracture the market.  This will result in an unacceptable outcome having
hardware created for each application, limited by various vendors making as
yet unknown claims, if implementing the DDP feature using TCP.  In the
future, should this feature become desired, iSCSI can adopt a generic method
that is possible through the use of SCTP.  An eventual adoption of SCTP
would also simplify much of the existing complexity found with the
multi-connection TCP standard now proposed for iSCSI.  Use of SCTP would
allow common hardware for many protocols desiring Direct Data Placement
without modification of the SCTP transport as it can deliver objects
unordered.  A shim providing the DDP feature could ensure objects are
disclosed in the order sent if desired, independent of the reception at the
shim while not modifying SCTP or adding layers beneath this transport as
would be required for TCP.

Only when constrained to conventional memory allocation, does DDP become
beneficial.  The concept of placing a layer beneath TCP that structures an
array to hold each individual packet based on application specific
structures is fundamentally flawed.  This presumes the layer beneath TCP
knows before hand the content of the byte stream prior to TCP composing the
stream.  This lower layer will also need to create exceptions for handling
duplicates, for placement information not aligned with the TCP segment, and
for application informational errors.  SCTP can easily avoid these
exceptional cases making such development less disruptive.  In addition, an
SCTP stack that does not offer DDP in hardware need not change the API
behavior to the application nor modify any layer functionality.

The ability of SCTP to support many different higher level protocols comes
from the capability of SCTP to deliver objects unordered.  TCP's restriction
on ordered delivery mandates that any direct data placement is done prior to
TCP through the addition of a new and application unique layer between IP
and TCP.  The saving caveat has been that disclosure to TCP of these packets
are sequential but the application interface to TCP will change as a result
of this modification in some undefined way to associate the steering
information with the desired buffers.  This change impacts the interface
between this new layer, as well as the application and TCP.  The desire to
keep this DDP scheme for TCP proprietary will ensure a fractured market in
terms of hardware, OS services, and perhaps interchange not to mention that
each application then also needs unique hardware.  The security risks
involved in creating this new and highly complex layer below TCP is yet
unknown as details are lacking.  In essence, an attempt to implement DDP by
creating a new layer beneath TCP in a proprietary fashion is in direct
competition with the far better practice of using SCTP to implement DDP.

Doug

> Mallikarjun,
>
> I doubt that FIM (or COWS) will fracture the market.
> Hardware and software vendors will gain experience in what it takes to use
> framing.
> a specialized DDP and that could be useful later.  The first generation
> although not imperiously needing any framing (I have proposed not
> less than
> 3 solutions!) will enable us to get a better second generation if we do
> something in this area.
>
> Julo
>
> I share the concern about iSCSI embracing a framing mechanism that is
> not a MUST implement.  For all the reasons that Jim pointed out, OTOH,
> I am not recommending a MUST framing either.  I suspect not many will
> implement framing if it's a MAY, so it appears that we're talking about
> a potential SHOULD.
>
> Given that SHOULD is a fairly strong requirement, one "significant
> justification"
> for not doing framing could be (even while the NIC *may be* on the
> expensive side) -
>  - has less design complexity since no OOO placement.
>  - can have quicker TTM since less design, testing, debugging etc.
>
> I think ultimately it boils down to: how many vendors would use a
> "significant
> justfication" to not implement a SHOULD-requirement?
>
> If that's a majority: let's say vendor X is implementing
> (whatever) framing
> for
> optimizing the memory requirements, it essentially means that X's product
> will
> perform poorly with (the majority) no-framing senders.  I don't think X
> would
> like that, nor the customer.  IOW, this situation doesn't seem to be
> long-lasting.
> OTOH, the situation of an almost equal number of  "framing" and
> "no-framing"
> products in the market (perhaps at different price points) could be
> unfortunately
> long-lasting....
>
> To summarize, it is a troubling prospect that a framing technique (if
> adopted as
> SHOULD) has the potential to somewhat fracture the market and in effect
> create "interoperability problems" (of performance sort) similar to that
> affect
> FC....
> --
> Mallikarjun
>
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions Organization
> Hewlett-Packard MS 5668
> Roseville CA 95747