RE: iSCSI: Framing Steps

["Douglas Otis" <dotis@sanlight.net>]

David,

> > > TCP is vulnerable to packet injection based on guessing or observing
> > > the initial sequence number in the absence of fixed-interval-markers
> > > (FIM).  The well-known connection hijack attack is an example.  An
> > > attacker can inject arbitrary data into a connection without FIM and
> > > cause TCP to misbehave as a result.  FIM does not make this any
> > > easier or change the ways in which TCP would misbehave when attacked
> > > in this fashion.
> >
> > FIM enables *processing* of packets out of sequence from the
> > initial send. The intent of FIM is to make such processing
> > possible and that *does* enable a greater opportunity for
> > attack especially if the measure is predictable within future
> > headers.
>
> Nonsense.  TCP processes packets in the same order (i.e., the order
> received) with or without FIM.  TCP would not need or use FIM to
> find TCP headers, hence FIM provides no additional opportunity to
> inject a fake TCP header. Every TCP receiver already has to perform
> processing on packets "out of sequence from the initial send" due to
> drops and reordering that may happen in the network; this processing
> may include queuing to wait for missing segments to arrive.  FIM
> changes only the memory locations of the packets that are processed,
> not the order of their processing.

The concerns are not with TCP but with this process running below TCP.  This
would be a stateful process that could indeed be confused by an
inappropriate reliance on a marker.  This change in location of content
arrival would be a complex process that includes application specific
structure handling.  This "only changes memory locations of" is an
application specific process.  In addition, it is not the packet location
that is being changed, but the content of the packet or it would be of
little value.  There are already zero copy schemes that do not rely on
markers to achieve appropriate packet placement.  The iSCSI appendix clearly
indicates the value of these markers is to locate iSCSI specific structures
within the TCP byte stream should there be dropped packets.  FIM is to allow
processing of these structures in the face of previously dropped packets.

> > > FIM does not entail any changes to TCP.  TCP does not care about
> > > where incoming data is placed in memory - it cares about the order
> > > in which data is processed by TCP and delivered to the application.
> >
> > A process that examines application specific structures at the
> > packet (IP) level, to then move portions of the expected TCP byte
> > stream into user space based on application references matching
> > these structures, is *not* a modification to TCP provided there
> > is a *sequential* order of delivery?
>
> That is correct.  RFC 793 specifically allows sharing of storage between
> TCP and the user provided that delivery order is maintained at the
> interface, and even gives an example of a data structure that could be
> used (see below).
>
> > Clearly placing a portion of the application below TCP is a modification
> > to TCP and network layering.  Are you referring to a means at the
> > TCP API to hide this activity? Are you saying that the TCP API does not
> > change? How is this done without changing TCP?
>
> The only change TCP sees is that the location of packets in memory is
> different.  As far as TCP is concerned, this is equivalent to replacing
> the operating system's memory allocator, including possible modifications
> to TCP code to use the new allocator (e.g., the BSD and streams
> implementations of TCP use rather different memory allocators).  As for
> the "TCP API" ...
>
> > The application specific processing below this modified TCP transport is
> > to de-encapsulate the iSCSI payloads into user space. iSCSI user space
> > may not be satisfied in the sequence they were established.
>
> Both the de-encapsulation into user space and the satisfaction of receive
> operations out of order are allowed by RFC 793.  Here's the text from
> pp. 48-49 (definition of Receive in the nominal TCP/user interface):
>
>  To distinguish among several outstanding RECEIVEs and to take
>  care of the case that a buffer is not completely filled, the
>  return code is accompanied by both a buffer pointer and a byte
>  count indicating the actual length of the data received.
>
>  Alternative implementations of RECEIVE might have the TCP
>  allocate buffer storage, or the TCP might share a ring buffer
>  with the user.
>
> The first paragraph appears to allow completion of RECEIVEs out of order
> and the second definitely allows sharing of TCP's data storage with
> "the user".  In practice, most TCP implementations complete RECEIVEs in
> order for simplicity and to avoid surprises at the socket API ...
>
> Picking up the "TCP API" issue from above, no modifications are needed
> to the nominal API defined in RFC 793.  The intended use of FIM is
> not possible through the socket API (not an IETF standard, FWIW), but
> there is no proposal to expose FIM through that API - it's intended
> for combined implementations of TCP, FIM, and iSCSI, and hence would
> only be specified as part of iSCSI.

TCP buffer treatment does not begin to compare to the complex process that
would be employed to handle the dispatching of the TCP byte stream
anticipated by iSCSI.  The TCP byte stream is commonly stored out of
sequence from the send but this process does not depend on the data.  This
iSCSI structure marker process goes well beyond a simple buffer assignment,
is highly stateful, and prone to errors where recovery would be difficult.
Indeed, TCP allows flexibility within the API with respect to buffer
handling, but I think it would be a mistake to assume this flexibility
allows modes of operation dependent upon the recognition and handling of
application specific structures.  iSCSI wishes to run a portion of the
application in front of TCP and to do this on a byte by byte basis.  TCP can
not "deliver" data to the application out of sequence so logically there
should be no need for such markers if TCP were allowed to run first.  iSCSI
structure recognition, synchronization, validation would be run prior to the
normal TCP delivery and is the reasoning for the use of these PDU markers.

> > The complexity involved will require modification of TCP to support
> > this buffer handling and application specific structure processing.
> > In the case of iSCSI, the content of the byte stream is not known
> > before hand.  This activity below TCP places and splices sections
> > of the TCP byte stream into designated iSCSI user space.  Clearly
> > there are many questions one may have with respect to what and when
> > TCP validates and delivers.
>
> Modifications to TCP implementations are definitely involved.  The
> answers to the questions can generally be found in RFC 793 and 1122
> because FIM makes no changes to "what and when TCP validates and
> delivers".
> For example ...
>
> > TCP could easily reject a packet based on the byte sequence within
> > headers that was not rejected by this "below the transport process"
> > that would be unable to see the header continuum.
>
> Since FIM performs no protocol processing (it only allocates
> memory for the incoming packet), TCP rejects the packet in the usual
> fashion, causing it never to be delivered to iSCSI.  Even with FIM,
> data is not delivered to iSCSI until TCP delivers it in sequence,
> hence the rejected packet has no effect on iSCSI.

The goal of the marker has nothing to do with what may be described as zero
copy TCP.  FIM is to locate iSCSI PDUs as defined in the iSCSI draft.  The
driving motivation behind this effort is to allow a direct placement of the
data payload portion of the iSCSI structures into the associated user space
referenced by the tags within these structures.  In other words, a type of
direct data placement.  The "deliveries" to TCP from this process would be
an interesting topic.  Deliveries in turn from TCP back into the application
would need to know if this "below the transport" process had finished
successfully or had even been run.  This is not TCP.

> > This added state below the TCP transport provides opportunity
> > for attack or in some implementations, poor behavior.
>
> Any work on implementations has the potential to introduce bugs; in
> this respect FIM is no different from any other implementation work.

This will impact many applications that will also be using the same stack.
Ensuring sound concepts would be to the benefit of IPS as well as other
working groups.  Should the implementation of FIM create substantial
security risks by nature of its design, then such design should be stopped.

> > > The intended usage of FIM affects only the placement of incoming
> > > data in memory and hence does not change TCP.
> >
> > Placement of application specific payloads into regions matched against
> > application specific structure content prior to being handled by the TCP
> > layer is a change to TCP.  TCP is not able to manipulate buffers in this
> > manner without extensive modification.  Such modification is quite
> > possible however, and all the details should be explored before the use
> > of this scheme is placed into a standards draft however.
>
> I would agree insofar as changes to TCP implementation code are
> required to use FIM in the fashion intended.  I maintain that FIM does
> not change the TCP protocol as specified in RFC 793 and 1122.  The memory
> and buffer allocation aspects of TCP have not been standardized
> by the IETF for very good reasons.

Please refer to my prior statement.

> > My concern remains that the IPS wg is NOT the place to discuss these
> > concerns as that is not their expertise nor their charter.
>
> I have no problem with bringing this sort of issue to tsvwg, as long
> as it's done in an honest fashion.  Specifically, given Doug's past
> insistence that the IPS wg not discuss TCP changes, he should not
> complain that the IPS wg has failed to adequately discuss TCP changes
> in the TSVWG TCP ULP framing draft.
>
> Finally, on requirements:
>
> > You have also asked the group to drive the framing issue to conclusion
> > but you failed to indicate that such discussion should not take place
> > within the IPS.
>
> iSCSI requirements for use of framing mechanisms are proper for discussion
> in IPS.  That requirements discussion will be driven to a conclusion in
> IPS because that conclusion will be reflected in the iSCSI specification.
>
> > Ensure that an error is not made where everyone expects such
> > modifications to TCP even if done unofficially by the IPS.
>
> Just in case anyone else is confused - TCP ULP framing is a tsvwg draft,
> and FIM is currently in the iSCSI draft because it does not modify the
> TCP protocol.

I think it would be a reasonable statement that I do not agree with your
assessment on the impact to TCP created by the optional markers used in
iSCSI.  The IPS group represents talented people highly skilled in the field
of storage but could benefit from the advice of the community at large with
respect to this topic.  As I do feel an implementation as intended of the
iSCSI marker scheme would be a modification to TCP, I also felt strongly
that this topic get raised within the tsvwg work group.  I would like to
thank you for your response.

Doug

> Thanks,
> --David
>
> ---------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 42 South St., Hopkinton, MA  01748
> +1 (508) 249-6449 *NEW*      FAX: +1 (508) 497-8500
> black_david@emc.com         Cell: +1 (978) 394-7754
> ---------------------------------------------------
>