Re: iSCSI: Markers

["Julian Satran" <Julian_Satran@il.ibm.com>]

One comment in text - julo

Paul Koning <pkoning@equallogic.com> Sent by: owner-ips@ece.cmu.edu 08-01-02 18:59

To:        John Hufferd/San Jose/IBM@IBMUS         cc:        Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu         Subject:        Re: iSCSI: Markers

>>>>> "John" == John Hufferd <hufferd@us.ibm.com> writes:

 John> OK, Julian, I buy that argument.  I think that you have made
 John> the point that the value of Markers is Greater then I was
 John> giving credit for, in iSCSI/TOE integrated HBAs that are
 John> operating in CRC mode.  That is great, since I am afraid we
 John> will not see the Framing stuff for some time.

 John> So the questions are which form of markers is the best.

 John> My vote comes down on the side of Fixed Interval Markers
 John> (FIM). For the following reasons:

 John> On the SW sending side, FIM works with low overhead whether CRC
 John> is being used or not.  COWS, is a dramatic overhead unless the
 John> SW is performing CRC computation anyway.

 John> The COWS has a built in conflict with the Pipeline HW HBA, that
 John> has two different needs regarding the direction of the Pointers
 John> when talking to another partner HW HBA that uses a Pipeline
 John> approach.

I think you are making general assumptions about what is cheap in
software that may be valid for SOME software implementations but are
certainly not valid for others.

In particular, FIM is NOT necessarily low overhead.  Scatter/gather is
a reasonably cheap mechanis IF it is available for this job.  (Note
that it is not all that cheap: adding a bunch of extra descriptor
fetches to the I/O device's job is extra overhead, the scale of which
depends on the hardware.

The problem is that scatter/gather may not be available at all.  If it
is available in principle, it may have restrictions that clash with
what FIM needs.  (For example, a particular implementation may support
breaking buffers up in many pieces, but all breaks must be on cache
line boundaries.  For a cache line size of 32, clearly you cannot use
scatter/gather in that implementation to do FIM.)

So FIM may force a full buffer copy, which is a major hit.

As for COWS, it's clearly nasty.  Even if you're doing CRC, it adds a
significant increment to the processing cost.

In conclusion, I am opposed to mandating markers.  Adding such a large
increment in processing overhead doesn't make sense to me, given that
I don't see how it can be expected to save all that much memory in
implementations that use it.  (As far as I can see, the gain doesn't
come anywhere near a factor of two.  Is there any analysis available
that discusses this in any detail -- covering not just the specific
buffering needs of the reassembly side but also the other memory
requirements in a node?  It seems to me a node needs at least as much
memory in the other direction -- quite possibly more if it's the
storage end, given that a lot of the traffic is reads.)

+++ the gain is not a factor of two - it is a not a constant - it is rather
proportional to the bandwidth delay product.
The rational is simple - once a header is lost without any framing mechanism 
the receiver has to accumulate data until the missing piece is recovered.
This and other lists had long discussion threads on the subject - and if you care about it you may want to go over the archives.
As David has said it is generaly agreed that this is not an issue at 1Gbs yet - but is a also agreed that this is a major issue at 10 Gb/s.
That is why some of us think that a stronger statement about some form of framing is required.

+++

                 paul