Re: iSCSI: Markers

[Paul Koning <pkoning@equallogic.com>]

>>>>> "John" == John Hufferd <hufferd@us.ibm.com> writes:

 John> OK, Julian, I buy that argument.  I think that you have made
 John> the point that the value of Markers is Greater then I was
 John> giving credit for, in iSCSI/TOE integrated HBAs that are
 John> operating in CRC mode.  That is great, since I am afraid we
 John> will not see the Framing stuff for some time.

 John> So the questions are which form of markers is the best.

 John> My vote comes down on the side of Fixed Interval Markers
 John> (FIM). For the following reasons:

 John> On the SW sending side, FIM works with low overhead whether CRC
 John> is being used or not.  COWS, is a dramatic overhead unless the
 John> SW is performing CRC computation anyway.

 John> The COWS has a built in conflict with the Pipeline HW HBA, that
 John> has two different needs regarding the direction of the Pointers
 John> when talking to another partner HW HBA that uses a Pipeline
 John> approach.

I think you are making general assumptions about what is cheap in
software that may be valid for SOME software implementations but are
certainly not valid for others.

In particular, FIM is NOT necessarily low overhead.  Scatter/gather is
a reasonably cheap mechanis IF it is available for this job.  (Note
that it is not all that cheap: adding a bunch of extra descriptor
fetches to the I/O device's job is extra overhead, the scale of which
depends on the hardware.

The problem is that scatter/gather may not be available at all.  If it
is available in principle, it may have restrictions that clash with
what FIM needs.  (For example, a particular implementation may support
breaking buffers up in many pieces, but all breaks must be on cache
line boundaries.  For a cache line size of 32, clearly you cannot use
scatter/gather in that implementation to do FIM.)

So FIM may force a full buffer copy, which is a major hit.

As for COWS, it's clearly nasty.  Even if you're doing CRC, it adds a
significant increment to the processing cost.

In conclusion, I am opposed to mandating markers.  Adding such a large
increment in processing overhead doesn't make sense to me, given that
I don't see how it can be expected to save all that much memory in
implementations that use it.  (As far as I can see, the gain doesn't
come anywhere near a factor of two.  Is there any analysis available
that discusses this in any detail -- covering not just the specific
buffering needs of the reassembly side but also the other memory
requirements in a node?  It seems to me a node needs at least as much
memory in the other direction -- quite possibly more if it's the
storage end, given that a lot of the traffic is reads.)

	paul