Re: iSCSI: draft 7: remove S bit and Status field from the Data-in PD

[Stephen Bailey <steph@cs.uchicago.edu>]

> That will negatively afftect receiver performance.
                              ^ software
                                       ^s running on unmodified TCP layers

I even argue with the claim that it has to negatively affect the
performance running against an unmodified TCP layer.

In the worst possible case, where your receiver is implemented as a
user-mode program, AND, you implement it as:

  for (;;) {
    if ((c = read(f, bhs, sizeof(*bhs)) < 0) { /* error */ }
    restLen = bhs->ahsLen + bhs->dataLen;
    if ((c = read(f, rest, restLen) < 0) { /* error */ }
    /* process PDU */
    . . .
  }

or worse, with 3 reads() (or even 1 read()), YES, having an extra PDU
slows you down.

However, this is not a high performance way to implement the
protocol.  In other words, you're not really trying if you do this, so
we shouldn't improve the performance for implementations that aren't
trying.  The performance way to implement this is something like:

  for (;;) {
    if (outOfBufferOrJustWantToTryForMore()) {
      fillBuffer(b);  /* nonblocking read unless outOfBuffer */
    }
    bhs = readFromBuffer(b, sizeof(*bhs));
    restLen = bhs->ahsLen + bhs->dataLen;
    rest = readFromBuffer(b, restLen);
    /* process PDU */
  }

With a structure LIKE (I know the above structure is `buggy') this,
your `reads' are very lightweight.  Typically, you'll get an entire
TCP segment (or possibly more) per real read, and pulling an arbitrary
number of PDUs from that segment costs about the same as pulling a
single PDU from that segment.

Completely independently of whether we agglutinate control with data,
if `pull from socket' (typically system) calls are expensive,
performance will stink if you use the `get next data == pull from
socket' model.  If pull from socket calls are cheap, the agglutination
doesn't matter either.

Steph