RE: iSCSI: Towards a more effective PDU format

[julian_satran@il.ibm.com]

Barry,

If you have no idea what follows BHS the you have to start by reading the
TLV first word or equivalent to.
The rest folllows. What we did with the current scheme is that we cut a
byte from the length (that was meant for data) and made out of it a
descriptor of the first TLV and the length the length part of it.
From here whatever follows BHS is known after BHS processing.  In most case
it is data - so that it is really simple - you read 48 bytes and you are
done (your fast path is really that fast and simple).
I am not sure that 48 bytes is that long and convoluted. The digest is
where some trouble arises.
If we use a single CRC/digest we have to keep reading while we don't know
yet if the data is good or bad.
The conventional TLV schemes have the TL at the beginnings of the block and
you read them then the Value.
Here again you use TL before knowing if it is good.

All the speculation about good or bad is however practically solve in
several attempts (you will get to a block with a good CRC (resynch).  I
suggested also adding some parity to the TL part to increase confidence in
it.

Regards,
Julo


Please respond to "Barry Reinhold" <bbrtrebia@mediaone.net>

To:   Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu
cc:
Subject:  RE: iSCSI: Towards a more effective PDU format




Julian,
     On the two drawbacks:

1. Agreed, there are two header digests and this isn't desirable. However,
I
suspect the TLVs will seldom be used so at least it is out of the fast
path.
[BTW - Can some digest expert suggest how bad this is vs. a single longer
digest?]

2. I'm not sure I fully understand what you are asking under two, so I'm
going to answer the question of "How would you read the TLV w/o using any
unverified info." If this is an answer to the wrong question, my apologies.
You'll have to ask again.

The TLV length field is obtained from the BHS header as, or after, it has
been verified. This TLV length field gives the size of the TLV data
(surprise) that has to be buffered in reading the TCP stream to get to the
TLV digest. The TLV digest is checked against the TLV data. The TLV's are
then parsed. (Humm - I guess I have an assumption in here that TLV data is
not *really* big, such that buffering the whole set of TLVs is a reasonable
thing to do)
I don't see where one ends up using unverified fields. I would expect to
read a fixed N bytes from the TCP stream (through the header digest),
verify
the header, use the fixed offset of the "TLV length field" that is in the
header, which has been verified, to find if I had TLV fields. If no, I'm on
to the data. If yes I read the TCP stream with the number of bytes given by
"TLV length field" + sizeof(TLV digest), verify the TLV fields, and then
parse the TLVs as needed.

A plus is that there is very little work to do for the fast path.
Another plus is that the TLV mechanism allows vendor options, and/or future
extensions with well understood backward compatibility behavior.
A third plus is that there should be no special cases relative to how the
PDU format has you read the TCP stream...read header, optionaly read TLV
stuff, read data. This should be friendly to hardware accelerated stuff.
A negative is that TLV parsing is sorta slow if you have complicated TLV
structures such as in DOCSIS. Our TLVs are very simple....so far at least.

>-----Original Message-----
>From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
>julian_satran@il.ibm.com
>Sent: Saturday, March 10, 2001 12:17 PM
>To: ips@ece.cmu.edu
>Subject: Re: iSCSI: Towards a more effective PDU format
>
>
>
>
>Barry,
>
>Thanks for outline.
>I will consider it as one of the variants I am going through to asses
>clarity.
>
>There are two things that are against it:
>
>1- several digests
>2- if I don't gather (how?) from BHS that TLVs are there then I have to do
>2 reads for each additional segment - and besides this you are using a
>field before knowing it is correct.
>
>We could however rearrange in any fashion the BHS and NWs to make it look
>very close to what you propose and still have one read per segment and
data
>used only after checking.
>
>Regards,
>Julo
>
>"Barry Reinhold - Lamprey POP3" <bbr@lampreyNetworks.com> on 10/03/2001
>04:02:28
>
>Please respond to "Barry Reinhold - Lamprey POP3"
<bbr@lampreyNetworks.com>
>
>To:   "ISCSI" <ips@ece.cmu.edu>
>cc:   "Robert Russell" <rdr@unh.edu>, "Ashish A. Palekar"
>      <ashishp@iol.unh.edu>, "Jon Sreekanth" <jon.sreekanth@trebia.com>,
>      "James Smart" <james.smart@trebia.com>
>Subject:  iSCSI: Towards a more effective PDU format
>
>
>
>
>All,
>
>I know there are well defined feelings about how a PDU should be
>structured,
>and not everyone is going to be in the same camp on this issue. However, I
>am concerned that the current PDU format with the WN field has not really
>achieved the goals that I, for one, would like to see it achieve. So, let
>me
>list a few objectives for the PDU format and then suggest one possible
>solution.
>
>Objectives for the iSCSI PDU (that could be improved upon).
>
>1. Simplicity - The PDU layout should be as straight forward as possible.
>Interoperability starts with simplicity. When the logical function is
>simple, documenting proper usage is a whole lot easier. When documentation
>is clear in a standard, the time to interoperability and market acceptance
>is reduced.
>
>2. Efficient operation in the common execution path. I think "rough
>consensus" can probably be achieved in terms of what parts of an iSCSI
>header are going to be needed in the normal flow and what parts are
>exception cases. If we have consensus on this we should be able to agree
on
>where to make things easy and where to add complexity.
>
>3. Data integrity - If a header digest is going to be provided, the
>information it covers should be checked against the digest before being
>used.
>
>I think we can improve the current PDU format, in terms of these
>objectives,
>by doing the following:
>
>1. Providing a fixed size header that contains the most commonly used
>fields.
>2. Placing less commonly used fields in an optional portion of the header.
>3. Providing a header digest over the fixed portion of the header and a
>separate digest for the optional portion of the header.
>
>Basic approach:
>
>1. The fixed "BHS" portion of the header is in all iSCSI PDU headers. It
is
>always the same size and is followed by a header digest that covers the
>"BHS", if header digests are being used. The fact that header digests are
>being used within a TCP byte stream is not encoded into the frame, but is
>obtained from state associated with the session. This allows the location
>of
>the header digest to be determined without using any information in the
>header. Only when the integrity of the header has been validated is the
>information in it used.
>
>2. The header contains information on the size of the complete PDU, and a
>length field for any additional headers. The integrity of this information
>has been validated as it is part of the "BHS". The location of the
>additional headers, if they exist (in most cases they will not) is after
>the
>header digest. Thus additional headers can be located using validated
>information.
>
>3. Each additional header is encoded using a standard TLV (type, length,
>value) format. In TLV format the fixed size type indicator tells how to
>interpret the data. The fixed size length field says how long the data is,
>and the variable sized value field contains the data. If an application
>doesn't understand the type, it can maintain sync. in the byte stream by
>skipping to the length field and then skipping the value.
>
>4. The complete set of TLV values are covered by a distinct digest. This
>digest is considered a part of the header digest in terms of iSCSI option
>negotiation but is physically distinct. Although this requires another
>digest calculation its usage is expected to be fairly low and it is most
>likely calculated over a small number of bytes.
>
>5. The data and data digest follow the TLV digest.
>
>In terms of layout, what you might have is the following:
>
>================= Start of iSCSI PDU =============================
>
>
>
>     This portion of the PDU contains the information in
>     the "BHS" plus:
>
>     1. The length field (4 bytes)
>     2. The length of the TLV information
>
>--------------- End of the fixed size header ---------------------
>--------------- Start of the header digest (optional) ------------
>
>--------------- End of the header digest -------------------------
>--------------- Start of TLV information (optional) --------------
>
>    (Additional header fields go here in a variable length space)
>
>
>--------------- End of TLV information ---------------------------
>------ Start of "TLV header" digest (if header digest present)----
>
>--------------- End of "TLV header" digest -----------------------
>--------------- Start of data ------------------------------------
>
>
>
>
>--------------- End of data --------------------------------------
>--------------- Start of data digest (optional) ------------------
>
>=============== End of data digest and PDU =======================
>
>[David B: This looks better in a .pdf....are we allowed to use pdfs on
this
>reflector?]
>
>I believe this to be a simple yet efficient structure for common usage.
>The optional headers require more involved processing, but allow
>incremental
>advancement of the TCP byte stream and avoids the loss of synchronization
>that might take place in other formats.
>The header can be verified, before being used.
>
>[Note 1: This PDU format does not attempt to address the problem of how to
>recover when there is a digest error. I believe this to be a distinct
>problem, though the PDU format could provide some additional level of
>robustness via redundancy of information at known locations.]
>
>[Note 2: Julian, If more detailed information on the PDU format is needed
I
>would be glad to present it. However I think that most of the significant
>issues are addressed by this outline. The other items don't appear, to me,
>to be as significant.]
>
>Barry Reinhold
>Barry.Reinhold@trebia.com
>603-659-0885
>
>
>
>