Re: Some thoughts on synchronization & Other outstanding questions

[julian_satran@il.ibm.com]

Very good points Mark.  The only exception I would take is that if would be
nice
if we would have a way not to force things to pile up on an adapter because
some
router handling QoS flows has decided to drop a packet from an iSCSI
stream.
And yes there are many good solutions to it - some where already discussed
here.
To those I can add one only to illustrate how trivial it could be to add
synch. to a stream -
you can have a special frame issued in the stream every so many bytes or
time or both
and the framing for it will be escaped if it appears in the regular data
stream (i.e. hardware or
software will delete it before it reaches SCSI. It will contain the
distance to the previous
and next SCSI header.  This is the textbook solution for resynch and it is
very old.
However ... I consider it a bad solution for our environment as it will not
operate on plain vanilla
TCP/IP adapters that have only TCP assists but nothing special for iSCSI.
That is why a solution based on a simple TCP options (like the RDMA or even
only
a TCP option indicating how far it is until the next/previous header will
help). RDMA s also
great as it allows you to place the bulk data in memory regardless of
order.

With all those said I would like to remind you all that we are close to the
end of the cycle
and as we are running out of time - at least for the this IETF session. I
suggest that
we deal with the following three outstanding issues in this order:

1-ordered delivery & recovery over multiple connections
2-security (I am still missing the text!)
3-third party commands and the related naming and mapping issues
4-synchronization

I think that the posted draft has 1 fixed (some minor corrections might
reach it today)

I am busy with 3 (hope to get it until tomorrow - as I promised)

I am lost on 2 (any help appreciated). We have to get in the text the
agreements reached
in Haifa.

I am not sure that we will be able to do something different from what we
have on 4
until the IETF deadline.

Regards,
Julo

Mark Bakke <mark.bakke@nuspeed.com> on 04/07/2000 00:12:32

Please respond to Mark Bakke <mark.bakke@nuspeed.com>

To:   ips@ece.cmu.edu
cc:    (bcc: Julian Satran/Haifa/IBM)
Subject:  Some thoughts on synchronization





After watching the synchronization discussion for a
few days, I'm wondering if this isn't starting to get
too complicated for our own good.  Basically, iSCSI
uses TCP connections in almost exactly the same way
as HTTP/1.1 and CIFS.

The only ways I could come up with where sync would
be lost are:

1. A bit error happens in a length field (or another
field controlling the length of a header or data),
and is not caught by a data-link layer CRC or by the
TCP checksum.  If an error is detected at either of
these levels, the layer-2 frame or TCP packet is
discarded, and allowed to retransmit, so detected
errors do not cause data integrity or iSCSI framing
problems.

2. The initiator is not formatting messages correctly,
or is expecting incorrect message formats.

3. The target is not formatting messages correctly,
or is expecting incorrect message formats.

2 and 3 have to be solved by interoperability testing
anyway; detecting these problems should cause some
sort of warning via the initiator's and/or target's
management interfaces.


An undetected error can happen in a few ways:

- A set of bit errors within a layer-2 network happens
that makes an L2 frame still have the same CRC, and
at the same time makes the overall TCP segment have the
same checksum.  This is extremely unlikely, but possible.

- Bit errors happen on a router or other device that
strips and regenerates a layer-2 CRC, and at the same
time keeps the same TCP segment checksum.  This is
probably more likely that the previous case, since
only one of the checks has to fail in its duty.

- Bit errors happen in a bus, switched fabric, or buffer
memory in either the initiator or target, without being
detected by whatever scheme (parity, ECC, etc.) is in
use.  This will likely have a probability similar to
the bit-error-in-a-router case.

These are all cases that should rarely happen, although
their probabilities depend heavily on the equipment and
services purchased by the end customer.  However, they
can happen, and in some cases it's important enough
to deal with the possibility.

If we end up with these types of bit errors, corrupting
length fields, and messing up iSCSI framing, is probably
the least (or at least just one of) our worries.  For
example, a read command could become a write, or the wrong
data could be written to a database block, or the block
address could be wrong that's read or written, or ...

Anyway, ALL of these cases are equally bad.

I see a few things that are reasonable to do.

1. If either the initiator or target detects a framing
problem (length fields out-of-range, unexpected values
in a particular field, etc), it should proceed to shut
down the connection, and allow the initiator-based
connection recovery to take place and initiate a new
connection in its place.  Since these types of errors
should not happen more than once in a while (once per
day seems like alot of undetected errors), connection
recovery will make use of a mechanism we already have,
instead of adding another one.  If a framing problem is
detected in this way, chances are that the previous
iSCSI header was corrupted, and we skipped too many
or too few bytes to find the next one.  In this case,
damage may have been done, and either end of the
implementation detecting this should attempt to warn
the user through its normal management interfaces.

2. A magic number added to the header could help detect
framing problems.  This makes error checking on iSCSI
headers much stronger.  Again, however, the liklihood
is that the damage was actually to the previous iSCSI
header, and other damage may have already been done
as a result.  Again, the implementation should warn
like crazy.

3. If many of these errors are seen in a real customer
environment, or if the application absolutely cannot
tolerate a chance of error, then TCP stream resync
is definitely too weak anyway.  In this case, the
implementation should provide a data integrity check,
such as a CRC, on iSCSI headers, commands, and data,
where CRC errors on these will also cause a connection
recovery sequence.

I believe that these three suggestions, taken together,
would solve our synchronization problems without extra
effort (other than connection recovery and data integrity
check mechanisms, which we were going to do anyway).

Any other thoughts?


--
Mark A. Bakke
NuSpeed, Inc.
mark.bakke@nuspeed.com
763.398.1054