RE: iSCSI: remove recovery from transport-layer connection failur e(?)

[julian_satran@il.ibm.com]

This pretty much summarizes our arguments for recovery and command
ordering.
I would like to add that Randy Haagens also made the point that new remote
tape and
disk applications could hide latency by striping over several (or one)
connection achieving
a pipelining effect provided that order is preserved.

Julo
---------------------- Forwarded by Julian Satran/Haifa/IBM on 02/10/2000
12:15 ---------------------------





JoeBre@exabyte.com on 29/09/2000 18:35:48

Please respond to JoeBre@exabyte.com

To:   steph@cs.uchicago.edu, ips@ece.cmu.edu
cc:    (bcc: Julian Satran/Haifa/IBM)

Subject:  RE: iSCSI: remove recovery from transport-layer connection failur
      e(?)





Comments inline...

> -----Original Message-----
> From: Stephen Bailey [mailto:steph@cs.uchicago.edu]
> Sent: Tuesday, September 26, 2000 10:16 PM
> To: ips@ece.cmu.edu
> Subject: Re: iSCSI: remove recovery from transport-layer connection
> failure(?)
>
>
> > Currently, iSCSI is spec'ed to recover from transport-layer
> > connection failures.
> >
> > The main motivation for this decision was to support tape backup
> > applications that are quite sensitive to any failures that get
> > propogated to their layer.
> >
> > So, perhaps we can remove the requirement of recovering from
> > transport-layer connection failures in iSCSI. This would simplify
> > the protocol somewhat.

Yes, it would simplify the protocol somewhat, and would also yield a
protocol that would not accomodate tapes (or any other sequential device).

I guess the question to be answered is wheter or not iSCSI will be a
suitable transport/LLP for all SCSI traffic, as defined in SAM/SAM-2. If
so,
some means of recovering from transport layer problems must be built into
the spec.

It may be worth pointing out here that one of the most significant barriers
to early widespread market acceptance of SANs has been the inability to
back
up data over the SAN. The most significant barrier to backup was the lack
of
support in FC for LLP/transport level recovery of commands. As I have
mentioned before, issuing the same command twice to a sequential device
corrupts data.

If, on the other hand, iSCSI is not to be SAM/SAM-2 compliant...

> >
> > Thoughts?
>
> I'm all for eliminating command recovery.
>
> There seem to be several reasons advanced for command recovery.
>
> The first seems to be based upon an inappropriate analogy to FCP.
> Command recovery had to be added to FCP-2 because the FC layer is
> unreliable.  A single dropped FC frame leads to a failed FCP command.
> This clearly upsets tape operation even when the link is performing
> nominally.  In FCP, without command recovery, with some observable
> frequency, you will get an expected error that leads to complete,
> irrecoverable failure of a transfer stream.  The other thing that
> makes FCP-2 command recovery work well is when you are doing a write,
> which is 90% (maybe it's 99%?) of tape operation, the target can
> return an early indication of most frame drops, rather than waiting
> for a timer to expire.
>
> TCP's reliability solves this problem in another way.  By the time you
> get a TCP connection failure, you have already exhausted a set of
> reliability mechanisms which guarantee, with high certainty, that
> further data can not be transferred between the two endpoints.
>
> `the two endpoints' phrase suggests the other reason advanced for
> command recovery.  That is, to permit path failover for commands which
> are not idempotent, such as tape write sequential.  The
> problem with this, is that it is not clear HOW iSCSI command recovery
> can actually work properly, given a TCP connection failure indication.
> It takes a long time for a TCP connection to fail, and by that time,
> I'm not sure recovery would reasonably be possible.  Perhaps I'm in
> error on this assumption.  Can a tape guru (Joe from Exabyte?) comment
> on whether recovery would be possible after many seconds (tens,
> hundreds) have elapsed?

Putting my tape guru hat on ;-)

There are numerous backup applications that set command timeouts in excess
of ten minutes. When one considers the sequential nature of tape, one will
realize that this is an eminently reasonable thing to do.

> The SCSI layer has never been solely responsible for ensuring reliable
> backup.  Macro scale things go wrong with tape (run off the end, get
> eaten, etc..) with relatively high frequency.  A low level backup
> engine like tar or dump will fail on a SCSI error, and that's OK.
> There must also be a higher level software component like Amanda,
> which manages retries, including operator intervention, to ensure
> reliable backup.

A mature tape drive, coupled with a mature backup application, is unlikely
to run off the end or get eaten. I guess 'relatively high frequency' must
be
qualified. Most installations will encounter neither of these problems ever
in the service life of the system.

The problem area that I foresee is where commands get lost in transmission.
With disks, the command can merely be resent. This is fine, as the
associated data goes to/is taken from the same LBA on the media. Whether it
was the command or the response that got lost in transmission, data is
fine.
With sequential devices, if it was truly the command that got lost in
transmission, the command can safely be resent. However, if the response
gets lost rather than the command, reissuing a WRITE command results in two
copies of the data on the tape - one at the LBA the host application
expects
it to be, and another starting at the next sequential LBA. The host's
mapping of what data it thinks is at each LBA is corrupted, leading to its
subsequent inability to recover data from that point to the end of the
tape.

> It seems like whether iSCSI has a command recovery mechanism should be
> a function of whether somebody can stand up and say for sure that it
> solves a real problem.  So far it only seems like it MIGHT solve a
> problem.  Who can say `this solves MY problem!'?

Command recovery solves the problem outlined above. It also solves the
problem of being a SAM/SAM-2 compliant LLP.

Joe
Exbayte

> Steph
>

(Embedded image moved to file: pic31154.pcx)
 - att1.htm






JoeBre@exabyte.com on 29/09/2000 19:16:08

Please respond to JoeBre@exabyte.com

To:   ips@ece.cmu.edu
cc:    (bcc: Julian Satran/Haifa/IBM)

Subject:  RE: Status summary on multiple connections





Much good discussion on SCSI device level flow control snipped....

> From: Robert Snively <rsnively@Brocade.COM>
> To: "'David Robinson'" <David.Robinson@EBay.Sun.COM>, Robert Snively
> <rsnively@Brocade.COM>
> Subject: RE: Status summary on multiple connections
> Date: Thu, 28 Sep 2000 09:36:50 -0700

...

> are again available.  Note that there is a possibility that commands
> that are inflight and have ordering constraints may be accepted out of
> order, a question that has caused lots of agonizing, but is apparently
> reasonably well managed by most file systems today by the selective
> use of ordering only for blocking boundaries of a particular logical
> stream of commands.

I am left with the following impression as to what was indicated here:
- In general, command ordering is not relevant
- If the initiator filesystem detects an ordering dependency, it will wait
until outstanding commands are complete before issuing the dependant
command.

This may be a reasonable means of operation for the disk world. It is
woefully inadequate for the tape world, as follows:
- In general, command ordering is crucial - out of order command processing
will lead to data corruption.
- This would require the initiator backup application to block on
completion
of every single write command of a backup operation before issuing the next
command.

If this blocking were performed, both the throughput and capacity of a tape
device/media would be negatively impacted by an order of magnitude or more.
This would occur even assuming an instantaneous transport.

Joe Breher
Exabyte Corp


(Embedded image moved to file: pic29236.pcx)
 - att1.htm

pic31154.pcx

pic29236.pcx