Re: Avoiding deadlock in iSCSI

["John Hufferd/San Jose/IBM" <hufferd@us.ibm.com>]

Costa,
I think I agree with the  comment from David Robinson, who said (among
other things) "...I don't see the issue.".

At least I do not see the issue with a command and a data Asymmetric
Session.  Perhaps all we have to do is make a rule, that no unsolicited
data can be sent, before its command is sent AND that unsolicited data must
be sent in the same order as the commands that reference it.  If that rule
is followed I do not see the problem.  The Data is (or should be) always
ready to be read off the data connection queue when needed by the command.

If the point is, even if the commands and data arrive in order, some
commands my execute quicker then others, then I would say that is even true
when the data follows immediately behind the command (on the same
connection), and it is the responsibility of a good controller to move the
command and data into an execution buffer/cache so that other commands can
get through and execute. We do this today.

If your controller does not have enough storage to get the slower commands
out of the way, then, they may block until the memory/buffer/cache is made
available (we all have this kind of mechanism), but there should not be any
Dead Locks.

If the concern is that in a TCP/IP network there may be some additional
memory needed in the storage controller to make the Blocking go away (at
least for 99+% of the time), I would say, sounds OK to me, that is part of
the system engineering that we all do today.

So Costa,  I think I probably missed your point, and if you would be so
kind as to lay the problem out again, perhaps addressing my points above,
I might finally "get it".

.
.
.
John L. Hufferd


csapuntz@cisco.com@ece.cmu.edu on 09/11/2000 03:02:42 PM

Sent by:  owner-ips@ece.cmu.edu


To:   ips@ece.cmu.edu
cc:   csapuntz@cisco.com
Subject:  Avoiding deadlock in iSCSI




The problem:

iSCSI, as currently spec'ed, allows SCSI commands and data to be
interleaved fairly freely on a TCP connection. A target that stops
reading from a TCP connection to avoid reading more command packets
also prevents itself from reading data packets.  Those data packets
may be criticial to making progress on the currently executing
command.

Note the issue appears with one TCP connection for control and data
and even appears in many of the multiple connection schemes.

Data in iSCSI comes in two forms:

     1) solicited - data requested by target via RTT
                  - data requested by initiator via a SCSI command
     2) unsolicited - data sent by initiator without having received an RTT

The analysis below assumes that unsolicited data travels over the same
TCP connection as SCSI commands. Otherwise, you run the risk of receiving
unsolicited data before the relevant SCSI command (thus making
implementations more complex).

Four solutions:

1) Don't overflow the command queue (i.e. use credits)
     - and what do you do if a misbehaving initiator overflows
        your command queue anyway? Drop the connection?

     - requires you to reserve resources per initiator. some people
        may want to overcommit

2) Allow dropping of SCSI commands when queue fills
     - how do you clean up after a dropped SCSI command?
         - there may be other commands in the pipeline

     One approach: On command drop, the target enters an error
     state. While in the error state, all newly received commands
     terminate with an error until the initiator explicitly clears
     the error state using a "clear error state" message.

     You might think that TASK SET FULL and ACA mechanisms from SCSI
        could be used to attack this problem. However, TASK SET FULL errors
     don't trigger ACA (in my reading of the SAM). Also, ACA is only
     triggered by the current enabled command, not by random commands
     entered into the task set.

3) Put solicited data on a dedicated TCP connection. Require that
unsolicited data MUST follow the command, ideally in the same iSCSI
PDU

4) (Do it like NFS) Make all transfers from initiator to target
unsolicited. Make sure unsolicited data follows the command
immediately.


Of all the options, #1 and #4 sound the easiest to implement. #2 is more
sophisticated than #1. #3 is just plain clever but that's rarely a good
thing. :)  #4 has large ramifications on current SCSI target designs.

-Costa