Re: Avoiding deadlock in iSCSI

["Randall R. Stewart" <randall@stewart.chicago.il.us>]

csapuntz@cisco.com wrote:
> 
> The problem:
> 
> iSCSI, as currently spec'ed, allows SCSI commands and data to be
> interleaved fairly freely on a TCP connection. A target that stops
> reading from a TCP connection to avoid reading more command packets
> also prevents itself from reading data packets.  Those data packets
> may be criticial to making progress on the currently executing
> command.
> 
> Note the issue appears with one TCP connection for control and data
> and even appears in many of the multiple connection schemes.
> 
> Data in iSCSI comes in two forms:
> 
>         1) solicited - data requested by target via RTT
>                      - data requested by initiator via a SCSI command
>         2) unsolicited - data sent by initiator without having received an RTT
> 
> The analysis below assumes that unsolicited data travels over the same
> TCP connection as SCSI commands. Otherwise, you run the risk of receiving
> unsolicited data before the relevant SCSI command (thus making
> implementations more complex).
> 
> Four solutions:
> 
> 1) Don't overflow the command queue (i.e. use credits)
>         - and what do you do if a misbehaving initiator overflows
>         your command queue anyway? Drop the connection?
> 
>         - requires you to reserve resources per initiator. some people
>         may want to overcommit
> 
> 2) Allow dropping of SCSI commands when queue fills
>         - how do you clean up after a dropped SCSI command?
>             - there may be other commands in the pipeline
> 
>         One approach: On command drop, the target enters an error
>         state. While in the error state, all newly received commands
>         terminate with an error until the initiator explicitly clears
>         the error state using a "clear error state" message.
> 
>         You might think that TASK SET FULL and ACA mechanisms from SCSI
>         could be used to attack this problem. However, TASK SET FULL errors
>         don't trigger ACA (in my reading of the SAM). Also, ACA is only
>         triggered by the current enabled command, not by random commands
>         entered into the task set.
> 
> 3) Put solicited data on a dedicated TCP connection. Require that
> unsolicited data MUST follow the command, ideally in the same iSCSI
> PDU
> 
> 4) (Do it like NFS) Make all transfers from initiator to target
> unsolicited. Make sure unsolicited data follows the command
> immediately.
> 
> 
> Of all the options, #1 and #4 sound the easiest to implement. #2 is more
> sophisticated than #1. #3 is just plain clever but that's rarely a good
> thing. :)  #4 has large ramifications on current SCSI target designs.
> 
> -Costa
Costa:

If you were using SCTP you could easily do #3 by not using
a seperate connection but instead use a specified stream in
the command for the data to come on...

Another options as well would be put each command/and data on
a different stream modulo the number of streams you were using.
This way one could always seperate out what one was reading :)

Some similar things could be worked out with TCP it would just
require a bit more work in the iSCSI layer... i.e. one could
pass a connection number in the command and this would be which
TCP connection you would read from :)

R
-- 
Randall R. Stewart
randall@stewart.chicago.il.us or rrs@cisco.com
815-342-5222 (cell) 815-477-2127 (work)