RE: Avoiding deadlock in iSCSI

["Douglas Otis" <dotis@sanlight.net>]

> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> David Robinson
> Sent: Monday, September 11, 2000 3:35 PM
> To: ips@ece.cmu.edu
> Subject: Re: Avoiding deadlock in iSCSI
>
>
> I think in following this discussion the terminology has been
> confusing me.  When I read "unsolicited data" I interpreted that
> to mean data for which no command has yet been sent.

David,

Here is my understanding.  Unsolicited may mean data sent with a command.
In SCSI, the target controls exchanges and thus makes all solicitations.
From the aspect of the SCSI model, unsolicited would imply data sent ahead
of a request from the target by the normal means of R2T.

> In general
> I consider that to be a bug and the receiver should just drop the
> data on the floor.

Not as I understand solicit.

> The only possible scenerio where it might
> not be a bug is if a command was sent on one connection and the
> data on the data connection arrived first, thus it is unsolicited.

Yes, unsolicited and out of sequence.

> My first assumption is that the sender would not send commands
> C1 and C2 and data D2 and D1 on the same connection. Doing that
> creates nasty ordering problems we want to avoid.

Order on the wire can not be controlled. Only the ULP can avoid such.

> So if the
> receiver simply allows the data connection TCP window to shrink
> the unsolicted data will flow control to a stop until the command
> queue catches up.

Resources are held until associated data is received to complete operations.
If the resource limit is not the data buffer nor freed by content already
within the data buffer, this will result in discarding commands.

> With multiple data connections, some may flow
> control but the active command will be able to make progress on
> one connection. This may not be the most efficient mechanism but
> it is "safe".

One connection per LUN or one connection per command, safe but expensive?

> Preferably the data will either follow the command
> on the same data/command connection or the sender will request a
> RTT (aka R2T). It is also a sender bug to request a connection
> for data transfer that it has already sent "unsolicited" data.

As a means for freeing resources, data is to be discarded within the iSCSI
architecture.  As such, even unsolicited data may be requested by the
target.

Doug

> Unless my assumptions and definitions are wrong, I don't see the issue.
>
> 	-David
>
> > The problem:
> >
> > iSCSI, as currently spec'ed, allows SCSI commands and data to be
> > interleaved fairly freely on a TCP connection. A target that stops
> > reading from a TCP connection to avoid reading more command packets
> > also prevents itself from reading data packets.  Those data packets
> > may be criticial to making progress on the currently executing
> > command.
> >
> > Note the issue appears with one TCP connection for control and data
> > and even appears in many of the multiple connection schemes.
> >
> > Data in iSCSI comes in two forms:
> >
> > 	1) solicited - data requested by target via RTT
> > 	             - data requested by initiator via a SCSI command
> > 	2) unsolicited - data sent by initiator without having
> received an RTT
> >
> > The analysis below assumes that unsolicited data travels over the same
> > TCP connection as SCSI commands. Otherwise, you run the risk of
> receiving
> > unsolicited data before the relevant SCSI command (thus making
> > implementations more complex).
> >
> > Four solutions:
> >
> > 1) Don't overflow the command queue (i.e. use credits)
> > 	- and what do you do if a misbehaving initiator overflows
> >         your command queue anyway? Drop the connection?
> >
> > 	- requires you to reserve resources per initiator. some people
> >         may want to overcommit
> >
> > 2) Allow dropping of SCSI commands when queue fills
> > 	- how do you clean up after a dropped SCSI command?
> > 	    - there may be other commands in the pipeline
> >
> > 	One approach: On command drop, the target enters an error
> > 	state. While in the error state, all newly received commands
> > 	terminate with an error until the initiator explicitly clears
> > 	the error state using a "clear error state" message.
> >
> > 	You might think that TASK SET FULL and ACA mechanisms from SCSI
> >         could be used to attack this problem. However, TASK SET
> FULL errors
> > 	don't trigger ACA (in my reading of the SAM). Also, ACA is only
> > 	triggered by the current enabled command, not by random commands
> > 	entered into the task set.
> >
> > 3) Put solicited data on a dedicated TCP connection. Require that
> > unsolicited data MUST follow the command, ideally in the same iSCSI
> > PDU
> >
> > 4) (Do it like NFS) Make all transfers from initiator to target
> > unsolicited. Make sure unsolicited data follows the command
> > immediately.
> >
> >
> > Of all the options, #1 and #4 sound the easiest to implement. #2 is more
> > sophisticated than #1. #3 is just plain clever but that's rarely a good
> > thing. :)  #4 has large ramifications on current SCSI target designs.
> >
> > -Costa
>