RE: Avoiding deadlock in iSCSI

["Y P Cheng" <ycheng@advansys.com>]

Costa Wrote:
> Here is the scenario I proposed. This scenario applies to the iSCSI
> model as presented in Pittsburg, in which there is only one TCP connection
> (shared between commands and data). There is no unsolicited data.
>
> 	1) Initiator sends command 1 (WRITE BLOCK)
> 	2) Initiator sends command 2 (WRITE BLOCK)
> 	3) Target reads command 1
> 	4) Target sends RTT to initiator for data in command 1
> 	5) Target reads from connection and notices comand 2
> 	6) Target stops reading from connection because it does
> 	   not have room for command 2
> 	7) Initiator sends WRITE data for command 1. WRITE data
>            is stuck behind command 2 in the stream.
>
> There are basically three points where you can solve the problem:
> <snip>

Thank you for the example.

The step #6 does not happen in the NIC adapter or iSCSI driver I was
referring to.  When the adapter or driver send out a SCSI command, it
already has a table entry created to serve the RTT. This is because the
interface between the SCSI-class driver and the iSCSI driver defines an
"atomic" SCSI request.  This interface does not require the ULP, i.e.
SCSI-class driver, to respond to the RTT with a write data call.  The data
transfer takes place immediately from the NIC adapter or the iSCSI driver
after receiving the RTT and ahead of command 2 --  just like the asymmetric
model.

Your other example showing "command out order due to QUEUE-FULL status" is
accurate. To avoid such problem, the initiator must adhere the QUEUE-DEPTH
rule by not sending more commands to overflow the target.

The fundamental assumption in my position is that the NIC adapter and iSCSI
driver handle hundreds or even thousands of commands without blocking.  In
other words, the commands just flow through the adapter or the driver
without ever being blocked.  This requires the initiator not waiting for an
ACK of the TCP from the target.  Ultimately, the ACK will come from the
status PDU or the RTT.  The initiator is self regulating because its
resource limits the number of commands sent.  The initiator pre-allocates
buffers for all incoming data.  The target limits the command by giving
queue-depth parameters to each initiator.

Waiting for ACK on a network with long latency between two endpoints is a
very expensive proposition.  This is why I talked about UDP.  Then, I
realized that this subject was discussed before.  What we need is the
"reliable reception" -- a term used in VI -- provided in TCP.  However, the
same can also be achieved by having the NIC adapter or the iSCSI driver
managing the RTT and status PDU without involving the ULP. The case of a
lost PDU due to traffic jam or broken connection is also detected and
handled by the same.  A smooth recovery is a necessary condition for the
"reliable delivery" protocol.