Command Queue Depth (was asymmetric/Symmetric)

["James Smart" <jssmart@nhinternet.com>]

Josh,

Wanted to double-check on the scsi-isms implied by the statements below,
and post a new question on managing command buffering ...

>>My question then is: does Data Balancing on multiple
>>connections cause a similar need for additional memory as does the
>>synchronous connections with Multiple Commands on different connections.
>
>In the symmetric model, the target must be prepared to buffer all of the
>commands and associated unsolicited data PDU's that may be sent between
>ExpCmdRN and MaxCmdRN.  This can be a lot of memory.  Either that or
discard
>commands.  But in order to realize the performance benefits of command
>windowing, the target must expand the window in order to let the initiator
>to "let her rip!" and put multiple commands in flight.  It seems to me that
>the existing symmetric model adds complexity which would only be of benefit
>if the target has lots of memory.  So my question is this:  The command
>windowing mechanism seems to require lots of resources (including memory)
>and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to
>implement, but does all this really buy very much?  In the end, symmetric
>implementations may end up discarding commands anyway, as in the asymmetric
>case, unless they want to install globs of memory.

First - I assume that "discard commands" actually means completing the
command with
a SCSI status of BUSY or QUEUE FULL - *NOT* simply dropping the command on
the floor.
As per previous statements on the reflector, OS's and existing drivers
expect
older-style SCSI behaviors - dropping of commands from lack of resources
expect
QUEUE FULL or BUSY indicators to invoke backoff algorithms. OS's typically
don't
deal well with i/o timeouts for error recovery.

Second, are the buffering expectations above in line with existing SCSI/FC
devices ?
Most scsi devices today, which support command queuing, do indeed support
buffering
up to N commands (N being the maximum queue depth and is shared across all
initiators). Simple disk drives typically support queue depths of 16, 32 or
64
(all to a single lun), while large controllers accept hundreds (shared
across
multiple luns).

Lastly, this topic touches on one of the more sensitive areas in the use of
SCSI over
packetized interfaces - that of managing the command queue. In parallel
SCSI, it was
dealt with more efficiently, as the QUEUE FULL or BUSY status was received
immediately on the interlocked bus before an additional commands could be
sent.
However, in FC and iSCSI, the commands will be in the pipe, and there will
(are)
storms of commands bombarding the target by multiple initiators. As
mentioned, unless
status was sent back to invoke the backoff algorithms, things continued
unabated.
I've seen FCP targets fully consumed just trying to hold off these storms.
It gets
worse as the initiators can be different OS's, with different backoff
policies
(some do onesy/twosy based on command completion and noting depth levels to
manage
how much they keep outstanding; while others simply delay for a time, then
let the gates wide open to whatever maximum they believe they can send (e.g.
a storm)).
Given the varied and uncoordinated rates at which the initiators can beat on
the
command queue resource, there are times where things melt down both on the
target
and on the initiator backoff/retry algorithms such that little real work
gets
achieved.

I know this last issue is not generic to iSCSI and may be classified as a
configuration
error. However - to the group - has iSCSI considered any mechanism to report
the
command queue depth (buffering) available to an initiator ? (my guess is
that the
delta between ExpCmdRN and MaxCmdRN via the login response implies this).
Further - any thoughts on how the resources could then be multiplexed across
multiple
initiators (thus the ExpCmdRN/MaxCmdRN can't really apply any more).

-- James

--------------------
James Smart   james.smart@nhinternet.com