RE: Command Queue Depth (was asymmetric/Symmetric)

[Joshua Tseng <jtseng@NishanSystems.com>]

James,

I agree with others that there may be an issue with the
command windowing mechanism in the existing iSCSI spec.  It is like 
"TCP in reverse", in that the target determines the size of the window, and
not the initiator as in TCP.  Rather, I believe that everything that this
windowing mechanism is attempting to achieve can be more easily obtained
by having the target communicate its buffer size to the initiator at
iSCSI login.  It should be the role of the initiator to determine how
many commands to put in flight simultaneously, given this input on available
buffer size from the target.

As far as multiple initiators, could this not be resolved by the target
refusing additional logins beyond the number of initiators it can safely
support?  Not being a storage expert, this is my best guess/suggestion
at how to do it.

Josh

-----Original Message-----
From: James Smart [mailto:jssmart@nhinternet.com]
Sent: Wednesday, September 06, 2000 6:37 AM
To: ips
Subject: Command Queue Depth (was asymmetric/Symmetric)



Josh,

Wanted to double-check on the scsi-isms implied by the statements below,
and post a new question on managing command buffering ...

>>My question then is: does Data Balancing on multiple
>>connections cause a similar need for additional memory as does the
>>synchronous connections with Multiple Commands on different connections.
>
>In the symmetric model, the target must be prepared to buffer all of the
>commands and associated unsolicited data PDU's that may be sent between
>ExpCmdRN and MaxCmdRN.  This can be a lot of memory.  Either that or
discard
>commands.  But in order to realize the performance benefits of command
>windowing, the target must expand the window in order to let the initiator
>to "let her rip!" and put multiple commands in flight.  It seems to me that
>the existing symmetric model adds complexity which would only be of benefit
>if the target has lots of memory.  So my question is this:  The command
>windowing mechanism seems to require lots of resources (including memory)
>and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to
>implement, but does all this really buy very much?  In the end, symmetric
>implementations may end up discarding commands anyway, as in the asymmetric
>case, unless they want to install globs of memory.

First - I assume that "discard commands" actually means completing the
command with
a SCSI status of BUSY or QUEUE FULL - *NOT* simply dropping the command on
the floor.
As per previous statements on the reflector, OS's and existing drivers
expect
older-style SCSI behaviors - dropping of commands from lack of resources
expect
QUEUE FULL or BUSY indicators to invoke backoff algorithms. OS's typically
don't
deal well with i/o timeouts for error recovery.

Second, are the buffering expectations above in line with existing SCSI/FC
devices ?
Most scsi devices today, which support command queuing, do indeed support
buffering
up to N commands (N being the maximum queue depth and is shared across all
initiators). Simple disk drives typically support queue depths of 16, 32 or
64
(all to a single lun), while large controllers accept hundreds (shared
across
multiple luns).

Lastly, this topic touches on one of the more sensitive areas in the use of
SCSI over
packetized interfaces - that of managing the command queue. In parallel
SCSI, it was
dealt with more efficiently, as the QUEUE FULL or BUSY status was received
immediately on the interlocked bus before an additional commands could be
sent.
However, in FC and iSCSI, the commands will be in the pipe, and there will
(are)
storms of commands bombarding the target by multiple initiators. As
mentioned, unless
status was sent back to invoke the backoff algorithms, things continued
unabated.
I've seen FCP targets fully consumed just trying to hold off these storms.
It gets
worse as the initiators can be different OS's, with different backoff
policies
(some do onesy/twosy based on command completion and noting depth levels to
manage
how much they keep outstanding; while others simply delay for a time, then
let the gates wide open to whatever maximum they believe they can send (e.g.
a storm)).
Given the varied and uncoordinated rates at which the initiators can beat on
the
command queue resource, there are times where things melt down both on the
target
and on the initiator backoff/retry algorithms such that little real work
gets
achieved.

I know this last issue is not generic to iSCSI and may be classified as a
configuration
error. However - to the group - has iSCSI considered any mechanism to report
the
command queue depth (buffering) available to an initiator ? (my guess is
that the
delta between ExpCmdRN and MaxCmdRN via the login response implies this).
Further - any thoughts on how the resources could then be multiplexed across
multiple
initiators (thus the ExpCmdRN/MaxCmdRN can't really apply any more).

-- James

--------------------
James Smart   james.smart@nhinternet.com