Re: iSCSI: Out of order commands, was current UNH Plugfest

[Santosh Rao <santoshr@cup.hp.com>]

Julian,

There's further reasons why iscsi must not mandate that the initiator
issue commands in order within a given connection :

1) The cmdsn assignment may be done in a host resident session manager
in an architecture that allows for multi-connection sessions that span
HBAs.
In such a model, the initiator may issue commands from the host driver
to the HBA in cmdsn order. However, within the HBA firmware, some
command[s] may be errored back to the host either due to resource
reasons or link down or mailbox command incorrectly initialized.
Requiring that the initiator issue commands in order implies that the
initiator HBA firmware needs to implement some sort of sequencing scheme
to ensure that it does not issue a lower cmdsn.

There's no need to add any of the above complexity into initiator HBA
firmware, since command ordering is already being implemented at the
target end. What we've ended up doing is inserting sequencing logic at
both ends in attempting to enforce compliance to this rule !

2) On a connection recovery, re-issued commands [on a cmdsn ack timeout]
may be issued on a different connection, thus, causing out of order
cmdsn's to be issued on a specific connection. There's no way for the
initiator HBA firmware to know that this is a re-issued command and
detect that this specific out of order cmdsn is to be allowed.

3) Initiator driver algorithms are quite flexible in their ordering
schemes today and imposing strict ordering at the initiator is beyond
what is required for the wire protocol. The layer that hands out cmdsn's
to outgoing commands may be a higher layer driver in the iscsi stack
(common across multiple vendor specific drivers) and re-ordering of
commands may occur in the lower layers of the iscsi stack.

For all of the above reasons, in addition to HBA optimizations that Rod
has voiced, I suggest that NO ordering requirement be imposed on
initiators within a specific connection. The cmdsn ordering scheme is
mandatory to implement/use and that is more than sufficient.

Regards,
Santosh

Julian Satran wrote:
> 
> Rod,
> 
> I don't see any reason why DMA operations cant be "multiplexed" with
> commands.
> If you have scheduled a long outbound DMA you are doomed regardless of the
> command ordering.
> And if you have scheduled DMA operations piecemeal then you can insert
> your commands in correct order.
> 
> Julo
> 
> "Rod Harrison" <rod.harrison@windriver.com>
> 05-11-01 20:48
> Please respond to "Rod Harrison"
> 
> 
>         To:     Julian Satran/Haifa/IBM@IBMIL, <ips@ece.cmu.edu>
>         cc:
>         Subject:        iSCSI: Out of order commands, was current UNH Plugfest
> 
> 
> 
>                  [ Subject changed ]
> 
> Julian,
> 
>                  The ordering difference is introduced between the host
> side driver
> and the iSCSI HBA. The host side driver must present SCSI commands to
> the HBA in the order they are received from the OS to prevent read
> after write dependency failures. The HBA might reorder the commands
> depending on when DMA completes. The reordering can't be done ahead of
> time in the host driver since it doesn't know how long each DMA might
> take. As long as the HBA assigns CmdSN in the order it receives
> commands the desired host ordering is preserved.
> 
>                  - Rod
> 
> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> Julian Satran
> Sent: Monday, November 05, 2001 12:35 AM
> To: ips@ece.cmu.edu
> Subject: RE: iSCSI: current UNH Plugfest
> 
> Rod,
> 
> I all examples give the point I find hard to understand is why is the
> ordering on the wire different from the presentation order to the
> initiator.  You can get as many overlaps as you want by presenting the
> commands to the initiator in the desired order.
> What we are considering here is the case in which you want to ship in
> an
> order different than the one you present the commands.
> 
> Julo
> 
> "Rod Harrison" <rod.harrison@windriver.com>
> Sent by: owner-ips@ece.cmu.edu
> 04-11-01 04:42
> Please respond to "Rod Harrison"
> 
>         To:     "Barry Reinhold" <bbrtrebia@mediaone.net>, "Dave
> Sheehy"
> <dbs@acropora.rose.agilent.com>, "IETF IP SAN Reflector"
> <ips@ece.cmu.edu>
>         cc:
>         Subject:        RE: iSCSI: current UNH Plugfest
> 
> Barry,
> 
>                  In general I agree but I don't think this is as much
> of a
> corner case
> as it at first appears. Targets will have code very similar to that
> needed to handle out of order commands to deal with digest errors.
> Targets also need to queue commands whilst waiting for both solicited
> and unsolicited data to arrive. Queuing out of order commands seems
> little extra work.
> 
>                  From an initiators point of view there are
> efficiency,
> and probably
> performance gains to be had from sending commands out of order. Bob
> Russell gave the example of a read being sent whilst write data DMA is
> happening, and a similar situation can arise with DMA for writes
> overtaking that of earlier writes if the initiator has multiple DMA
> engines. In this case the initiator might be forced to let the wire go
> idle if it can't send the data from completed DMAs as soon as
> possible.
> 
>                  We already have a command queue at the target to
> enforce
> correct
> serialisation of commands, doing the same thing at the initiator is
> redundant.
> 
>                  Finally, I don't believe we should be writing a
> standard
> to work
> around poor coding and test coverage, especially at the cost of
> potential efficiency gains.
> 
>                  I agree with Dave and Santosh that commands being
> sent
> out of order
> on a single session should be allowed by the standard.
> 
>                  - Rod
> 
> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> Barry Reinhold
> Sent: Friday, November 02, 2001 5:24 PM
> To: Dave Sheehy; IETF IP SAN Reflector
> Subject: RE: iSCSI: current UNH Plugfest
> 
> Using features such as out of order command delivery on a connection
> tend to
> be the sort of things that lead to interoperability problems. It is
> unexpected and probably going to hit poorly tested code paths even if
> the
> standard is written to allow it.
> 
> >-----Original Message-----
> >From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf
> Of
> >Dave Sheehy
> >Sent: Friday, November 02, 2001 4:19 PM
> >To: IETF IP SAN Reflector
> >Subject: Re: iSCSI: current UNH Plugfest
> >
> >
> >
> >> 3. Can commands be sent out of order on the same connection?
> >>
> >>    The behavior of targets is clearly specified in Section 2.2.2.3
> on
> >>    page 25 of draft 8, which says:
> >>      "Except for the commands marked for immediate delivery the
> iSCSI
> >>      target layer MUST eliver the commands for execution in the
> order
> >>      specified by CmdSN."
> >>
> >>    Section 2.2.2.3 on page 26 of draft 8 also says:
> >>      "- CmdSN - the current command Sequence Number advanced by 1
> on
> >>      each command shipped except for commands marked for immediate
> >>      delivery."
> >>    but the meaning of the term "shipped" is vague, and does not
> >> necessarily
> >>    require that the PDUs arrive on the other end of a TCP
> connection
> >>    in the same order that the CmdSN values were assigned to these
> PDUs.
> >>
> >>    Some initiators have been designed to send commands out of CmdSN
> >>    order on one connection.  Consider the situation where there is
> only
> >>    one connection and a high-level dispatcher creates a PDU for a
> SCSI
> >>    command that involves writing immediate data to the target.
> This PDU
> >>    is enqueued to a lower-level layer which has to setup, start,
> and
> >>    wait-for a DMA operation to move the immediate data into an
> onboard
> >>    buffer before the PDU can be put onto the wire.  While this is
> >>    happening, the dispatcher creates another unrelated PDU for a
> SCSI
> >>    read command (for example), and when this PDU is passed to the
> >>    lower-level layer it can be sent immediately, ahead of the
> previous
> >>    write PDU and therefore out of order on this connection.
> >>
> >>    The standard clearly allows this to happen if the two PDUs were
> sent
> >>    on different connections, and seems to imply that this can also
> happen
> >>    when the two PDUs are sent on the same connection.
> >>
> >>    The suggestion is to put in the standard an explicit statement
> that
> >>    this is allowed or not allowed, as appropriate.
> >>
> >>    If this is allowed, such a statement would avoid the erroneous
> >>    assumption being made by some target implementers that within a
> single
> >>    connection, commands will arrive in order.
> >>
> >>    If this is not allowed, such a statement would avoid the
> erroneous
> >>    assumption being made by some initiator implementers that within
> a
> >>    single connection, commands can be put on the wire out of order.
> >>
> >> +++
> >>
> >> will add an explicit statement saying that this behaviour is
> forbidden.
> >> 2.2.2.1 will contain:
> >>
> >> On any given connection, the iSCSI initiator MUST send the
> >commands in the
> >> order specified by CmdSN.
> >>
> >> +++
> >
> >Why do you feel this behavior should be forbidden? Targets already
> have to
> >order commands across the session. I don't see why it's a problem to
> extend
> >that to the connection as well. I, for one, believe we should take
> >a liberal
> >stance on this.
> >
> >Dave Sheehy
> >

-- 
##################################
Santosh Rao
Software Design Engineer,
HP-UX iSCSI Driver Team,
Hewlett Packard, Cupertino.
email : santoshr@cup.hp.com
Phone : 408-447-3751
##################################