Re: iSCSI:out-of-order notification proposal

[Sandeep Joshi <sandeepj@research.bell-labs.com>]

Answering two responses in one email.  This would be mighty 
easier if there was a whiteboard around here :-)

> What exactly does it buy you? (that is not already in).

The ability to execute subsequent PDUs in the presence of holes.
(Holes due to multiple connections OR due to skipping onto next marker)

If the hole is due to a REAL ordering issue (e.g. ORdered task
was lost) then the target will pause for the hole to fill.

But if the hole is due to simple task, then the target
can continue execution of the new commands it is receiving.

Currently, target will *always* pause due to cmdSN ordering,
which is somewhat like a primitive microprocessor which wont 
reorder its instruction stream, though its feasible.

> 
> And FWIW David (B) raised a question related to the requirements doc.

My mistake, the rephrasing of requirements seemed to match the intent 
of previous discussions on why total cmdSN ordering is overkill.
That was the only point i was trying to reinforce.

> 
> Julo



"Mallikarjun C." wrote:
> 
> Sandeep,
> 
> Some comments on your proposal.
> 
> - The "strict_order" flag that you mention appears to carry
>   information that's already contained by the SCSI task attributes
>   (ATTR field).

But now the information is also contained in PDUs following it.
(e.g. A simple command PDU may also have strict_order=1)

If an Ordered task PDU is lost, the target *knows* there is a REAL
ordering issue (later PDUs have info) so it must wait for the hole 
to be filled.

> 
> - This proposal requires the target iSCSI layer to look at more
>   variables (SCSI-level information in that) for making its
>   sequencing decisions.

Only one more bit.. to decide if the command can be passed
to SCSI layer now or later in cmdSN sequence.  

To build an efficient SCSI transport, some hints would need
to be passed between layers.

> 
> - Simple commands (intended to be) received prior to an Ordered
>   command must be executed before the Ordered one.  If you lose a
>   Simple command, you must wait for it before you act on the Ordered
>   that you received out-of-order.

Good pt.  This, I forgot to mention below.  The target must clear
out all PDUs received with strict-order=0 before commencing 
execution of PDUs received with strict-order=1.

> 
> This appears more an implementaion optimization where desired than
> something we want to get into the draft.  You are proposing that
> even in the absence of errors, iSCSI should make ordering decisions
> based on SCSI task attributes.  I disagree with that.

The optimization needs a bit in the PDU, otherwise I wouldnt
have gone to the trouble of writing this out!  And you would agree
that there are a bunch of TCP optimizations in various drafts.

Lastly, what is the point in building a sync-and-steering layer to
skip ahead in the command stream if you cannot execute that next command
?
With this little more work, you could as well continue execution.

> --
> Mallikarjun
> 
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions Organization
> MS 5668 Hewlett-Packard, Roseville.
> cbm@rose.hp.com
> 
> >This is a proposal to allow the initiator to inform the target
> >if out-of-order execution within the command stream is possible.
> >
> >The target execution can rotate between "in-cmdSN-order" and
> >"out-of-order" during runtime as informed by the initiator..
> >
> >Appreciate comments on subtleties that I may have missed.
> >
> >thanks,
> >-Sandeep
> >
> >http://ips.pdl.cs.cmu.edu/mail/msg03152.html
> >   by Costa provides a good summary of the issue at hand.
> >
> >http://ips.pdl.cs.cmu.edu/mail/msg04255.html
> >  David provides the "new" reqts.  In particular, this one
> >
> >   > MUST specify the ability to preserve ordered delivery
> >   > of SCSI commands even in the presence of transport
> >   > errors.  A mechanism MUST be provided to allow
> >   > Initiators and Targets to negotiate this preservation
> >   > on a per-session or finer granularity basis
> >
> >Note :
> >======
> >1) This does not rely on SCSI cmdRN, but operates at iSCSI level.
> >2) Flow control using cmdSN works as designed.
> >3) This solution is not a per-session negotiation option but can be
> >   disabled and re-enabled again at "runtime" by the initiator if it
> >   notices that Ordered/HOQ tasks (or any other need) have entered
> >   the iSCSI command stream which is being dispatched.
> >
> >Problem:
> >=========
> >In case of out-of-order arrival or digest errors, it is NOT possible
> >to know if the initiator had sent an ordered command before the one
> >which was received.
> >
> >Solution:
> >=========
> >To notify target of presence of in-flight ordered commands, we set
> >a flag on *every* PDU following the ordered command *until* the target
> >moves it expCmdSN above the cmdSN of the ordered command.  The
> >expCmdSN indicates target has found the ordered command.
> >
> >( Those familiar with "ECN over IP" (Floyd, et al) may see this is
> >similar to how a congestion bit keeps being set until the sender acks
> >that it has received the notification)
> >
> >Figuratively:
> >=============
> >So, assume there was a new "strict_order" flag in the BHS.
> >In the figure below, braces shows value of (cmdSN, strict_order)
> >
> >Initiator                       Target
> >---------                       -------
> >
> >simple cmd (cmdSN=100, strict_order=0) ->
> >simple cmd (101, 0) ->
> >simple cmd (102, 0) ->
> >
> >   these may get reordered or have digest errors ->
> >
> >                        target executes as they arrive
> >                             exec simple cmd(102, 0)
> >                             exec simple cmd(100, 0)
> >                             exec simple cmd(101, 0)
> >
> >Now say the initiator wants to send a HOQ task.
> >It sets strict_order=1 on all PDUs
> >
> >ordered cmd (103, 1) ->
> >simple  cmd (104, 1) ->
> >simple  cmd (105, 1) ->
> >                        in case of reordering or digest errors
> >                        target must wait & execute in cmdSN order!
> >simple  cmd (106, 1) ->
> >simple  cmd (107, 1) ->
> >
> >         <---- now target sends expCmdSN=103
> >
> >This implies target has seen command(cmdSN=103) and target will do the
> >appropriate ordering and delivery to SCSI layer.  This is left to
> >the target implementation to tackle.
> >
> >Initiator checks (expCmdSN >= cmdSN of ordered cmd) and then resets
> >strict_order to zero for all subsequent PDUs.
> >
> >simple  cmd (108, 0) ->
> >simple  cmd (109, 0) ->
> >
> >Other issues:
> >=============
> >If the basic scheme is ok, then we could later tackle other questions
> >"what about multiple ordered commands" and the like..