Re: iSCSI:out-of-order notification proposal

[Sandeep Joshi <sandeepj@research.bell-labs.com>]

Comments in text..

"Mallikarjun C." wrote:
> 
> Sandeep,
> 
> It's just not a single SCSI info-bit that the target iSCSI layer should
> look at for the optimization you're proposing - the LUN needs to be
> examined as well since the task attributes are scoped per-LU.
> 
> In effect, what you're suggesting is an iSCSI-level per-LU ordering with
> a session-global command sequencing scheme - that is a quite a bit of
> complexity for targets to manage!

Ah, I did not understand what you were getting at.

Let me emphasize - this is just a HINT of what's in the pipeline, 
and not a method to replace or duplicate SCSI functionality.

In the (common?) case where you have an incoming stream of  
unrelated tasks and a large command window, you might as well 
pass them onto SCSI layer right away.  

In the worst case, the hint will be a bad one and the target will 
perform no better than is currently planned.

A high-performance transport architecture with multiple connections and
multi-NIC support is bound to see PDU reordering, and consequently
suffer if it does not overcome this strict cmdSN ordering rule. 

> 
> BTW, synch and steering layer is *not* meant to help iSCSI (or SCSI)
> process payloads out-of-order.  It is targetted only for out-of-order
> data placements to reduce implementation costs, as the current draft
> clearly states.

ok..I havent looked at this section in depth yet :-)

> 
> Regards.
> --
> Mallikarjun
> 
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions Organization
> MS 5668 Hewlett-Packard, Roseville.
> cbm@rose.hp.com
> 
> >Answering two responses in one email.  This would be mighty
> >easier if there was a whiteboard around here :-)
> >
> >> What exactly does it buy you? (that is not already in).
> >
> >The ability to execute subsequent PDUs in the presence of holes.
> >(Holes due to multiple connections OR due to skipping onto next marker)
> >
> >If the hole is due to a REAL ordering issue (e.g. ORdered task
> >was lost) then the target will pause for the hole to fill.
> >
> >But if the hole is due to simple task, then the target
> >can continue execution of the new commands it is receiving.
> >
> >Currently, target will *always* pause due to cmdSN ordering,
> >which is somewhat like a primitive microprocessor which wont
> >reorder its instruction stream, though its feasible.
> >
> >>
> >> And FWIW David (B) raised a question related to the requirements doc.
> >
> >My mistake, the rephrasing of requirements seemed to match the intent
> >of previous discussions on why total cmdSN ordering is overkill.
> >That was the only point i was trying to reinforce.
> >
> >>
> >> Julo
> >
> >
> >
> >"Mallikarjun C." wrote:
> >>
> >> Sandeep,
> >>
> >> Some comments on your proposal.
> >>
> >> - The "strict_order" flag that you mention appears to carry
> >>   information that's already contained by the SCSI task attributes
> >>   (ATTR field).
> >
> >But now the information is also contained in PDUs following it.
> >(e.g. A simple command PDU may also have strict_order=1)
> >
> >If an Ordered task PDU is lost, the target *knows* there is a REAL
> >ordering issue (later PDUs have info) so it must wait for the hole
> >to be filled.
> >
> >>
> >> - This proposal requires the target iSCSI layer to look at more
> >>   variables (SCSI-level information in that) for making its
> >>   sequencing decisions.
> >
> >Only one more bit.. to decide if the command can be passed
> >to SCSI layer now or later in cmdSN sequence.
> >
> >To build an efficient SCSI transport, some hints would need
> >to be passed between layers.
> >
> >>
> >> - Simple commands (intended to be) received prior to an Ordered
> >>   command must be executed before the Ordered one.  If you lose a
> >>   Simple command, you must wait for it before you act on the Ordered
> >>   that you received out-of-order.
> >
> >Good pt.  This, I forgot to mention below.  The target must clear
> >out all PDUs received with strict-order=0 before commencing
> >execution of PDUs received with strict-order=1.
> >
> >>
> >> This appears more an implementaion optimization where desired than
> >> something we want to get into the draft.  You are proposing that
> >> even in the absence of errors, iSCSI should make ordering decisions
> >> based on SCSI task attributes.  I disagree with that.
> >
> >The optimization needs a bit in the PDU, otherwise I wouldnt
> >have gone to the trouble of writing this out!  And you would agree
> >that there are a bunch of TCP optimizations in various drafts.
> >
> >Lastly, what is the point in building a sync-and-steering layer to
> >skip ahead in the command stream if you cannot execute that next command
> >?
> >With this little more work, you could as well continue execution.
> >
> >> --
> >> Mallikarjun
> >>
> >> Mallikarjun Chadalapaka
> >> Networked Storage Architecture
> >> Network Storage Solutions Organization
> >> MS 5668 Hewlett-Packard, Roseville.
> >> cbm@rose.hp.com
> >>
> >> >This is a proposal to allow the initiator to inform the target
> >> >if out-of-order execution within the command stream is possible.
> >> >
> >> >The target execution can rotate between "in-cmdSN-order" and
> >> >"out-of-order" during runtime as informed by the initiator..
> >> >
> >> >Appreciate comments on subtleties that I may have missed.
> >> >
> >> >thanks,
> >> >-Sandeep
> >> >
> >> >http://ips.pdl.cs.cmu.edu/mail/msg03152.html
> >> >   by Costa provides a good summary of the issue at hand.
> >> >
> >> >http://ips.pdl.cs.cmu.edu/mail/msg04255.html
> >> >  David provides the "new" reqts.  In particular, this one
> >> >
> >> >   > MUST specify the ability to preserve ordered delivery
> >> >   > of SCSI commands even in the presence of transport
> >> >   > errors.  A mechanism MUST be provided to allow
> >> >   > Initiators and Targets to negotiate this preservation
> >> >   > on a per-session or finer granularity basis
> >> >
> >> >Note :
> >> >======
> >> >1) This does not rely on SCSI cmdRN, but operates at iSCSI level.
> >> >2) Flow control using cmdSN works as designed.
> >> >3) This solution is not a per-session negotiation option but can be
> >> >   disabled and re-enabled again at "runtime" by the initiator if it
> >> >   notices that Ordered/HOQ tasks (or any other need) have entered
> >> >   the iSCSI command stream which is being dispatched.
> >> >
> >> >Problem:
> >> >=========
> >> >In case of out-of-order arrival or digest errors, it is NOT possible
> >> >to know if the initiator had sent an ordered command before the one
> >> >which was received.
> >> >
> >> >Solution:
> >> >=========
> >> >To notify target of presence of in-flight ordered commands, we set
> >> >a flag on *every* PDU following the ordered command *until* the target
> >> >moves it expCmdSN above the cmdSN of the ordered command.  The
> >> >expCmdSN indicates target has found the ordered command.
> >> >
> >> >( Those familiar with "ECN over IP" (Floyd, et al) may see this is
> >> >similar to how a congestion bit keeps being set until the sender acks
> >> >that it has received the notification)
> >> >
> >> >Figuratively:
> >> >=============
> >> >So, assume there was a new "strict_order" flag in the BHS.
> >> >In the figure below, braces shows value of (cmdSN, strict_order)
> >> >
> >> >Initiator                       Target
> >> >---------                       -------
> >> >
> >> >simple cmd (cmdSN=100, strict_order=0) ->
> >> >simple cmd (101, 0) ->
> >> >simple cmd (102, 0) ->
> >> >
> >> >   these may get reordered or have digest errors ->
> >> >
> >> >                        target executes as they arrive
> >> >                             exec simple cmd(102, 0)
> >> >                             exec simple cmd(100, 0)
> >> >                             exec simple cmd(101, 0)
> >> >
> >> >Now say the initiator wants to send a HOQ task.
> >> >It sets strict_order=1 on all PDUs
> >> >
> >> >ordered cmd (103, 1) ->
> >> >simple  cmd (104, 1) ->
> >> >simple  cmd (105, 1) ->
> >> >                        in case of reordering or digest errors
> >> >                        target must wait & execute in cmdSN order!
> >> >simple  cmd (106, 1) ->
> >> >simple  cmd (107, 1) ->
> >> >
> >> >         <---- now target sends expCmdSN=103
> >> >
> >> >This implies target has seen command(cmdSN=103) and target will do the
> >> >appropriate ordering and delivery to SCSI layer.  This is left to
> >> >the target implementation to tackle.
> >> >
> >> >Initiator checks (expCmdSN >= cmdSN of ordered cmd) and then resets
> >> >strict_order to zero for all subsequent PDUs.
> >> >
> >> >simple  cmd (108, 0) ->
> >> >simple  cmd (109, 0) ->
> >> >
> >> >Other issues:
> >> >=============
> >> >If the basic scheme is ok, then we could later tackle other questions
> >> >"what about multiple ordered commands" and the like..
> >