Re: iSCSI:out-of-order notification proposal

["Mallikarjun C." <cbm@rose.hp.com>]

Sandeep,

It's just not a single SCSI info-bit that the target iSCSI layer should 
look at for the optimization you're proposing - the LUN needs to be 
examined as well since the task attributes are scoped per-LU.

In effect, what you're suggesting is an iSCSI-level per-LU ordering with 
a session-global command sequencing scheme - that is a quite a bit of 
complexity for targets to manage!  

BTW, synch and steering layer is *not* meant to help iSCSI (or SCSI)
process payloads out-of-order.  It is targetted only for out-of-order
data placements to reduce implementation costs, as the current draft 
clearly states.

Regards.
--
Mallikarjun 


Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions Organization
MS 5668	Hewlett-Packard, Roseville.
cbm@rose.hp.com


>Answering two responses in one email.  This would be mighty 
>easier if there was a whiteboard around here :-)
>
>> What exactly does it buy you? (that is not already in).
>
>The ability to execute subsequent PDUs in the presence of holes.
>(Holes due to multiple connections OR due to skipping onto next marker)
>
>If the hole is due to a REAL ordering issue (e.g. ORdered task
>was lost) then the target will pause for the hole to fill.
>
>But if the hole is due to simple task, then the target
>can continue execution of the new commands it is receiving.
>
>Currently, target will *always* pause due to cmdSN ordering,
>which is somewhat like a primitive microprocessor which wont 
>reorder its instruction stream, though its feasible.
>
>> 
>> And FWIW David (B) raised a question related to the requirements doc.
>
>My mistake, the rephrasing of requirements seemed to match the intent 
>of previous discussions on why total cmdSN ordering is overkill.
>That was the only point i was trying to reinforce.
>
>> 
>> Julo
>
>
>
>"Mallikarjun C." wrote:
>> 
>> Sandeep,
>> 
>> Some comments on your proposal.
>> 
>> - The "strict_order" flag that you mention appears to carry
>>   information that's already contained by the SCSI task attributes
>>   (ATTR field).
>
>But now the information is also contained in PDUs following it.
>(e.g. A simple command PDU may also have strict_order=1)
>
>If an Ordered task PDU is lost, the target *knows* there is a REAL
>ordering issue (later PDUs have info) so it must wait for the hole 
>to be filled.
>
>> 
>> - This proposal requires the target iSCSI layer to look at more
>>   variables (SCSI-level information in that) for making its
>>   sequencing decisions.
>
>Only one more bit.. to decide if the command can be passed
>to SCSI layer now or later in cmdSN sequence.  
>
>To build an efficient SCSI transport, some hints would need
>to be passed between layers.
>
>> 
>> - Simple commands (intended to be) received prior to an Ordered
>>   command must be executed before the Ordered one.  If you lose a
>>   Simple command, you must wait for it before you act on the Ordered
>>   that you received out-of-order.
>
>Good pt.  This, I forgot to mention below.  The target must clear
>out all PDUs received with strict-order=0 before commencing 
>execution of PDUs received with strict-order=1.
>
>> 
>> This appears more an implementaion optimization where desired than
>> something we want to get into the draft.  You are proposing that
>> even in the absence of errors, iSCSI should make ordering decisions
>> based on SCSI task attributes.  I disagree with that.
>
>The optimization needs a bit in the PDU, otherwise I wouldnt
>have gone to the trouble of writing this out!  And you would agree
>that there are a bunch of TCP optimizations in various drafts.
>
>Lastly, what is the point in building a sync-and-steering layer to
>skip ahead in the command stream if you cannot execute that next command
>?
>With this little more work, you could as well continue execution.
>
>> --
>> Mallikarjun
>> 
>> Mallikarjun Chadalapaka
>> Networked Storage Architecture
>> Network Storage Solutions Organization
>> MS 5668 Hewlett-Packard, Roseville.
>> cbm@rose.hp.com
>> 
>> >This is a proposal to allow the initiator to inform the target
>> >if out-of-order execution within the command stream is possible.
>> >
>> >The target execution can rotate between "in-cmdSN-order" and
>> >"out-of-order" during runtime as informed by the initiator..
>> >
>> >Appreciate comments on subtleties that I may have missed.
>> >
>> >thanks,
>> >-Sandeep
>> >
>> >http://ips.pdl.cs.cmu.edu/mail/msg03152.html
>> >   by Costa provides a good summary of the issue at hand.
>> >
>> >http://ips.pdl.cs.cmu.edu/mail/msg04255.html
>> >  David provides the "new" reqts.  In particular, this one
>> >
>> >   > MUST specify the ability to preserve ordered delivery
>> >   > of SCSI commands even in the presence of transport
>> >   > errors.  A mechanism MUST be provided to allow
>> >   > Initiators and Targets to negotiate this preservation
>> >   > on a per-session or finer granularity basis
>> >
>> >Note :
>> >======
>> >1) This does not rely on SCSI cmdRN, but operates at iSCSI level.
>> >2) Flow control using cmdSN works as designed.
>> >3) This solution is not a per-session negotiation option but can be
>> >   disabled and re-enabled again at "runtime" by the initiator if it
>> >   notices that Ordered/HOQ tasks (or any other need) have entered
>> >   the iSCSI command stream which is being dispatched.
>> >
>> >Problem:
>> >=========
>> >In case of out-of-order arrival or digest errors, it is NOT possible
>> >to know if the initiator had sent an ordered command before the one
>> >which was received.
>> >
>> >Solution:
>> >=========
>> >To notify target of presence of in-flight ordered commands, we set
>> >a flag on *every* PDU following the ordered command *until* the target
>> >moves it expCmdSN above the cmdSN of the ordered command.  The
>> >expCmdSN indicates target has found the ordered command.
>> >
>> >( Those familiar with "ECN over IP" (Floyd, et al) may see this is
>> >similar to how a congestion bit keeps being set until the sender acks
>> >that it has received the notification)
>> >
>> >Figuratively:
>> >=============
>> >So, assume there was a new "strict_order" flag in the BHS.
>> >In the figure below, braces shows value of (cmdSN, strict_order)
>> >
>> >Initiator                       Target
>> >---------                       -------
>> >
>> >simple cmd (cmdSN=100, strict_order=0) ->
>> >simple cmd (101, 0) ->
>> >simple cmd (102, 0) ->
>> >
>> >   these may get reordered or have digest errors ->
>> >
>> >                        target executes as they arrive
>> >                             exec simple cmd(102, 0)
>> >                             exec simple cmd(100, 0)
>> >                             exec simple cmd(101, 0)
>> >
>> >Now say the initiator wants to send a HOQ task.
>> >It sets strict_order=1 on all PDUs
>> >
>> >ordered cmd (103, 1) ->
>> >simple  cmd (104, 1) ->
>> >simple  cmd (105, 1) ->
>> >                        in case of reordering or digest errors
>> >                        target must wait & execute in cmdSN order!
>> >simple  cmd (106, 1) ->
>> >simple  cmd (107, 1) ->
>> >
>> >         <---- now target sends expCmdSN=103
>> >
>> >This implies target has seen command(cmdSN=103) and target will do the
>> >appropriate ordering and delivery to SCSI layer.  This is left to
>> >the target implementation to tackle.
>> >
>> >Initiator checks (expCmdSN >= cmdSN of ordered cmd) and then resets
>> >strict_order to zero for all subsequent PDUs.
>> >
>> >simple  cmd (108, 0) ->
>> >simple  cmd (109, 0) ->
>> >
>> >Other issues:
>> >=============
>> >If the basic scheme is ok, then we could later tackle other questions
>> >"what about multiple ordered commands" and the like..
>