RE: iSCSI: Out Of Sequence due to null sequence with multipleconnections.

["Douglas Otis" <dotis@sanlight.net>]

Charles,

My concern was to avoid creating errors within the transport then seen in
the SCSI layer.  Unlike FCP, there is a connection sequencer and there are
no strict and short fabric time limits.  As this scheme can get stuck, an
ability for a driver to extricate the situation while ensuring the state of
the target remains deterministic was my concern.  Should commands get
rejected, that same driver at the initiator can resend them unless the
intent of the "Reject Pending PDUs" flagged command was to abort these stuck
commands.  It would be the initiator that would make that determination and
not the middle box.  I would hope this type of behavior would be used only
in the situation of an apparent target failure.  The task attributes within
the command would be used at the target.

iSCSI should only make an effort to ensure commands get there in the proper
sequence.  My recommendation is far simpler than your suggestion.  Every
command receives a sequence number.  If this command is to bypass any
commands pending being sent to the target within the middle box, these
pending commands are rejected.  This would also apply to commands still in
flight.  Unsigned arithmetic to handle the test of what is prior is not
difficult nor does it require a timer.  Filtering out duplicates and waiting
would be a greater effort.  In this manner, there are no sequence holes and
you have a simple way of purging a queue heading into the target.  The
commands work only at the target and not within the sequencing queue.

Doug

> I sent this Friday, but never saw it come back, so I'm resending.....
>
> Charles Binford
> Pirus Networks
> 316.315.0382 x222
>
>
> Doug, If I'm hearing you correct, you are saying process the task
> managment
> command immediately, but toss any commands waiting for a CmdSN hole to be
> filled.   The problem with this approach, is there are many different
> flavors of task managemet commands and not all effect all commands.  The
> iSCSI layer would have to duplicate the SCSI layer effort of parsing the
> task managment request and the LU/tags of all commands it comtemplates
> aborting.  I don't like that solution.
>
> David,  I don't think the current state of things is acceptable
> as you imply
> with your final statement of 'I'm not sure anything needs to be changed.'
>
> FC solves this problem by requiring the initiator FCP layer
> (note: FCP, not
> SCSI)to send an ABTS for every exchange 'in an ambiguous state' after any
> task management functin.  It is ugly, but is covers the hole of the
> initiator not knowing for sure which commands were properly aborted and
> which were not.
>
> I have suggested in the past (although, from the response I've gotten, I
> think I have been misunderstood) that the iSCSI initiator send all task
> management functions down all connections of a session.  I'm NOT saying
> duplicate the task management function.  The distinction being only one
> instance of the task management function should be passed up to the SCSI
> layer on the target side.
>
> To address the issue of the task management function being delayed because
> of a slow connection I suggest the following:  upon receipt of the first
> task management function request on any connection, the iSCSI target shall
> start a relatively short timer (I'm thinking of a range from a
> few 100 ms to
> a couple of seconds).  As soon as the task management function is received
> on all connections, cancel the timer and pass a single instance of the
> request up to the SCSI layer.  If the timer expires, close the
> connection(s)
> the task management function(s) did not arrive on and pass up a single
> instance of the request to the SCSI layer.
>
> Charles Binford
> Pirus Networks
> 316.315.0382 x222
>
>
> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> Douglas Otis
> Sent: Tuesday, March 27, 2001 9:42 AM
> To: julian_satran@il.ibm.com; ips@ece.cmu.edu
> Subject: RE: iSCSI: Out Of Sequence due to null sequence with
> multipleconnections.
>
>
> Julian,
>
> It could be that a "stuck" command in flight or at the sequencer is the
> reason for the task management which is made problematic by the null
> sequence.  Even if implementers are aware of this problem, there is not a
> good solution until null sequence is removed.  Making a flag to indicate
> "immediate" or perhaps "reject prior pending iSCSI commands" is a means of
> ensuring the initiator remains in control with respect to all situations.
> This ensures the initiator is aware of the state of the target
> and sequence
> of commands can be maintained if required.  Be careful about
> being too disk
> centric.  Treating these SN numbers as unsigned allows a simple means of
> tracking.  Here is an example explaination:
>
>    Comparisons and arithmetic on SNs in this document SHOULD use Serial
>    Number Arithmetic as defined in [RFC1982] where SERIAL_BITS = 32.
>
> Doug
>
> > David,
> >
> > Your summary is correct. And (except for a minor point) it is
> all a matter
> > of target implementation.
> > SCSI is not a "completely layered" stack and others have gone
> so far as to
> > do task cancellation by an action at link layer (parallel and fiber).
> > There might be 2 funny side-effects though if an implementer chooses to
> > cancel "holes" (commands in flight on other connections):
> >
> > 1)the cancelled command is a another task management command
> (there can be
> > only one active but what it the one active gets stucked?)
> > 2)(academic I admit) the cancelled command arrives after a wrap
> around in
> > command sequencing; this is a bit harder (although not
> impossible) to fix
> > in the implementation
> >
> > Implementers should be aware of those side effects.
> >
> > Regards,
> > Julo
> >
> >
> >
> > Black_David@emc.com on 26/03/2001 18:57:48
> >
> > Please respond to Black_David@emc.com
> >
> > To:   dotis@sanlight.net
> > cc:   ips@ece.cmu.edu
> > Subject:  RE: iSCSI: Out Of Sequence due to null sequence with
> > multipleconn
> >       ections.
> >
> >
> [stuff deleted]
> >
> > (4) is hard.  One SCSI task management command generates one
> > response.  That response can either be generated immediately
> > (command arrives, is passed to SCSI, SCSI does its thing) or
> > at the right point in the sequence (command arrives, is
> > sequenced by iSCSI, passed to SCSI at the right point in the
> > sequence, and SCSI does its thing), but NOT both.  As things
> > currently stand, having a task management command apply to
> > in-flight commands requires sending the task management
> > command for ordered delivery - so if it's desired to have
> > the task management command take immediate effect and also
> > catch everything in flight, it's going to have to be sent
> > twice.  I'm not enthusiastic about the idea of the task
> > management command taking immediate effect but delaying the
> > response until everything in flight that might be affected
> > arrives, as I suspect the Initiator would like to know what
> > happened sooner rather than later.
> >
> > (3) is "interesting".  The results of applying a SCSI task
> > management command to a SCSI operation are known only to
> > SCSI, and hence asking that a command stuck in the iSCSI
> > sequencer be affected immediately by a task management
> > command is asking that the task management command have
> > the side effect of changing some of the commands it affects
> > to immediate delivery so that it can immediately do its
> > (SCSI) thing to them.  I wouldn't want to mandate this,
> > nor would I want to prohibit it, BUT ... if the above
> > discussion of in-flight commands is correct, I would
> > observe that the application on the Initiator side
> > can't tell the difference between commands that are in-flight
> > vs. waiting for something in-flight on another connection,
> > and hence is going to have to issue the task management
> > command for ordered delivery if it wants to affect operations
> > in either place (and issue a second copy if it wants
> > immediate action).
> >
> > The upshot is that, aside from a longer discussion of this
> > issue, I'm not sure anything needs to be changed.  Comments?
> >
> > Thanks,
> > --David
> >
> > ---------------------------------------------------
> > David L. Black, Senior Technologist
> > EMC Corporation, 42 South St., Hopkinton, MA  01748
> > +1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
> > black_david@emc.com       Mobile: +1 (978) 394-7754
> > ---------------------------------------------------
> >
> >
> >
> >
> >
>
>