RE: iSCSI: Addition of CmdSN in Data-out PDU

[Robert Snively <rsnively@Brocade.COM>]

Weren't all the other ordering issues associated with 
sessions (like CSN for example)?  Doesn't that interact
with the requirements for ordering of immediate data?
As far as I can see, the ordering becomes significantly
less important at the connection level 
because the session level management
has to take things out of order in any case.

Bob

> -----Original Message-----
> From: Julian Satran [mailto:Julian_Satran@il.ibm.com]
> Sent: Saturday, October 20, 2001 3:18 PM
> To: ips@ece.cmu.edu
> Subject: RE: iSCSI: Addition of CmdSN in Data-out PDU
> 
> 
> 
> Robert,
> 
> The ordering rule is for a connection (not for a session).  It's sole
> purpose is to minimize the chance for a deadlock.
> 
> Regards,
> Julo
> 
> 
>                                                               
>                           
>                     Robert Snively                            
>                           
>                     <rsnively@broc       To:     "'Rod 
> Harrison'"                       
>                     ade.com>              
> <rod.harrison@windriver.com>, Julian          
>                                           
> Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu       
>                     16-10-01 18:02       cc:                  
>                           
>                     Please respond       Subject:     RE: 
> iSCSI:  Addition of CmdSN in  
>                     to Robert             Data-out PDU        
>                           
>                     Snively                                   
>                           
>                                                               
>                           
>                                                               
>                           
> 
> 
> 
> I am not quite sure what meaning "in-order" data has in
> a multi-connection session.  Does it mean that you actually
> expect the data across multiple connections to arrive temporally
> in the same order that is specified by the command sequence
> numbers?  Or does it mean that the data has to be arranged
> in some order by the target session processing?
> 
> Or is it better to step aside from the in-order question
> for data transmission and do as Rob Harrison suggests:
> 
> "As an initiator I would send the command PDU as soon as
> I could if there were no immediate data, then queue the
> DMA for the unsolicited data. Since there could be many
> other DMAs queued ahead of the unsolicited data it is quite
> possible that other commands, both read and write, will
> be sent before the DMA completes."
> 
> This would imply no strict ordering requirement.
> 
> If an ordering requirement is necessary, I submit that it
> is because of the problems with unsolicited data reception,
> not with the actual data order.
> 
> Bob
> 
> > -----Original Message-----
> > From: Rod Harrison [mailto:rod.harrison@windriver.com]
> > Sent: Monday, October 15, 2001 4:06 AM
> > To: Julian Satran; ips@ece.cmu.edu
> > Subject: RE: iSCSI: Addition of CmdSN in Data-out PDU
> >
> >
> > Julian,
> >
> >          I think you might have misunderstood what I meant. 
> I was not
> > advocating we change the spec in favour of relaxing the 
> data ordering
> > requirements. I was just point out there is a problem for the
> > initiator to solve in order maintain correct ordering, and that I
> > believe it is not a terribly difficult one.
> >
> >          - Rod
> >
> > -----Original Message-----
> > From: owner-ips@ece.cmu.edu 
> [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> > Julian Satran
> > Sent: Sunday, October 14, 2001 9:30 PM
> > To: ips@ece.cmu.edu
> > Subject: RE: Addition of CmdSN in Data-out PDU
> >
> >
> >
> > Immediate data are not mandatory (even if enabled). Having 
> one or ten
> > DMAs
> > does not change the fact that they share a TCP "pipe" and a decent
> > transmitter will ship the TCP data in order - otherwise it 
> just throws
> > his
> > problems in the receiver's lap. Our additional requirement on order
> > should
> > make no difference.  And it would be time to change the name of the
> > thread.
> >
> > Julo
> >
> >
> >
> >
> >
> >                     "Rod Harrison"
> >                     <rod.harrison@wind       To:     John 
> Hufferd/San
> > Jose/IBM@IBMUS, "Robert
> >                     river.com>                Snively"
> > <rsnively@Brocade.COM>
> >                     Sent by:                 cc:
> > <somesh_gupta@silverbacksystems.com>,
> >                     owner-ips@ece.cmu.        "BURBRIDGE,MATTHEW
> > (HP-UnitedKingdom,ex2)"
> >                     edu
> > <matthew_burbridge@hp.com>, "'Binford, Charles'"
> >                                               <CBinford@pirus.com>,
> > <ips@ece.cmu.edu>
> >                                              Subject:     
> RE: Addition
> > of CmdSN in Data-out PDU
> >                     13-10-01 13:12
> >                     Please respond to
> >                     "Rod Harrison"
> >
> >
> >
> >
> >
> >
> >            Even more common would be the DMA latency. As an 
> initiator
> > I
> > would
> > send the command PDU as soon as I could if there were no immediate
> > data, then queue the DMA for the unsolicited data. Since there could
> > be many other DMAs queued ahead of the unsolicited data it is quite
> > possible that other commands, both read and write, will be 
> sent before
> > the DMA completes.
> >
> >            You are spot one when you say the delay in obtaining the
> > unsolicited
> > data may be large and variable. If there are multiple DMA engines it
> > does become a challenge for the initiator to ensure the correct
> > ordering of the unsolicited data. Note that since there may be
> > multiple unsolicited data PDUs in the same burst, and those may be
> > filled with separate DMAs the ordering problem propagates down to
> > within individual commands.
> >
> >            I don't believe it is a particularly difficult 
> problem for
> > the
> > initiator to solve, but I do agree things would be easier for the
> > initiator if the ordering requirement were removed. And 
> more so if the
> > requirement to send in dataSN order were also removed, but this is
> > probably too much pain for the target.
> >
> >            - Rod
> >
> > -----Original Message-----
> > From: owner-ips@ece.cmu.edu 
> [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> > John Hufferd
> > Sent: Saturday, October 13, 2001 1:01 AM
> > To: Robert Snively
> > Cc: 'somesh_gupta@silverbacksystems.com'; BURBRIDGE,MATTHEW
> > (HP-UnitedKingdom,ex2); 'Binford, Charles'; ips@ece.cmu.edu
> > Subject: RE: Addition of CmdSN in Data-out PDU
> >
> >
> >
> > Robert,
> > I think that an Initiator being able to send a waiting Read command,
> > without having to wait for many large write segments -- 
> that are being
> > sent
> > (as unsolicited data) -- is very useful.  And that would mean, the
> > unsolicited data is waiting to be sent until the Read Commands are
> > sent.
> > This might be a very frequent case.
> >
> > .
> > .
> > .
> > John L. Hufferd
> > Senior Technical Staff Member (STSM)
> > IBM/SSG San Jose Ca
> > Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> > Home Office (408) 997-6136
> > Internet address: hufferd@us.ibm.com
> >
> >
> > Robert Snively <rsnively@Brocade.COM>@ece.cmu.edu on 10/12/2001
> > 03:56:06 PM
> >
> > Sent by:  owner-ips@ece.cmu.edu
> >
> >
> > To:   "'somesh_gupta@silverbacksystems.com'"
> >       <somesh_gupta@silverbacksystems.com>, "BURBRIDGE,MATTHEW
> >       (HP-UnitedKingdom,ex2)" <matthew_burbridge@hp.com>, "'Binford,
> >       Charles'" <CBinford@pirus.com>, ips@ece.cmu.edu
> > cc:
> > Subject:  RE: Addition of CmdSN in Data-out PDU
> >
> >
> >
> > I have two small questions that would help me understand this.
> >
> > How do you know that unsolicited data is expected?  By nature,
> > unsolicited data is received as a "maximum surprise" which can
> > only be determined after unpacking and parsing at least a
> > part of the command structure, possibly even including the
> > SCSI command (although there are some hints contained in the
> > command header information).
> >
> > How can you constrain a generic system to posting the
> > unsolicited data in the same order that the commands were
> > emitted?  In general, I would have expected the system to
> > be emitting command followed immediately by the corresponding
> > unsolicited data.  If that is not the case, it means that there
> > was a delay in obtaining the unsolicited data for transfer and
> > that the delay was sufficient to allow the insertion of commands.
> > If the delay is that large (and probably variable), the enforcement
> > of transfer of unsolicited data in the same order as the
> > commands are emitted seems to me to be a significant challenge,
> > and certainly shouldn't be required as normal behavior.  While it
> > would make things simpler for targets (already challenged by
> > unsolicited data), it seems to me that it would make things
> > much more complex for initiators.
> >
> > Bob
> >
> > > -----Original Message-----
> > > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com]
> > > Sent: Friday, October 12, 2001 2:51 PM
> > > To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); 'Binford, Charles';
> > > ips@ece.cmu.edu
> > > Subject: RE: Addition of CmdSN in Data-out PDU
> > >
> > >
> > > Matthew,
> > >
> > > Since the unsolicited data does not follow the
> > > command, you need the link list. But since the
> > > unsolicited data must be sent in the same order
> > > as the commands, a link list is enough.
> > >
> > > Let us say that you have 8 commands. The ones
> > > for which we expect unsolicted data are marked
> > > as Cnn(ud). And I have marked the unsolicted
> > > data PDUs as UD(nn). The (nn) with data implies
> > > that it is implicit and not actually carried with
> > > the PDU itself.
> > >
> > > C01(ud) C02(ud) C03(ud) C04 C05 C06(ud) UD(01) --->
> > > --> C07(ud) UD(02) UD(03) D04 D04 D05 D05 UD(06) --->
> > > --> UD(07) C08(ud) UD(08) --->
> > >
> > > After the target receives the command C01, C02, and C03
> > > for which it expects unsolicited data, it puts them in
> > > a link list. It also receives C04 and C05 for which
> > > unsolicited data is not expected and they don't go
> > > on the list. It then receives C06 for which unsolicited
> > > data is expected, and it is added to then end of the list.
> > > Then an unsolicited data PDU is received. It must go
> > > with the command at the head of the list which is C01.
> > > Use the ITT to make sure and you can then take C01 off
> > > the list and so on.
> > >
> > > Somesh
> > >
> > >
> > > > -----Original Message-----
> > > > From: owner-ips@ece.cmu.edu
> > > [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> > > > BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
> > > > Sent: Friday, October 12, 2001 7:53 AM
> > > > To: 'Binford, Charles'; ips@ece.cmu.edu
> > > > Subject: RE: Addition of CmdSN in Data-out PDU
> > > >
> > > >
> > > > Charles,
> > > >
> > > > As you have described the spec states that "that
> > > unsolicited data MUST be
> > > > sent in the same order as the commands".  This is not 
> the same as
> > > > unsolicited data must follow the command associated with
> > > it:  For example:
> > > >
> > > > (Cx = SCSI Command PDU, Dx = The unsolicited data PDUs. The
> > > x in all the
> > > > example can be the ITT. It is not the CmdSN.
> > > >
> > > > This is allowed:
> > > >
> > > >   C1 C2 C3 D1 D2 C4 D3 D4
> > > >
> > > > and the target will have to use the ITT to associate the
> > > data with the
> > > > command.
> > > >
> > > > Matthew
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Binford, Charles [mailto:CBinford@pirus.com]
> > > > Sent: Friday, October 12, 2001 2:50 PM
> > > > To: ips@ece.cmu.edu
> > > > Subject: RE: Addition of CmdSN in Data-out PDU
> > > >
> > > >
> > > > I have not verify version 8 is still the same, but version 07-97
> > > > had a rule
> > > > that unsolicited data MUST be sent in the same order as the
> > > commands.
> > > >
> > > > Thus, there is no need for a search on the ITT.  The target
> > > just needs to
> > > > keep of linked list of I/Os waiting on unsolicited data.
> > > New commands are
> > > > added to the tail, any unsolicited data *should* be associated
> > > > with the I/O
> > > > at the head of the list.  The ITT is used as a sanity check and
> > > > you're done.
> > > >
> > > > What am I missing?
> > > >
> > > > Charles Binford
> > > > Pirus Networks
> > > > 316.315.0382 x222
> >
> >
> >
> >
> >
> >
> >
> >
> 
> 
> 
> 
>