RE: iSCSI: changing MaxPDUDataLength

[Eddy Quicksall <eddy_quicksall@ivivity.com>]

Mallikarjun,

This is not a case of arguing ... we are simply trying to understand each
others point over EMAIL, which is difficult at best. Please try to work with
me instead of against me. I could be wrong but I just need a little more
cooperation instead of using the overworked terms like "if you will just
read ...".

You went into enough detail in one EMAIL to show that it is a messy job.
When a solution is too messy and hard to explain, it is usually a clue that
something needs to be simplified ... and that is what I am after.

So far, I have not been able to catch the reason why we could not simply
specify that the initiator must idle the commands before issuing a new size.
Julian hinted that it would be a performance hit but I don't believe that
will be a performance hit because it should be rare. Please explain why it
would be a performance hit.

I would also like to know why the SNACK just doesn't simply ask for an
offset and a data length because that would simplify everything. Could you
please explain that?

You mention data-out below but I'm not talking about data-out, I'm talking
about data-in.

I don't know what this has to do with a long write. Can you please explain
that?

BTW, I don't have much background in ULPDU so maybe that is the key as to
why the initiator can't idle first. Can you please explain that?

In re-reading your statement below, I am wondering if you understand the
problem ... The problem is that when a SNACK is sent and the PDU sizes have
changed due to MaxPDUDataSegmentLength, then it puts a burden on the target
to compute the proper offset of the BegRun (a.k.a messy).

This is solved by the target in at least two ways:

1) The target can record the last PDU size when the change takes place. But
it would also have to keep track of the number of already completed PDU's
per outstanding command. This is because some commands may have missing
PDU's with the old size and others may have missing PDU's with the new size.
To make matters worse, some commands could have n PDUs already sent and
others could have m PDUs already sent. If you want, I could make up a
diagram of this and send it to you.

2) The target could "force an idle of data-in commands" (by queuing them)
before it responds to the change request. Doing this would probably be no
different than the initiator doing it but it would be easier and less
intrusive for the initiator to do it.


Eddy

-----Original Message-----
From: Mallikarjun C. [mailto:cbm@rose.hp.com]
Sent: Wednesday, June 12, 2002 2:06 PM
To: Eddy Quicksall; Rod Harrison; Julian Satran
Cc: ips@ece.cmu.edu
Subject: Re: iSCSI: changing MaxPDUDataLength


I am afraid this thread is going in circles, let me respond to this message
before I rest my case on this topic.

> I think you have misunderstood something... I'm not suggesting that you
> mandate "no text negotiation till quiesced". I'm suggesting that only when
> Max length changes and then only for the READS. This suggestion is to
> simplify the logic you point out below.
> 
> What are the harmful effects you mention below?

As I earlier hinted twice, the Data-Out PDUs will not be ULPDU-contained
anymore - and that could mean target would need to spend more memory/bus
bandwidth/computation to deal with those till the long write is done. 

I do not understand your specific proposal yet.  If you're arguing that a
special case
be made for reads, a cogent, clear rationale for the special case, coupled
with the 
specifics of your proposal, would help the WG consider this further.  Based
on 
my current understanding of your position, I currently don't see the need
for the 
special case.

Thanks.
--
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668 
Roseville CA 95747
cbm@rose.hp.com

> 
> Eddy
> 
> -----Original Message-----
> From: Mallikarjun C. [mailto:cbm@rose.hp.com]
> Sent: Tuesday, June 11, 2002 8:54 PM
> To: Eddy Quicksall; Julian Satran
> Cc: ips@ece.cmu.edu
> Subject: Re: iSCSI: changing MaxPDUDataLength
> 
> 
> > Can you suggest how this would be handled otherwise?
> 
> I earlier said I will not get into implementation details, I will 
> however provide a generic answer.
> 
> Given that the changed PDU size is not specific to one command,
> I see the history of "past max PDU size(s)" as a connection attribute.  
> All you need on a per-task basis is really the value of one DataSN 
> *where* it had changed.
> 
> We could further debate how to deal with multiple number of PDU size 
> changes during the life of an I/O - but I'm reluctant to be involved in
that
> debate because a) it's most unlikely, and b) there's no hard requirement
> that 
> the target must always satisfy SNACKs under extreme conditions, and
finally
> c) it's too implementation-specific to be discussed on the ips reflector.
> 
> Finally, I'd like to remind you that mandating "no text negotiation till
> quiesced"
> has harmful effects on targets dealing with long writes and wanting ULPDU-
> containment - whose case you may be arguing.
> --
> Mallikarjun
> 
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions
> Hewlett-Packard MS 5668 
> Roseville CA 95747
> cbm@rose.hp.com
> 
> 
> ----- Original Message ----- 
> From: "Eddy Quicksall" <eddy_quicksall@ivivity.com>
> To: "Mallikarjun C." <cbm@rose.hp.com>; "Julian Satran"
> <Julian_Satran@il.ibm.com>
> Cc: <ips@ece.cmu.edu>
> Sent: Tuesday, June 11, 2002 4:45 PM
> Subject: RE: iSCSI: changing MaxPDUDataLength
> 
> 
> > Consider the following (using Julian's suggestion):
> > 
> > -> cmd 1  
> > -> req to change to length 2
> > -> cmd 2
> > 
> >       <- stat 1
> > <- data 2, PDU 0, Length 1
> > 
> >    -> cmd 3, ack 1 so change the length
> > 
> > <- data 2, PDU 1, Length 2
> > 
> > -> SNACK data 2, PDU 1
> > 
> > So we have to calculate the offset for Data 2, PDU 1 using old length
and
> > send data after that offset using new length. That means keeping track
of
> > PDU lengths which I would like to avoid.
> > 
> > Or, the target would have to force idling of commands before it gives
the
> > response to the change.
> > 
> > Can you suggest how this would be handled otherwise?
> > 
> > Eddy
> > 
> > -----Original Message-----
> > From: Mallikarjun C. [mailto:cbm@rose.hp.com]
> > Sent: Tuesday, June 11, 2002 6:57 PM
> > To: Eddy Quicksall; Julian Satran
> > Cc: ips@ece.cmu.edu
> > Subject: Re: iSCSI: changing MaxPDUDataLength
> > 
> > 
> > Eddy,
> > 
> > I am not sure if you're agreeing with me or not... but let me add some
> > comments.
> > 
> > I don't expect the change in PDU size to happen frequently - no change
> > in what I earlier said.
> > 
> > You are making an assumption that a target must record the PDU size for
> > every
> > PDU  if SNACKs are to be satisfied across changed
> MaxRecvDataSegmentLength.
> > 
> > As Julian earlier said, you don't have to.  I will not go into
> > implementation details, but
> > I believe just the value of the DataSN from which the new size is
> effective
> > should 
> > be all that's needed.  Besides, the spec guarantees that only
RunLength=0
> is
> > used 
> > on any SNACK after the change.
> > 
> > You are also implying that targets can declare their changed
> > MaxRecvDataSegmentLength
> > anytime just for writes.  There are two problems with this - a) reads
and
> > writes may both
> > be active in the typical case on an iSCSI connection, and b) a target
can
> > declare its changed
> > value only if an initiator is allowed to start the Text Request (and I
> > thought you were suggesting
> > that we explicitly disallow that).
> > 
> > Hope that clears it up.
> > --
> > Mallikarjun
> > 
> > Mallikarjun Chadalapaka
> > Networked Storage Architecture
> > Network Storage Solutions
> > Hewlett-Packard MS 5668 
> > Roseville CA 95747
> > cbm@rose.hp.com
> > 
> > 
> > ----- Original Message ----- 
> > From: "Eddy Quicksall" <eddy_quicksall@ivivity.com>
> > To: "Mallikarjun C." <cbm@rose.hp.com>; "Julian Satran"
> > <Julian_Satran@il.ibm.com>
> > Cc: <ips@ece.cmu.edu>
> > Sent: Tuesday, June 11, 2002 3:18 PM
> > Subject: RE: iSCSI: changing MaxPDUDataLength
> > 
> > 
> > > You are correct and that is exactly how I have had to code it. With
fast
> > > path and slow path code split between processors, it becomes even
worse.
> > > 
> > > When I read your comments on why the PDU size could change (from way
> back
> > in
> > > March I think), I got did not get the impression that it would happen
> very
> > > often and that in fact it may only happen when you are maintaining
> > > allegiance to another connection.
> > > 
> > > If it is something that is not going to happen very often, then it
would
> > be
> > > so much easier to have the initiator idle the commands first (all it
> > really
> > > needs to do is idle the READ commands). If it is going to change so
> often
> > > that idling would be degrading, I suspect that just doing the
> negotiation
> > > that often would itself be degrading.
> > > 
> > > And, be aware that the target will probably idle the R commands.
> > Otherwise,
> > > it will have to track PDU sizes and that would be really
> > > "counter-productive" (especially when there should be no SNACKs on a
> good
> > > connection anyway).
> > > 
> > > I don't see any problem with the target negotiating its size at any
> time.
> > > And the initiator would not have to idle the WRITE or no-data
commands.
> > > 
> > > Eddy
> > > 
> > > -----Original Message-----
> > > From: Mallikarjun C. [mailto:cbm@rose.hp.com]
> > > Sent: Tuesday, June 11, 2002 3:35 PM
> > > To: Eddy Quicksall; Julian Satran
> > > Cc: ips@ece.cmu.edu
> > > Subject: Re: iSCSI: changing MaxPDUDataLength
> > > 
> > > 
> > > Eddy,
> > > 
> > > I am not clear on why you think the target code becomes messy on 
> > > a PDU size change.  The last para in section 4.4 states the following
-
> > > 
> > > "Parameters negotiated by a text exchange negotiation sequence become
> > > effective only after the negotiation sequence is completed."
> > > 
> > > Since the target gets to complete a text negotiation with a Text
> Response
> > > PDU, it can time the applicability of a changed
> MaxRecvDataSegmentLength.
> > > 
> > > Requiring that an initiator must idle the connection before changing
> this
> > > parameter, IMHO, is counter-productive when there's a dynamic PMTU
> > > degradation in the presence of long-running commands (writes in
> > particular).
> > > Once the initiator starts the renegotiation, target would ideally want
> to
> > > quickly 
> > > renegotiate its receive size as well to receive ULPDU-contained
> segments.
> > > 
> > > In short, seems to me that the draft is saying what you would like.
> > > --
> > > Mallikarjun
> > > 
> > > Mallikarjun Chadalapaka
> > > Networked Storage Architecture
> > > Network Storage Solutions
> > > Hewlett-Packard MS 5668 
> > > Roseville CA 95747
> > > cbm@rose.hp.com
> > > 
> > > 
> > > ----- Original Message ----- 
> > > From: "Eddy Quicksall" <eddy_quicksall@ivivity.com>
> > > To: "Julian Satran" <Julian_Satran@il.ibm.com>
> > > Cc: <ips@ece.cmu.edu>
> > > Sent: Tuesday, June 11, 2002 7:56 AM
> > > Subject: RE: iSCSI: changing MaxPDUDataLength
> > > 
> > > 
> > > > I think it should be a requirement because if it is not, all target
> code
> > > > will have to have the messy code. Or, we could say that if it is not
> > idle
> > > > commands, then the target may reject it.
> > > >  
> > > > Eddy
> > > > 
> > > > -----Original Message-----
> > > > From: Julian Satran [mailto:Julian_Satran@il.ibm.com]
> > > > Sent: Tuesday, June 11, 2002 9:11 AM
> > > > To: Eddy Quicksall
> > > > Cc: ips@ece.cmu.edu; pat_thaler@agilent.com
> > > > Subject: RE: iSCSI: changing MaxPDUDataLength
> > > > 
> > > > 
> > > > 
> > > > That is a fair request - we may slip in a recommendation to that
> effect
> > > (in
> > > > chapter 11?) 
> > > > 
> > > > Julo 
> > > > 
> > > > 
> > > > 
> > > > Eddy Quicksall <eddy_quicksall@ivivity.com> 
> > > > 
> > > > 
> > > > 06/11/2002 04:28 AM 
> > > > Please respond to Eddy Quicksall 
> > > > 
> > > > 
> > > >         
> > > >         To:        Julian Satran/Haifa/IBM@IBMIL 
> > > >         cc:        ips@ece.cmu.edu, pat_thaler@agilent.com 
> > > >         Subject:        RE: iSCSI: changing MaxPDUDataLength 
> > > > 
> > > >        
> > > > 
> > > > 
> > > > How about if we say that an initiator must not change the
> MaxPDUDataSize
> > > > unless it first idles the commands (at least the ones with the R
bit)
> or
> > > if
> > > > ErrorRecoveryLevel==0? 
> > > >   
> > > > That would simplify target code greatly and would meet the design
> goals;
> > > and
> > > > since it should be rare to change it, it would be of little impact
to
> > idle
> > > > the commands for a split second. 
> > > >   
> > > >   
> > > > Eddy 
> > > > -----Original Message-----
> > > > From: Julian Satran [mailto:Julian_Satran@il.ibm.com]
> > > > Sent: Monday, June 10, 2002 8:06 PM
> > > > To: Eddy Quicksall
> > > > Cc: ips@ece.cmu.edu; pat_thaler@agilent.com
> > > > Subject: RE: iSCSI: changing MaxPDUDataLength
> > > > 
> > > > 
> > > > I said only that when you change length you can ask for all PDUs
after
> > the
> > > > ack! (no need to keep a tall - the old offsets are good up to the
> hole).
> > 
> > > > 
> > > > Julo 
> > > > 
> > > > 
> > > > Eddy Quicksall <eddy_quicksall@ivivity.com> 
> > > > 
> > > > 
> > > > 06/11/2002 12:32 AM 
> > > > Please respond to Eddy Quicksall 
> > > > 
> > > >         
> > > >        To:        Julian Satran/Haifa/IBM@IBMIL 
> > > >        cc:        ips@ece.cmu.edu, pat_thaler@agilent.com 
> > > >        Subject:        RE: iSCSI: changing MaxPDUDataLength 
> > > > 
> > > >       
> > > > 
> > > > 
> > > > 
> > > > Julian, 
> > > >  
> > > > Another problem here is that the target has to calculate the offset
> from
> > > the
> > > > DataSN #. And as BegRun can be any value. E.g., BegRun=4 and
> RunLngth=0
> > > > means starting DataSN=4, send all the data for that sequence. 
> > > >  
> > > > I think it would be a performance hit and waist of memory to keep a
> > tally
> > > of
> > > > all PDU sizes just for an occasional SNACK. 
> > > >  
> > > > It's not that it can't be done ... it is more that it is messy and I
> > think
> > > > there is a solution that will satisfy the design requirements but
keep
> > the
> > > > software simpler. 
> > > >  
> > > > Eddy 
> > > > -----Original Message-----
> > > > From: Julian Satran [mailto:Julian_Satran@il.ibm.com]
> > > > Sent: Saturday, June 08, 2002 10:16 PM
> > > > To: Eddy Quicksall
> > > > Cc: ips@ece.cmu.edu; owner-ips@ece.cmu.edu; pat_thaler@agilent.com
> > > > Subject: RE: iSCSI: changing MaxPDUDataLength
> > > > 
> > > > 
> > > > That is not completely accurate. 
> > > > The only problem is when PDU size decreases and then SNACK must be
for
> > all
> > > > data. 
> > > > Target can also keep a mapping of numbers/to offsets (the list
should
> be
> > > > small and if it gets long ask for ack (A-bit). 
> > > > 
> > > > Julo 
> > > > 
> > > > Eddy Quicksall <eddy_quicksall@ivivity.com> 
> > > > Sent by: owner-ips@ece.cmu.edu 
> > > > 
> > > > 
> > > > 06/08/2002 10:54 PM 
> > > > Please respond to Eddy Quicksall 
> > > > 
> > > >         
> > > >       To:        pat_thaler@agilent.com 
> > > >       cc:        ips@ece.cmu.edu 
> > > >       Subject:        RE: iSCSI: changing MaxPDUDataLength 
> > > > 
> > > >      
> > > > 
> > > > 
> > > > 
> > > > 
> > > > Thanks,
> > > > 
> > > > As a target, I won't be able to let it change until all of the
> > outstanding
> > > > commands are finished (running with ErrorRecoveryLevel>=1). This is
> > > because
> > > > I must use the PDU size to compute the offset from a SNACK's
> > > > BegRun/RunLength.
> > > > 
> > > > So, I plan to not give the text response for a MaxPDURecvDataLength
in
> > FFP
> > > > until I get back an ExpStatSN == next StatSN.
> > > > 
> > > > This will cause a delay of unknown time before the PDU size can
> actually
> > > > change ... do you see any problem with this?
> > > > 
> > > > Eddy
> > > > 
> > > > -----Original Message-----
> > > > From: pat_thaler@agilent.com [mailto:pat_thaler@agilent.com]
> > > > Sent: Friday, June 07, 2002 1:13 PM
> > > > To: eddy_quicksall@ivivity.com; ips@ece.cmu.edu
> > > > Subject: RE: iSCSI: changing MaxPDUDataLength
> > > > 
> > > > 
> > > > Eddy,
> > > > 
> > > > If one uses a message sync and steering that relys on the transport
> > > segments
> > > > carrying a full PDU, e.g. TCP ULP Framing Protocol (TUF), then if
the
> > path
> > > > MTU changes one would want to change the PDU data length to fit the
> new
> > > path
> > > > MTU.
> > > > 
> > > > Pat
> > > > 
> > > > -----Original Message-----
> > > > From: Eddy Quicksall [mailto:eddy_quicksall@ivivity.com]
> > > > Sent: Wednesday, June 05, 2002 12:24 PM
> > > > To: ips@ece.cmu.edu
> > > > Subject: iSCSI: changing MaxPDUDataLength
> > > > 
> > > > 
> > > > Does anybody know a case where it is necessary to support a new PDU
> data
> > > > length during full feature phase?
> > > > 
> > > > Eddy
> > > > mailto: Eddy_Quicksall@iVivity.com
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > 
>