RE: ISCSI: Unsolicited data in draft v12

["John Hufferd" <hufferd@us.ibm.com>]

What I thought I said is that it should be just as efficient to send a
normal (non immediate) unsolicited data-out PUD, following the immediate
data, as it is to wait for an R2T and then send the data-out PDU, and
probably a lot more efficient.  So I do not see why anyone would do other
then that.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: hufferd@us.ibm.com


"Rod Harrison" <rod.harrison@windriver.com> on 04/09/2002 03:39:15 PM

To:    John Hufferd/San Jose/IBM@IBMUS
cc:    "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)"
       <matthew_burbridge@hp.com>, "'Julian Satran'"
       <Julian_Satran@il.ibm.com>, <ips@ece.cmu.edu>,
       <owner-ips@ece.cmu.edu>
Subject:    RE: ISCSI: Unsolicited data in draft v12




 My reading of the new rule was that it covered both immediate and
non-immediate
unsolicited data. I think though the logic in my example would still apply,
there is probably an efficiency gain in sending all the data in single
solicited
DATA-OUT versus 8k in an unsolicited DATA-OUT and 4k in a solicited
DATA-OUT.
Just thinking in terms of DMA operations on the initiator HBA I suspect
there is
an advantage to a single DMA for 12k over one for 8k followed by one for
4k. I
am assuming here that the initiator can not speculatively DMA ahead because
of
buffer space constraints.

 John, I'm not quite sure what you meant in your second sentence. Are you
saying
even with the new rule the initiator can choose to not send immediate data
even
if it has been negotiated to be available?

 - Rod

-----Original Message-----
From: John Hufferd [mailto:hufferd@us.ibm.com]
Sent: Tuesday, April 09, 2002 11:20 PM
To: Rod Harrison
Cc: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); 'Julian Satran';
ips@ece.cmu.edu; owner-ips@ece.cmu.edu
Subject: RE: ISCSI: Unsolicited data in draft v12



I understand what you said, however, I thought the issue was for non
immediate unsolicited data.  The case you made is normal, I think, for the
choice for immediate vrs non immediate (either R2T or other unsolicited
data).

If the Initiator has agreed to use non immediate unsolicited data then it
is not clear, using your example, why one would not send the data via a
normal unsolicited Data-Out PDU, when ready.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: hufferd@us.ibm.com


"Rod Harrison" <rod.harrison@windriver.com> on 04/09/2002 12:28:27 PM

To:    John Hufferd/San Jose/IBM@IBMUS, "BURBRIDGE,MATTHEW
       (HP-UnitedKingdom,ex2)" <matthew_burbridge@hp.com>
cc:    "'Julian Satran'" <Julian_Satran@il.ibm.com>, <ips@ece.cmu.edu>,
       <owner-ips@ece.cmu.edu>
Subject:    RE: ISCSI: Unsolicited data in draft v12




 As John has guessed my thinking, at least in part, was of dealing with
congestion in the HBA. I can imagine there might be times when the
initiator receives a command and cannot provide buffer space to accommodate
the unsolicited data, in which case it might be able to ship the command
and deal with the data when the target sends R2Ts.

 I also think there might be efficiency gains to be had by allowing the
initiator to ship commands without immediate data in some instances.
Consider the case where MaxRecvPduSize for the target is 64k and
FirstBurstSize is 8k, we've seen this sort of thing quite a bit at
plugfests. If the initiator receives a SCSI command with a transfer length
of 12k it is probably more efficient to ship the command immediately and
then send the whole payload in one DATA-OUT in response to a single R2T
from the target, than to ship 8k of immediate data and then 4k in response
to an R2T. In fact for any SCSI transfer length greater than FirstBurstSize
and less than MaxRecvPduSize this is probably true.

 I response to Mathew's concern about pre-allocated buffer space being
wasted if unsolicited data isn't sent, I believe this might be more of a
theoretical concern than a practical one. Any buffer space that is set
aside for unsolicited data that can't be used for anything else will be
wasted on every command where the SCSI transfer length is greater than
FurstBurstSize when the task moves into "R2T mode" anyway. For large
transfers that could be a significant proportion of the time. Consider a
modest 12 MB transfer with a generous FurstBurstSize of 1MB, that leaves 11
MB of payload to be transferred under the auspices of R2T, during which
time the unsolicited buffers are unavailable. Multiply that by the command
window size, and then by the number of initiators the target might service
and you quickly end up with an impossible situation.

 - Rod

-----Original Message-----
From: John Hufferd [mailto:hufferd@us.ibm.com]
Sent: Tuesday, April 09, 2002 12:41 AM
To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
Cc: 'Julian Satran'; BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2);
ips@ece.cmu.edu; owner-ips@ece.cmu.edu; Rod Harrison
Subject: RE: ISCSI: Unsolicited data in draft v12



Matthew,
I would have thought that if there is some special buffer space set aside
for the session, whether physical set aside or as a high/low water mark, it
would still be available for other tasks in the session, even if some tasks
do not use it, so I fail to see the true impact.

Perhaps you have seen something or fear something that I do not understand
about why a Initiator would negotiate the unsolicited buffer space
(FirstBustSize) and then not use it, except for when it had some kind of
congestion, or the like.

If you state why you think this would happen, perhaps those persons (Rod)
that want this "MUST" changed to "MAY", should state why they think it is
important to them.

I actually do not see the point of either side.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: hufferd@us.ibm.com


"BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" <matthew_burbridge@hp.com> on
04/08/2002 03:44:47 PM

To:    "'Julian Satran'" <Julian_Satran@il.ibm.com>, "BURBRIDGE,MATTHEW
       (HP-UnitedKingdom,ex2)" <matthew_burbridge@hp.com>
cc:    ips@ece.cmu.edu, John Hufferd/San Jose/IBM@IBMUS, "BURBRIDGE,MATTHEW
       (HP-UnitedKingdom,ex2)" <matthew_burbridge@hp.com>,
       owner-ips@ece.cmu.edu, Rod Harrison <rod.harrison@windriver.com>
Subject:    RE: ISCSI: Unsolicited data in draft v12



It would not necessarily need separate buffers but it does need to keep
some
buffers pre-allocated for unsolicited data so when the data arrives
unsolicited there is a buffer available in which to place the data.

Matthew


-----Original Message-----
From: Julian Satran [mailto:Julian_Satran@il.ibm.com]
Sent: Monday, April 08, 2002 12:00 PM
To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
Cc: ips@ece.cmu.edu; 'John Hufferd'; BURBRIDGE,MATTHEW
(HP-UnitedKingdom,ex2); owner-ips@ece.cmu.edu; Rod Harrison
Subject: RE: ISCSI: Unsolicited data in draft v12



I am with John here (the third guy that is right) - why would an
implementer
have separate buffers for solicited and unsolicited data?

Julo


"BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" <matthew_burbridge@hp.com>
08-04-02 21:43
Please respond to "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)"

        To:        John Hufferd/San Jose/IBM@IBMUS, "BURBRIDGE,MATTHEW
(HP-UnitedKingdom,ex2)" <matthew_burbridge@hp.com>
        cc:        Julian Satran/Haifa/IBM@IBMIL, Rod Harrison
<rod.harrison@windriver.com>, ips@ece.cmu.edu, owner-ips@ece.cmu.edu
        Subject:        RE: ISCSI: Unsolicited data in draft v12




John,

It's not so much an implementation problem but one resource management
problem in that if unsolicited data has been negotiated then target MUST
pre-allocate buffers with which to store the unsolicited when it arrives.
The target implementors will decided whether they want to use unsolicted
data and take the buffer resource hit in doing so.  However, if they do
wish
to take this hit but the initators decide not to use unsolicited data (even
though they have negotiated to use it) then there is potientially a lot of
valuable buffer resources tied in up for unsolicited data but which is not
being used.

Matthew

-----Original Message-----
From: John Hufferd [mailto:hufferd@us.ibm.com]
Sent: Monday, April 08, 2002 11:13 AM
To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
Cc: 'Julian Satran'; Rod Harrison; ips@ece.cmu.edu;
owner-ips@ece.cmu.edu
Subject: RE: ISCSI: Unsolicited data in draft v12



Please explain, why an initiator deciding to not send unsolicited data for
a specific command causes an implementation problem.  That was not clear
from your statements.  You still need the R2T capability, so what is lost?

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: hufferd@us.ibm.com


"BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" <matthew_burbridge@hp.com>
@ece.cmu.edu on 04/08/2002 10:25:55 AM

Sent by:    owner-ips@ece.cmu.edu


To:    "'Julian Satran'" <Julian_Satran@il.ibm.com>, Rod Harrison
      <rod.harrison@windriver.com>
cc:    ips@ece.cmu.edu, owner-ips@ece.cmu.edu
Subject:    RE: ISCSI: Unsolicited data in draft v12



I must express my concern on this issue.  From a target point of view once
it has negoiated the use of unsolicited data it has to allocate buffer
space
for that unsolicited data.  Now depending on the various parameters this
may
be a sizeable chunk of valuable resources which it is making available.
Now
if the decision to use unsolicited data is being moved from a per session
to
per task basis (which is what this change effectively does) then it puts an
awful lot of resource overhead on the target which may never be used.

For the reasons above I propose that we do not relax the v12 restriction
and
keep it as:

"An iSCSI initiator MUST send as unsolicited data either the negotiated
amount or all the data if the total amount is less than the negotiated
amount for unsolicited data."

Matthew Burbridge
Principal Engineer
NSAS-Bristol
Hewlett Packard

-----Original Message-----
From: Julian Satran [mailto:Julian_Satran@il.ibm.com]
Sent: Monday, April 08, 2002 9:36 AM
To: Rod Harrison
Cc: ips@ece.cmu.edu; owner-ips@ece.cmu.edu
Subject: Re: ISCSI: Unsolicited data in draft v12



OK - Julo


"Rod Harrison" <rod.harrison@windriver.com>
Sent by: owner-ips@ece.cmu.edu
08-04-02 14:52
Please respond to "Rod Harrison"

       To:        <ips@ece.cmu.edu>
       cc:
       Subject:        ISCSI: Unsolicited data in draft v12





               I propose we slightly relax the new restriction in draft
v12
that the
initiator MUST send the maximum permissible amount of unsolicited data. I
suggest we change the rule to allow the initiator to either send no
unsolicited data, or the maximum permissible.

               There is no difficulty for the target here since the lack
of
unsolicited
data will be clearly indicated by a command PDU with F bit set and
dataSegLen=0. The target will have all the information it needs to
immediately issue R2Ts as appropriate.

               I believe the initiator should be able to make a policy
decision on which
individual commands should be sent with unsolicited data and which should
not.

               In draft 11.91 section 2.2.4 I suggest we change

"An iSCSI initiator MUST send as unsolicited data either the negotiated
amount or all the data if the total amount is less than the negotiated
amount for unsolicited data."

               to something like

"An iSCSI initiator MAY choose to send no unsolicited data with a command,
or if any unsolicited data is sent it MUST be either the negotiated amount
or all the data if the total amount is less than the negotiated amount for
unsolicited data."

               - Rod