RE: iSCSI: Use of the A bit

[Eddy Quicksall <Eddy_Quicksall@ivivity.com>]

I think we may need better explanation about why some folks don't want to do
the "positive ack".

Then maybe we can work out something that works for all of us.

Eddy

-----Original Message-----
From: kevin_lemay@agilent.com [mailto:kevin_lemay@agilent.com]
Sent: Thursday, March 14, 2002 2:49 PM
To: hufferd@us.ibm.com; kevin_lemay@agilent.com
Cc: Eddy_Quicksall@ivivity.com; matthew_burbridge@hp.com;
rod.harrison@windriver.com; ips@ece.cmu.edu
Subject: RE: iSCSI: Use of the A bit


I cannot negotiate the maxBurstSize to a "nice" size because regardless of
what size is chosen, the Ios that I am processing will not line up nicely on
maxBurstSize boundaries.

Consider:

Assume the MaxBurstSize = 128K, This is the only IO on the connection.

1. The initiator sends a 64K read request to the target.
2. The target sends 64K to the initiator. Now it wants to know that all of
the data packets have been received so that resources for the IO can be
released and the IO complete sent to the OS. I cannot set the A-Bit on the
final data PDU because that would violate the clause that says:

"it MAY set the A-bit to 1 once at most every
MaxBurstSize bytes, and MUST NOT do so any more frequently than that."

But since I have the entire IO completed, I could send a Nop-Ping. That
would force the initiator to send the new StatSN which would imply all data
packets have been received. 

But there is also a more complicated case. Our chip allows the higher level
software to break IOs into its own data phases. If more than one phase is
used, then a response is not the last PDU sent and there is no work around
to get the data Ack'd.

Furthermore, negotiating the burstsize low so that I can set the a_bit when
I want has the undesired side effect of allowing only that much data per R2T
request. A definite performance squelcher.

Kevin Lemay

-----Original Message-----
From: John Hufferd [mailto:hufferd@us.ibm.com]
Sent: Thursday, March 14, 2002 11:36 AM
To: LEMAY,KEVIN (A-Roseville,ex1)
Cc: 'Eddy Quicksall'; BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); 'Rod
Harrison'; ips@ece. cmu. edu (E-mail)
Subject: RE: iSCSI: Use of the A bit



We got to this position, since so many folks did not want to support the
positive ack.  And this approach was put in as a compromise.  I would
suggest, that the thing we are chasing is a small corner case that may not
be smart enough to operate at anything other then ErrorRecoverylevel=0.  In
any event I do not see a problem with limiting the response to MaxBurstSize
intervals.  If the interval is too big, why did the unit negotiate it to
that size?

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: hufferd@us.ibm.com


"LEMAY,KEVIN (A-Roseville,ex1)" <kevin_lemay@agilent.com>@ece.cmu.edu on
03/14/2002 10:43:31 AM

Sent by:    owner-ips@ece.cmu.edu


To:    "'Eddy Quicksall'" <Eddy_Quicksall@ivivity.com>, "BURBRIDGE,MATTHEW
       (HP-UnitedKingdom,ex2)" <matthew_burbridge@hp.com>, "'Rod Harrison'"
       <rod.harrison@windriver.com>, John Hufferd/San Jose/IBM@IBMUS
cc:    "ips@ece. cmu. edu (E-mail)" <ips@ece.cmu.edu>
Subject:    RE: iSCSI: Use of the A bit



I also do not like the comment about having to wait until maxBurstsize byte
to set the A-bit.

I wanted to use it as Eddie suggested. To force the initiator to
acknowledge
the IO so that I can release resources on the target.

This will be especially useful for hardware accelerated IOs where the chip
cannot declare the IO complete until it has been Ack'd.

Could we just change the wording to:
----
For sessions with ErrorRecoveryLevel 1 or higher the target sets this bit
to
1 to indicate that it requests from the initiator a positive
acknowledgement
for the data received. The target should use the A bit moderately; the
A-bit
MAY only be set to 1 at the end of a data sequence.

On receiving a Data-In PDU with the A bit set to 1 the initiator MUST
issued
a SNACK of type DataACK. If the initiator has detected holes in the input
sequence, it MUST postpone issuing the SNACK of type ACKN until the holes
are filled.
---

This way I can always set the A bit at the end of an IO.

Kevin Lemay

-----Original Message-----
From: Eddy Quicksall [mailto:Eddy_Quicksall@ivivity.com]
Sent: Thursday, March 14, 2002 5:51 AM
To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); 'Rod Harrison'; John
Hufferd
Cc: ips@ece. cmu. edu (E-mail)
Subject: RE: iSCSI: Use of the A bit


I see a lot of responses talking about ErrorRecoveryLevel = 0. That was not
my original point. ErrorRecoveryLevel = 0 already has this solved. I am
talking about ErrorRecoveryLevel > 0 (I made an error in my first EMAIL and
the point may have been lost).

If the initiator has the option to ignore the A bit, the target will get
stuck. What is wrong with requiring the initiator to respond? Can we just
get that answered first?

I suspect that there is a concern that this may cause too much traffic. My
answer would be "don't buy a cheep target ... their lack of memory may
cause
degraded performance" (not intended for the spec, of course :-) ).

I also do not see the rational in not allowing the A bit more often than
MaxBurstSize. The ULP layers may not know anything about the transport
layers. So their resource issues are not known by the transport(s). The
MaxBurstSize is an iSCSI issue and may not apply to future protocols.

So, why not just say something like:

1) The initiator MUST send a DataACK SNACK if it sees the A bit set (and if
ErrorRecoveryLevel > 0)
2) The A bit may be set only when the F bit is set.

And leave it go at that?

Eddy

-----Original Message-----
From: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
[mailto:matthew_burbridge@hp.com]
Sent: Thursday, March 14, 2002 7:22 AM
To: 'Rod Harrison'; John Hufferd
Cc: Eddy Quicksall; ips@ece. cmu. edu (E-mail)
Subject: RE: iSCSI: Use of the A bit


I think that what Santosh and others have stated is the correct think to do
for initiators.  If the revovery level is 0 and the A bit is set then the
initiator is quite with in its rights to ignore the A bit (If the target is
at fault then penalise the target and not the initiator).

IMHO changing the spec to make it more specific is not the right thing and
it could lead to inter operability problems.  However, I do think something
needs to be added so I propose adding the statement:

"If ErrorRecoveryLevel=0 then the initiator MAY ignore the A bit (i.e.
assume A=0)".

One could argue for the MAY to be a MUST but if there is a kind initiator
out there then let it do its stuff - it can only help the rogue target.

Matthew

-----Original Message-----
From: Rod Harrison [mailto:rod.harrison@windriver.com]
Sent: Thursday, March 14, 2002 9:27 AM
To: John Hufferd
Cc: Eddy Quicksall; BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); ips@ece.
cmu. edu (E-mail)
Subject: RE: iSCSI: Use of the A bit


John,

 Actually 9.7.2 says almost exactly that ...

 "For sessions with ErrorRecoveryLevel 1 or higher, the target sets
this bit to 1 to indicate that it requests a positive acknowledgement
from the initiator for the data received."

 We've interpreted that to mean our initiator should ignore the A bit
if ErrorRecoveryLevel is 0.

 Having said that it is very easy to support just the positive form
of
SNACK, so I would have no objection to removing the A bits current
link to ErrorRecoveryLevel.

 - Rod

-----Original Message-----
From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
John Hufferd
Sent: Thursday, March 14, 2002 3:12 AM
To: Rod Harrison
Cc: Eddy Quicksall; BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); ips@ece.
cmu. edu (E-mail)
Subject: RE: iSCSI: Use of the A bit



Actually it does not actually say that.  It says "This is limited to
sessions that support error recovery and is implemented through the A
bit
..."

I know what I am about to say, is nit picking, but ....
ErrorRecoveryLevel=0 is an error recovery technique.

Having said that, we probably should just change that statement to
"This is
implemented through the A bit ..."


.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: hufferd@us.ibm.com


"Rod Harrison" <rod.harrison@windriver.com>@ece.cmu.edu on 03/13/2002
01:05:51 PM

Sent by:    owner-ips@ece.cmu.edu


To:    "Eddy Quicksall" <Eddy_Quicksall@ivivity.com>,
"BURBRIDGE,MATTHEW
       (HP-UnitedKingdom,ex2)" <matthew_burbridge@hp.com>, "ips@ece.
cmu.
       edu (E-mail)" <ips@ece.cmu.edu>
cc:
Subject:    RE: iSCSI: Use of the A bit



Eddy,

 You were right first time, in draft 11 section 2.5.1.5 paragraph 5
and 9.7.2 state that the A-bit is only supported if ErrorRecoveryLevel
is 1 or higher.

 - Rod

-----Original Message-----
From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
Eddy Quicksall
Sent: Wednesday, March 13, 2002 8:03 PM
To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); ips@ece. cmu. edu
(E-mail)
Subject: RE: iSCSI: Use of the A bit


Then my only concern is that the initiator may ignore the A bit if it
deems that the bit is being set aggressively.

If it ignored it, then the target would be stalled waiting for he ACK.

Eddy
-----Original Message-----
From: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
[mailto:matthew_burbridge@hp.com]
Sent: Wednesday, March 13, 2002 1:39 PM
To: 'Eddy Quicksall'; ips@ece. cmu. edu (E-mail)
Subject: RE: iSCSI: Use of the A bit
Importance: High


Eddy,

The target is quite within its rights to use the A bit when at
recovery level 0.  If the session is re-established due to recovery
7.11.4 then the relevant command is aborted anyway and so there is no
reason to keep hold of the data any way: With recovery level 0 there
is no recovery mechanism that requires the target to keep the data.
Therefore the A bit is redundant when the recovery level is 0.

The spec says that the initiator MUST issue a SNACK if the A bit is
set.  However, the MaxBurstSize restriction is there to prevent the
initiator from having to send a SNACK on every PDU in the case where a
target inadvertently sets the A bit in (for example) every data in
PDU. The target may set the A bit more often than the MaxBurstSize but
it should not expect a SNACK more often than this.

Matthew Burbridge
-----Original Message-----
From: Eddy Quicksall [mailto:Eddy_Quicksall@ivivity.com]
Sent: Wednesday, March 13, 2002 3:12 PM
To: ips@ece. cmu. edu (E-mail)
Subject: iSCSI: Use of the A bit


Here is a case that I want to go over and if there is not already a
solution, I think a refinement to the A bit could solve it.

The problem is that a target (perhaps an iSCSI disk drive) does not
have enough memory to transfer the full READ request so it must read
from the medium as much as it can, transmit that, when that
transmission is known to be good, read the next bunch, transmit that
and so on.

The problem we have is that the target must keep the buffer around
until the transfer has been "ack'd" via ExpStatSN. But that status
can't be sent because all of the requested data has not been sent. So
the target would have to refuse to do the command.

I was going to use the A bit for this thinking it would force the
initiator to give an "ack" but our current wording does not make this
a sure fire thing:

1) The initiator may not want to run at ErrorRecoveryLevel 1.
2) The initiator may ignore the A bit if it deems that the bit is
being set aggressively.
3) The target may set the A bit no more frequently than MaxBurstSize.

Comments?

mailto:Eddy_Quicksall@iVivity.com