Re: iSCSI ERT: data SACK/replay buffer/"semi-transport"

[julian_satran@il.ibm.com]

Santosh,

SNACK and SACK are the same thing (I just renamed them to avoid confusion
with TCP SACK).
The status is acked by ExpStatSN (and only indirectly by SNACK). SNACK
enables fast recovery of
a hole (whithout having to resort to a timeout).  We decided long ago
against individual acks as bulk acking through a window is cheaper and
safer (repetition).

Julo

Santosh Rao <santoshr@cup.hp.com> on 05/04/2001 21:06:18

Please respond to Santosh Rao <santoshr@cup.hp.com>

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   ips@ece.cmu.edu
Subject:  Re: iSCSI ERT: data SACK/replay buffer/"semi-transport"




Julian,

The existing StatSN SNACK mechanism will NOT work if it is made
optional. The original request that was made in the thread
http://ips.pdl.cs.cmu.edu/mail/msg03257.html was to allow a SACK
mechanism that would allow individual Status PDUs to be acknowledged
[not SNACK which requests re-send of missing Status PDU, and thereby,
requires the target to retain state information until StatSN is
acknowledged].

If StatSN SNACK is made optional, a target that does not support SNACK
will result in holes never being filled in StatSN sequence, and thereby,
initiators being unable to acknowledge status PDUs received after the
hole. This can cause targets to hold onto stale I/O state information
for very long periods. [or forever].

With the current StatSN SNACK scheme, a target can NEVER discard its old
I/O state information, since if it does so, it cannot satisfy SNACK
requests. If SNACK requests are not satisfied, holes remain in the
StatSN sequence at the initiator and it cannot acknowledge Status PDUs
received thereafter.

If we must retain the StatSN mechanism in iSCSI, then, the SACK
mechanism [as opposed to a SNACK], wherein, the initiator ack's
individual status PDUs received when a hole occurs should be the
preferred scheme. This alows both sides to continue the handshake of
resource release even in the presence of holes, without imposing
requirements on targets to retain I/O state information.

The holes created in StatSN are implicitly filled by the initiator based
on the result of its "retry" of the failed command. Alternatively, the
StatSN hole is considered to be filled if the initiator chooses not to
retry the command [ex: on ULP timeout].

- Santosh


julian_satran@il.ibm.com wrote:


>
> Mallikarjun,
>
> You are right. Too much travel and jet lag.  The reason we made SACK and
> status recovery
> practically a MUST is that without them we are bound to have only session
> drop as an alternative.
> If the target does not keep any information after it has sent out status
it
> can't even retry a command.  And if it can retry a command it should be
> able to do SACK.
>
> But perhaps there is a place in the market for the kind of devices Somesh
> is suggesting that do all recovery at SCSI level (and that can't copy a
> terabyte of data without a session drop).
>
> If that is true (which I doubt) we can make SNACK support optional.
>
> Julo
>
> "Mallikarjun C." <cbm@rose.hp.com> on 05/04/2001 04:09:46
>
> Please respond to cbm@rose.hp.com
>
> To:   ips@ece.cmu.edu
> cc:
> Subject:  Re: iSCSI ERT: data SACK/replay buffer/"semi-transport"
>
> >
> >Santosh,
> >
> >I can't find the place where this is stated. SNACK as a PDU type is
> >mandated. But it can be rejected outright.
>
> Sorry, you agreed that status SACK is mandatory in ERT forum last
> week in response to my comments.  Has there been a change in your
opinion?
>
> Attached is the email in a long email thread (issue 3) where you agreed
> to make this explicit in rev06.
> --
> Mallikarjun
>
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions Organization
> MS 5668   Hewlett-Packard, Roseville.
> cbm@rose.hp.com
>
> >1.2.2.2 show explicitely that SACK can be rejected. We can add a
protocol
> >specific parameter in the target Logical Unit Control Page (non-setable)
> by
> >which the target will indicate support for SNACK.
> >
> >Julo
>
> Santosh Rao <santoshr@cup.hp.com> on 04/04/2001 23:53:32
>
> Please respond to Santosh Rao <santoshr@cup.hp.com>
>
> To:   Julian Satran/Haifa/IBM@IBMIL
> cc:   Jon Hall <jhall@emc.com>, ips@ece.cmu.edu
> Subject:  Re: iSCSI ERT: data SACK/replay buffer/"semi-transport"
>
> julian_satran@il.ibm.com wrote:
> >
> > Jon,
> >
> > Inexpensive implementation are always free to do away with recovery.
That
> > si true for targets too.
>
> That's not the interpretation one gets from reading the spec and prior
> discussions on this list. Per the spec, support for Status SACK is
> mandatory while support for data SACK is optional.
>
> IOW, targets MUST retains state information to satisfy a potential
> status SACK request.
>
> - Santosh
>
>
------------------------------------------------------------------------------

>
> >From julian_satran@il.ibm.com Tue Mar 27 05:16:54 PST 2001
> Received: from mailhub.rose.hp.com (mailhub.rose.hp.com [15.96.64.24]) by
> core.rose.hp.com with ESMTP (8.8.6 (PHNE_14041)/8.8.6 SMKit7.02) id
> FAA26277 for <cbm@core.rose.hp.com>; Tue, 27 Mar 2001 05:16:52 -0800
(PST)
> From: julian_satran@il.ibm.com
> Received: from atlrel2.hp.com (atlrel2.hp.com [15.10.184.10]) by
> mailhub.rose.hp.com with ESMTP (8.7.1/8.7.3 SMKit7.02) id FAA10600 for
> <cbm@rose.hp.com>; Tue, 27 Mar 2001 05:15:51 -0800 (PST)
> Received: from d06lmsgate.uk.ibm.COM (d06lmsgate.uk.ibm.com
[195.212.29.1])
>      by atlrel2.hp.com (Postfix) with ESMTP id AFC11120
>      for <cbm@rose.hp.com>; Tue, 27 Mar 2001 08:15:49 -0500 (EST)
> Received: from d12relay01.de.ibm.com (d12relay01.de.ibm.com
[9.165.215.22])
>      by d06lmsgate.uk.ibm.COM (1.0.0) with ESMTP id NAA50078;
>      Tue, 27 Mar 2001 13:55:50 +0100
> Received: from d12mta02.de.ibm.com (d12mta02_cs0 [9.165.222.253])
>      by d12relay01.de.ibm.com (8.8.8m3/NCO v4.95) with SMTP id PAA174358;
>      Tue, 27 Mar 2001 15:09:58 +0200
> Received: by d12mta02.de.ibm.com(Lotus SMTP MTA v4.6.5  (863.2
5-20-1999))
> id C1256A1C.00483A46 ; Tue, 27 Mar 2001 15:08:55 +0200
> X-Lotus-FromDomain: IBMIL@IBMDE
> To: cbm@rose.hp.com
> Cc: someshg@yahoo.com, steph@cs.uchicago.edu, hufferd@us.ibm.com,
>         cbm@rose.hp.com, ldalleore@snapserver.com,
>         Venkat Rangan <venkat@rhapsodynetworks.com>, Black_David@emc.com
> Message-ID: <C1256A1C.0048399C.00@d12mta02.de.ibm.com>
> Date: Tue, 27 Mar 2001 15:12:01 +0200
> Subject: Re: iSCSI ERT: error recovery comments
> Mime-Version: 1.0
> Content-type: text/plain; charset=us-ascii
> Content-Disposition: inline
> Status: RO
>
> Comments in text.  Thanks Julo
>
> "Mallikarjun C." <cbm@rose.hp.com> on 27/03/2001 01:41:48
>
> Please respond to cbm@rose.hp.com
>
> To:   cbm@rose.hp.com, someshg@yahoo.com, steph@cs.uchicago.edu, Julian
>       Satran/Haifa/IBM@IBMIL, John Hufferd/San Jose/IBM@IBMUS
> cc:   Black_David@emc.com
> Subject:  iSCSI ERT: error recovery comments
>
> Hi Julian and the Team,
>
> Here are some comments on error recovery issues.  I hope these will
> be addressed soon.  Thanks.
>
> 1. The draft should clearly state that if a target doesn't support
>   retry (replay in my previous memo's terminology), it must not silently
>   accept a command with retry bit and re-do the I/O.
>
> 2. Consequent to the above -
>      - Clarification required on section 6.7.1, page 83, last para.
>           Please confirm and clarify in the draft: If the target sends
>           a response with an iSCSI error response of "SACK-rejected" that
>           implicitly terminates the task - no retries are allowed. If the
>           target sends a Reject PDU with "Data SACK Reject" code, the
task
>           stays open and the initiator may try to recover using
SACK/retry.
> +++ I will clarify
> it will read:
>    An iSCSI target MAY reject a data-SNACK and terminate the command with
>    an iSCSI command response of SNACK rejected. In this case, the task is
>    terminated and no future action is expected at target and initiator.
>
>    Alternatively, an iSCSI target MAY reject a data-SNACK with a reject
>    response of data SNACK rejected. In this case the task is still open
and
>    may be recovered using the retry.
>
> +++
>         - On a data digest error on a data PDU without the F-bit, the
draft
>           states that the target must wait for a data PDU with the F-bit
>           (per section 6.2), then a command termination is signalled with
>           a Reject PDU!  I like the formulation in 2.4.2 better.  I
> strongly
>           recommend that similarly, the target shall send a SCSI Response
>           with a iSCSI response of "delivery subsystem failure".  In
> general,
>           I suggest that anytime a target terminates a task internally,
it
>           must generate a SCSI Response PDU with an appropriate response
> code.
> +++ It reads now:
>
>    When a target receives an iSCSI PDU with a header digest error or a
>    payload digest error in an iSCSI PDU, it MUST answer with a Reject
iSCSI
>    PDU with a Reason-code of Header-Digest-error or Data-Digest-Error and
>    discard the offending PDU.  If the error is a Data-Digest-Error in a
>    Data-PDU, the target MUST either request retransmission with a R2T or
>    answer with a command response PDU with a response-code of
>    delivery-subsystem-failure and abort the task. If the target is
>    answering with an error in the command response PDU it must wait for
the
>    target to receive all the data (signaled by a Data PDU with the final
>    bit Set for all outstanding R2Ts) the command response PDU.
> ++++
>
> 3. While the following is implied in different sections, it is not
>   obvious.  Please clarify the following in the draft - "Status SACK
>   support is mandatory, whereas data SACK support is not."
>
> +++ will do in 2.16.1 +++
>
> 4. The general policy of retry should be that all ordered commands
>   shall support retry bit, since the loss of an ordered command
>   creates a hole in target scoreboarding and stalls the target
>   pipeline.  Retry hopefully can plug the hole quickly to avoid this.
>
> 5. As a fallout of the above comment, Retry bit must be supported
>   for Text Commands.
>
> +++ I have added the X-bit.  The reason I did no earlier was that I could
> not foresee
> a case in which the command is not idempotent - I can allways be resent -
> but I guess it is cleaner with the X +++
>
> 6. Section 2.20, page 71 on Reject must specify if a retry of the
operation
>   is allowed for each Reject PDU reason code.  Lack of specification
could
>   lead to interoperability issues down the road with "retry wars" raging
>   between heterogeneous implementations (ex., target rejects the retry
bit,
>   initiator retries the "retry" bit,....).
> +++ the part now reads:
>
>    The reject Reason is coded as follows:
>
>       1 - Format Error
>       2 - Header Digest Error
>       3 - Data (payload) Digest Error
>       4 - Data-SNACK Reject
>       5 - Command Retry Reject
>       15 - Full Feature Phase Command before login
>
>       Some of the reject reasons terminate or prevent the creation of a
>       task at the target and no retry is possible in those cases. Format
>       error for a command, Command Retry Reject and Full Feature Phase
>       Command before login are in this category.
>
> 7. NOP-OUT does not require CmdSNs.  Why make it an ordered command
>   and run the risk of a digest error on it leading to a hole in
>   command ordering?
>
> +++ the reason I wanted it ordered is to check the whole command path -
but
> you may try to convince me that it is not a good idea +++
> --
> Mallikarjun
>
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions Organization
> MS 5668   Hewlett-Packard, Roseville.
> cbm@rose.hp.com
 - santoshr.vcf