RE: iSCSI: Recognizing Recovery R2Ts

["Julian Satran" <Julian_Satran@il.ibm.com>]

Nick,

In case you have ErrorRecoveryLevel >=1 and DataSequenceInOrder=Yes then then either:

• the target has only one outstanding R2T (to avoid at all price an out-of-order R2T)
• or the initiator can go by an out-of-order R2T if it is a data-offset it has "passed-over" (even if not touched). Please observe that an initiator is aware of the gap in the R2T sequence when an R2T is lost if the target has more than one R2T missing and the missing one is not the last. If the missing one is the last (or only) there will be no violation of ordering.
  
  Regards,
  Julo.
  The ordering rules require the offsets to be in non-decreasing order.
  
  
  

  Nick Bellinger <nickb@attheoffice.org> 16/09/02 09:18

  To:        Julian Satran/Haifa/IBM@IBMIL         cc:        Black_David@emc.com, ips@ece.cmu.edu, cbm@rose.hp.com         Subject:        RE: iSCSI: Recognizing Recovery R2Ts

  
  
  Julian,
  
  On Sun, 2002-09-15 at 23:50, Julian Satran wrote:
  > Nick,
  >
  > I understand that your issue is mainly related to an initiator expecting
  > "in order" R2Ts and seing an R2T that is out of order.
  >
  > The simplest answer (and the one that we had in mind when we designed the
  > protocol) is that in general nodes willing to accept ErrorRecoveryLevels 1
  > and 2 will probably accept R2Ts out of order. In this case an initiator
  > does not need to make any distinction.
  >
  > But not wanting to be that restrictive we let even nodes accepting only
  > ordered R2Ts do recovery but then (if they care) they may::
  >
  > initiators:
  >
  > reject any R2T that is unordered (i.e. do not recover)
  > accept R2T based on having "previously seen it"  (as David Black suggests)
  > - with the caveat that if the R2T is lost you may have to fall back to  1
  
  Considering ErrorRecoveryLevel>=1 and DataSequenceinOrder=Yes:
  
  The potentially lost R2T your referring to is the original one correct?
  Original meaning the R2T the initiator can use to determine if the R2T
  is infact a recovery R2T and is requesting retransmissing of an
  previously requested offset.  
  
  If this is the case,  I dont see how this situation could possibly
  happen.  If the R2T the target originally sent gets lost and the
  initiator cannot send any data out without the lost R2T,  how could the
  target possibly be able to generate a recovery r2t with differing
  offsets for a data out that was never received?  
  
  So if this is indeed correct,  the target generates a sequence timeout
  under the guise of within-command recovery,  and sends a Recovery R2T
  with identical values to the one no data out was received for and
  presumed lost. And then if there is a Data Digest failure on data out
  the target generates a recovery R2T with a previously requested offset,
  and the initiator is able to tell the difference in R2Ts.  
  
  I think this is a bit confusing (to me anyways) as a recovery R2T can
  either mean:
                  a) A identical retransmission due to a header digest error (no                     data
  out received, manifests as a sequence timeout)
                  b) Previously requested offset requested due to a data digest                    
  error.
       
  Is there something I am missing as to how the initiator with
  DataSequenceinOrder=Yes could lose an original R2T and still somehow
  receive a Recovery R2T (of type b) and have within-command recovery
  fail?
  
  Thanks Julian!
  
                  Nicholas Bellinger - Pyxtechnologies.com
  
  > targets:
  > have only one R2T outstanding if you have both "in order" and error
  > recovery (a rare thing we believe)
  >
  >
  > Regards,
  > Julo
  >
  >
  >
  >
  > Black_David@emc.com
  > Sent by: owner-ips@ece.cmu.edu
  > 16/09/02 04:58
  >
  >  
  >         To:     nickb@attheoffice.org
  >         cc:     ips@ece.cmu.edu
  >         Subject:        RE: iSCSI: Recognizing Recovery R2Ts
  >
  >  
  >
  > One additional important note - the Recovery R2T generates
  > its own data sequence, so in the following:
  >
  > > > Following a)
  > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=16384 DataSN=2, DataLength=8192
  > > > I-> ISCSI_INIT_SCSI_DATA_OUT F_BIT, Offset=24576, DataSN=3,
  > > > DataLength=8192
  > > >
  > > > Following B)
  > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=16384 DataSN=2, DataLength=8192
  >
  > The DataSN's should be 0 and 1, not 2 and 3 (and in the B) case above,
  > the F bit needs to be set, as I previously noted).  The initiator really
  > does respond to a Recovery R2T in the same fashion as any other R2T.
  >
  > Thanks,
  > --David
  >
  > > -----Original Message-----
  > > From: Black_David@emc.com [mailto:Black_David@emc.com]
  > > Sent: Sunday, September 15, 2002 9:45 PM
  > > To: nickb@attheoffice.org
  > > Cc: ips@ece.cmu.edu
  > > Subject: RE: iSCSI: Recognizing Recovery R2Ts
  > >
  > >
  > > > Im still lost here.  Please consider the example:
  > > >
  > > > MaxDataSegmentLength=8192, MaxBurstLength=32768, ImmediateData=No,
  > > > InitialR2T=Yes.
  > > >
  > > > I-> ISCSI_INIT_SCSI_CMND, EDTL=131072, SCSI Opcode WRITE_10
  > > > T-> ISCSI_TARG_R2T, DDTL=32768, Offset=0
  > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=0, DataSN=0, DataLength=8192
  > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=8192, DataSN=1, DataLength=8192
  > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=16384, DataSN=2,
  > > DataLength=8192
  > > >
  > > > (Data Payload Fails on DataSN=2)
  > > >
  > > > T-> ISCSI_TARG_RJT, Reason=Data Digest (Payload) Error
  > > > I-> ISCSI_INIT_SCSI_DATA_OUT, F_BIT, Offset=24576, DataSN=3,
  > > > DataLength=8192
  > > >
  > > > Can the Target either: (Are both of these legal?)
  > > >
  > > >              a) Accept DataSN=3 even though DataSN=2 Failed
  > > >                 Only request retransmisson of Failed DataOut
  > > >              T-> ISCSI_TARG_R2T (Recovery) DDTL=8192, Offset=16384
  > > >
  > > >              or b) Discard DataSN=3, Request retransmission of
  > > >                    all DataOut from failed to MaxBurstLength
  > > >              T-> ISCSI_TARG_R2T (Recovery) DDTL=16384, Offset=16384
  > >
  > > Yes, both are legal, a) is preferred for obvious reasons.
  > >
  > > > Initiator: (This is where im confused)
  > > >              Received either a) or b),  and knows the R2T is a
  > Recovery
  > > >              R2T because the DDTL is not 32768 (MaxBurstLength or rest
  > of
  > > >              the original EDTL) and Offset is not 32768.
  > >
  > > Not exactly.  The R2T is a recovery R2T because it requests data
  > > covered by the previous R2T (Offset < 32768).  This doesn't affect
  > > the initiator's behavior - it just responds to the R2T it receives.
  > >
  > > > Following a)
  > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=16384 DataSN=2, DataLength=8192
  > > > I-> ISCSI_INIT_SCSI_DATA_OUT F_BIT, Offset=24576, DataSN=3,
  > > > DataLength=8192
  > > >
  > > > Following B)
  > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=16384 DataSN=2, DataLength=8192
  > >
  > > I think you flipped the a) and b) cases above, so you clearly are
  > > confused :-).  In both cases, the initiator looks at the R2T to
  > > determine what data the Target is asking for and sends it - a) asked
  > > for 8k, b) asked for 16k.  The one PDU that responds to a) needs
  > > to have the F bit set, as it completes the response to the second
  > > R2T.  If the target needs to distinguish the recovery data transfer
  > > from the original transfer, it should use a different Target Transfer
  > > Tag in the recovery R2T.
  > >
  > > > Target:
  > > >              MaxBurstLength is Completed:
  > > > T-> ISCSI_TARG_R2T, Offset=32768, DDTL=32768
  > > >
  > > > I-> ISCSI_INIT_SCSI_DATA_OUT, Offset=32768, DataSN=0,
  > > DataLength=8192
  > >
  > > Ok.
  > >
  > > > Your statement "Whether the initiator knows this depends on
  > > whether it
  > > > saw the previous R2T" is still confusing to me.  The logic
  > > used in the
  > > > above example (EDTL and Offset) I assume is incorrect.
  > > Would you mind
  > > > elaborating with a specific example of the proper method of
  > > doing so?
  > >
  > > I don't understand the problem here.  In all cases, the initiator
  > > looks at the offset and length in the R2T it receives and sends the
  > > data requested by that R2T.  In both cases a) and b) above,
  > > the initiator
  > > can tell that it's a recovery R2T because the requested data
  > > range overlaps
  > > a previous R2T.  The statement I wrote covers a completely different
  > > case in which the target sends an R2T that doesn't arrive
  > > (e.g., header
  > > digest error), and decides to try to recover from that.
  > >
  > > I also don't understand the reference to EDTL in the command
  > > (as opposed to DDTL in the R2T0.  If a Recovery R2T causes
  > > more than EDTL of data to be transferred for a command that's fine -
  > > the target is in charge of data flow and will tell the initiator when
  > > the command was complete and how much was actually transferred
  > > (consider unexpected end-of-tape on a tape device to understand
  > > why this is done this way).
  > >
  > > Thanks,
  > > --David
  > > ---------------------------------------------------
  > > David L. Black, Senior Technologist
  > > EMC Corporation, 42 South St., Hopkinton, MA  01748
  > > +1 (508) 249-6449            FAX: +1 (508) 497-8018
  > > black_david@emc.com       Mobile: +1 (978) 394-7754
  > > ---------------------------------------------------
  > >
  >
  >