Re: iSCSI: error recovery

[Matt Wakeley <matt_wakeley@agilent.com>]

Julian,

My comments to your comments:

julian_satran@il.ibm.com wrote:

> Now for some issues I have with the (current) iSCSI draft:
>
> In section 2.2.2 it states "As the only cause for long delays in responses
> can
> be failed connections and received responses free-up resources, we felt that
> score boarding responses at the initiator could be accomplished by simple
> bitmaps and there is no need to flow-control responses."
>
> Score boarding, especially with bit maps,  is an operation that can be
> somewhat CPU heavy in the normal "performance path" of the iSCSI layer. If
> the ExpStatRN was local to each TCP connection, rather than global across the
>
> iSCSI session, then there would be no requirement for score boarding.  The
> initiator would simply increment the StatRN received on each connection for
> use in the ExpStatRN for that connection.
> /<JS>
> I formulated it wrong.  As commands are associate with a connection at both
> initiator and target
> we always know what commands to reissue and there is no need to flow
> control responses.
> there is no scoreboarding involved.
> <JS>/

Julian, re-read the above text.  It is referring to Response messages, not
Command messages.  If StatRN is global across the sesstion, then the initiator
must perform scoreboarding of the StatRN values received in order to figure out
what value to set the ExpStatRN to.  Consider the example of an iSCSI session
consisting of 3 TCP connections (A, B and C).  The target queues up response
messages numbered (StatRN) 1 thru 9 on the 3 TCP connections thusly: 1->A,
2->B, 3->C, 4->A, 5->B, 6->C, 7->A, 8->B, 9->C.  Now lets assume that TCP
connections A and C are relatively idle, but B is blocked (perhaps performing
recovery due to a dropped frame).  The initiator will receive responses 1, 3,
4, 6, 7, 9.  The next ExpStatRN value the initiator can supply is 2.  If/when
response #2 is received, the initiator must supply ExpStatRN of 5, and when
that's received, 8.  Thus, with a global StatRN, score boarding must be
performed by the initiator on StatRNs received in order to supply the
appropriate ExpStatRN.

>
> >From an earlier email: "1.1.1.3   Data PDU numbering
> Incoming Data PDUs MAY be numbered by a target to enable fast recovery of
> long
> running READ commands. Data PDUs are numbered with DataRN.  NOP command
> PDUs
> carrying the same Initiator Tag as the Data PDUs are used to acknowledge
> the
> incoming Data PDUs."
>
> Since the only "error" we are trying to recover from is the very rare event
> that a physical link fails, I fail to see what the benefit is to be able to
> "recover" at the PDU level.  Plus, you'll have to build into the protocol a
> mechanism to request retransmission of particular data PDUs.  Let's
> simplify
> and just send the command with the "retry" bit set.
> /<JS>
> You don't have to build a mechanism to request retransmission.

I don't?  Then how is the target supposed to know that it must retransmit data
that was lost due to connection failure?  How is the target to know what
connection to transmit that data on?  Of course there needs to be a mechanism.
A simple mechanism is to resend the command with the retry bit set.

>
> I assume that a clever target will keep only unacked data (the whole point
> of data PDU numbering is to lower the amount of data a target has to keep
> for recovery).

I don't think there is any advantage to retransmitting only data that was not
"acked".  Let's say the data was being sent over a tcp connection to initiator
HBA #1 of a multi tcp connection iSCSI session.  I think for simplicity most
iSCSI HBAs will inform the host driver of completed I/Os, not "partial" I/Os
(especially indicating what was received and what wasn't).  When the connection
is "failed over" to another connection, perhaps running on a different HBA, it
will be much simpler to retry the whole I/O over the new connection, rather
than piece together two partial I/Os.  Simpler is better isn't it?

> At command restart it will resent what it has.

No, command retry (restart) was meant to retry the whole I/O.  It was not meant
to be "send what you think I didn't get".

> Obviously a
> target may decide to ignore data acks (especially if it can reread the
> media) and I assume disk targets will do just that and tapes will use the
> acks.

I assume by the time iSCSI is implemented, tapes will be behave just like
discs.  Tapes are already moving to that model and there is a question of
whether the "fc-tape" error recovery in FCP2 will be needed when it is
completed.

>
> <JS>/
> Also from an earlier email:
>
> > >Mallikarjun,
> > >
> > >Thanks for your comments.
> > >
> > >Initiator scoreboarding is not considered. I will try to emphasize this
> > >even more in the new draft.
> > >The party responsible for reporting length is the target.  As
> overlapping
> > >ranges are not explicitly
> > >forbidden this would be a harder task than apparent. Reporting counts
> > >becomes entirely a question of faith!
> >
> > I didn't realize that (what FC calls as) data overlay is allowed, FCP
> > requires this initiator capability to be explicitly stated in session
> > establishment (process login).  Is there a particular reason why this
> > is chosen to be allowed by default in iSCSI?
> >
>
> Again, in the interests of simplicity, I request that data overlay be
> forbidden.  Period.  Otherwise, the initiator would have to perform score
> boarding at the byte level to be positively sure that each byte was really
> received.
> /<JS>
> That is an interesting point.  I would argue that in the interest of
> simplicity
> we will stay neutral.  If we explicitely forbid it the every Initiator is
> bound to check (enforce) it and that is a lot of work.  I assume we will want
> to
> use SHOULD.  My point about scoreboarding is that initiators are not required
>
> to check (enforce) the overlap.

I disagree Julian.  Forbidding a target from performing a function does not
mandate that the initiator play policeman and verify that the target is not
doing what it's forbidden to do.

If we do not forbid it, then the initiator will have to support it, and that
would be a lot of work.

>
>
> <JS>/

-Matt Wakeley