RE: iSCSI: Flow Control

[Black_David@emc.com]

With my WG co-chair hat off:

> In your example C1, C2, D1, D2 will always work (as will any ordered
combination).
> The troublesome situation is C1, C2+D2 (immediate), C3  (window closed)
> followed by an R2T for D1. This is why I think we should either explicitly
> forbid immediate data if R2T is enabled or specify that this behavior may
> get an initiator into trouble.

I think we need to be careful about reinventing things that work in the SCSI
world.  Focus in a discussion like this really needs to be on how the
presence
of a IP-based network changes things.  So, let's work backwards.

The window closes because the initiator doesn't receive window updates from
the target.  Either the updates were lost, or the target didn't send them.

If the updates are lost, the R2T will reopen the window, so the problem must
be that the target didn't send the updates.  If the target is temporarily
overloaded,
the window will reopen shortly.  If it doesn't, then something is broken in
the
target and possibly the initiator because the same thing would go wrong on
any transport.

Between the technique already described that allows a SCSI target to
set limits on the max size of immediate data and the target's responsibility
for controlling resource allocation, a properly engineered target should not
leave windows closed for extended periods of time.  A target that permits
transfer of more data than it can safely buffer is prone to breakage, and
is likely to lose market share to targets that don't screw up in this
fashion.

Recalling that SCSI puts a target in complete control of its resource
allocation,
if immediate data is allowed, then we need to reuse the existing SCSI
mechanism
that allows a target to limit the amount of immediate data.
Targets that overcommit their buffering resources are responsible for the
consequences, and efforts should not be spent here to accommodate poor
implementation decisions.  We should definitely permit targets to follow
Julian's approach of prohibiting immediate data, but I don't think that we
should require it.

The one thing that does change here is that the presence of the immediate
data makes it easier to fill a TCP window, costing a round trip delay to get
the next update -- this is an opportunity for implementers to apply some
intelligent tuning of the sizes of advertised TCP windows to make this
unlikely in practice.

Also note that for the case in which C1 and C2 are ordered, the initiator
has done something colossally stupid by using R2T for C1's data and sending
C2's data in line.  This is a case in which the best that can be expected
is for things to work, and not a case for which performance ought
to be optimized.

--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
black_david@emc.com       Mobile: +1 (978) 394-7754
---------------------------------------------------