iSCSI sessions: Let's try again

[Black_David@emc.com]

Taking another swipe at the hornet's nest labeled
"multi-connection iSCSI sessions" ...

With the consensus on the minimum number of TCP connections
per iSCSI session having come unraveled, we need to start
over on this whole area of multiple connection sessions,
beginning with requirements and applicability.  This only
applies to TCP, as SCTP would only use a single connection
per session.  This email is a mixture of an attempt to
summarize discussions and propose WG consensus with  some
additional technical contributions from yours truly with my
co-chair hat off.  I've tried to flag the latter with
<IMHO></IMHO>, but I may not have gotten all of them.

Multiple connection sessions respond to two requirements:
[1] Use of physical parallelism in the network fabric to
	improve throughput over a single connection that
	cannot exploit physical parallelism.  This is an
	optimization.
[2] Improvements in error recovery over SCSI's usual
	approach to error recovery.  This is an optimization.
[3] A TCP connection may get into a state where no
	data can be sent, requiring another connection.
	This has been referred to as window closure, and
	avoiding persistent forms of it is a requirement

<IMHO> There appears to be an additional goal that is
implicit in some of this discussion:
[4] A multiple connection session may simplify implementations
	that support multiple physical connections by using one
	SCSI for multiple TCP instances rather than one-for-one.
	This is a design/implementation-specific optimization.
</IMHO>

I don't see a requirement that multiple TCP connections per
iSCSI session be used to increase utilization of a single
physical link, although in practice if we specify multi-connection
sessions, it's hard to prevent their usage in this fashion.  I
don't think this merits further discussion, because even if 
we prohibit multi-connection iSCSI sessions, we probably
can't stop someone from opening up multiple iSCSI sessions
over the same physical link.

There are three application scenarios of interest:
[A] Direct host or server access to disk storage.
[B] Access to tape, usually by a host or server, but
	third party copy engines that support tape
	targets would fall into this category.
[C] Disk storage to disk storage mirroring, replication, etc.

So picking up on the requirements and goals:

--- [1] Utilize physical network parallelism.
- [A] Server/host to disk storage.

The conclusion of the long discussion of wedge drivers is that
multiple connection sessions are not required to use network
parallelism for disk access.

- [B] Tape access

Wedge drivers don't work for tape, hence multiple TCP sessions
per iSCSI session is the only way for iSCSI tape access to take
advantage of network parallelism.

- [C] Disk mirroring/replication, etc.

<IMHO> While these may start out as just remote disk interfaces,
in practice, they seem to rapidly acquire additional functionality
above and beyond basic SCSI, resulting in a wedge driver-like
protocol layer that can handle the ordering issues that arise in
using network parallelism.  Examples of this include EMC's SRDF,
and IBM's PPRC, both of which include significant enhancements
above the basic SCSI or ESCON disk interface.  Hence this case
does not appear to generate a requirement for multi-connection
sessions.
</IMHO>

--- [2] Error recovery

[A] and [C] are similar in that there's higher level logic
(e.g., a SCSI driver, a wedge driver, a mirroring/replication
protocol) above it that can attempt to recover by retrying
command.  A concern here is that these retries tend to be
expensive.  [B] tape is different because a command retry may
result in the command being executed twice; while disks generally
behave correctly in this situation, tapes often do not.

<IMHO> The technical case for error recovery seems weak on a
couple of grounds.  The first is that TCP's response to
something not getting through is to try again and again.  If
TCP has to give up, it has likely done so after a significant
delay comparable to the cost of the higher level retry,
making the gain from optimizing the retry less clear.
There are ways to make TCP give up quickly - something
else on the LAN noticing that connectivity is gone and issuing
an RST comes to mind, but I wonder if these sort of situations
are sufficiently common to be worth optimizing for.

Second, TCP handles most/many of the failure situations that
are causing the recovery logic to be put into FCP-2.  If Fibre
Channel drops a frame from a backup session with a tape device,
with current FCP the backup has to be aborted.  The corresponding
situation with TCP is that the frame will be retried and succeed,
and the backup session will continue.

My conclusion is that I don't see a strong technical case for
use of multi-connection sessions for error recovery.
</IMHO>

--- [3] TCP Window Closure

There's been a lot of traffic here, but most of the problems
appear to be scenarios that are consequences of poor design
choices (e.g., if a large blob of data causes a target not
to be able to accept commands for a long time, the target
should never have permitted the data to be sent).  I haven't
seen anything that makes multiple TCP connections per session
to avoid the consequences of window closure a requirement.

<IMHO>
--- [4] Implementation simplification

If I were building a multi-port iSCSI over Ethernet controller,
I could see how only having one SCSI instance rather than
multiple instances might make the design simpler and hence
more feasible.  The "If" at the start of this sentence seems
to make this implementation specific, and hence, while this
may be a nice thing to have, it doesn't seem to be a requirement.
</IMHO>

Summarizing the above discussion:

[Consensus Attempt 1]  I don't see anything that REQUIRES
two TCP connections in all iSCSI sessions, because I haven't
seen any TCP window closure scenarios [3] that aren't avoidable
by reasonable design/engineering of the target.  I therefore
propose to re-establish the WG consensus that multiple TCP
connections per iSCSI session is OPTIONAL.

[Consensus Attempt 2]  One of the points of this long email
was to explore whether sessions are important enough that
they need to be specified as part of iSCSI.  Between tape
parallelism [B] and <IMHO> implementation optimization [4]
</IMHO>, I think sessions are important enough to specify
even though they are OPTIONAL to implement.

<IMHO>
[Procedural Suggestion]  It appears to me that the implementation
optimization goal [4] that spans all uses of iSCSI is more
important than the network parallelism [1] which is only
required by tape.  This would suggest according more weight to
the concerns of those working on implementations, especially hardware.

[Issue] The technical arguments for error recovery among
the TCP connections in a multi-connection iSCSI session
strike me as weak.  It is not clear whether error recovery
should remain a requirement for multi-connection sessions.
</IMHO>

Where we go from here:
- There are two consensus attempts above, please register
	any objections to them promptly on the list.
- There are a number of places where I've expressed my
	technical opinion, which are mostly set off by <IMHO></IMHO>.
	Feel free to express an alternate opinion.
- Comments on the procedural issue would be appreciated,
	as it may affect how I attempt to call consensus in the
	future.
- In addition to general comments on the error recovery
	issue, could Joe Breher from Exabyte and anyone else
	familiar with the specifics of tape behavior comment
	on whether TCP's error recovery (retry with duplicate
	detection that achieves reliable delivery in the face of
	transient communication drops and errors) is good
	enough for tape?  Resolving that specific point will
	help resolve the general issue.
- Can we refrain from discussion of the actual session model
	proposals until the items raised in this email are
	relatively settled?

Ok, fire away ...

Thanks,
--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
black_david@emc.com       Mobile: +1 (978) 394-7754
---------------------------------------------------