RE: iSCSI: Session Partial Resolution

["HAAGENS,RANDY (HP-Roseville,ex1)" <randy_haagens@hp.com>]

David,

I. Re: proposed points of consensus

Add my name to the list of those objecting to (1) below.  OTOH, (2) The use
of more than one TCP connection per session *is* optional in the current
draft, or at least in the last version of the draft that I read, and the one
that we're all implementing.

(1) Command sequencing was added to the draft for several reasons, but the
important point here is that command flow control is necessary when using a
single TCP connection in order to prevent a command queue full condition
from preventing the reception of write data.  We could just leave command
flow control to the good behavior of the SCSI layer (there is no SCSI
mechanism defined for it, and so it is implementation-dependent), but we
decided that it was legitimate to treat it at the iSCSI "transport" layer,
since from SCSI's pov, it is a transport problem.  Command sequencing also
solved two other problems (ordered command striping and recovery from TCP
connection failure), so it seemed like an efficient mechanism overall.

Some have objected that command windowing may result in the command queue's
being blocked, and urgent commands' not getting through.  If I recall the
draft correctly, we made provision for this with the notion of unnumbered
commands.  These can be sent at any time by the initiator; but since they do
not have a reserved place in the command queue, their reception must be
considered unreliable: the target can drop them at its discretion.  I
believe this situation is familiar to the T10 community, and acceptable from
a SAM-2 perspective.  But if not, we could create a separate, sequenced,
command queue for urgent commands.

The question of the storage controller's overcommitting command queue
credits has been raised.  Our initial thought was that this would not be
allowed.  But if it must be allowed, then we could extend the command
sequencing scheme to allow to target to drop commands and later demand their
retransmission.  The command sequence numbers would ensure that, ultimately,
all commands are delivered in order, with no commands lost or duplicated.

In general, if one is going to maintain queues of things of different types,
then intermixing these things on a single TCP connection is going to require
higher-level flow control, in order to prevent congestion at the TCP
connection level.  The alternative is to separate these queues into separate
TCP connections: one connection for normal commands, one for urgent
commands, one or more for write data, etc.  

(2) Use of more than one TCP connection per iSCSI session currently is
optional.  I don't think that there's any requirement for a target to
support or for an initiator to use more than one TCP connection per session.
But note that a single connection is necessarily symmetrical.  Starting from
the desire to support simple iSCSI sessions consisting of only one
connection, one is led to symmetry and then to a solution to the
command/data multiplexing problem (1).

In summary:

(a) A single TCP connection per session is considered desirable by many.
This single connection is necessarily "symmetrical".  And it requires a
solution to the command/data head-of-line blocking problem.  Command
sequencing provides an elegant solution to this problem, and several others
as well.  Given a solution to these problems, it's possible additionally to
implement a symmetrical connection model where multiple TCP connections are
aggregated into one session.  But this aggregation capability is not
required.

(b) Alternatively, if we use TCP connections liberally and asymmetrically,
then we can dispense with higher-level flow control mechanisms, relying
instead on TCP flow control for each distinct message type.  But in this
case, we cannot build a session consisting of a single connection.  It must
include at least one connection of each type.

II.  Re: "session model for multiple connection sessions"

Here the real issue is not symmetrical/asymmetrical.  Rather, it's whether
to support connection/bandwidth aggregation at all.  Possibly we should
revisit the requirement.  But I doubt your plan of separating it from the
main body of the iSCSI work, as I'm fearful of not being able to
re-integrate it later.

To my mind, aggregation is an obvious requirement: at every stage of
technology, its desirable to use N of the fastest things you have at once,
to go even faster.  SCSI defines a lot of task queue management features
that are dependent on commands' being delivered in order by the SCSI
transport.  So any single-transport aggregation technique must support
ordered command delivery.

Today, we all build hosts and storage controllers with multiple Fibre
Channel interfaces.  And until recently, the SCSI SAM-2 document explicitly
recognized multiport storage controllers.  One limitation of today's
devices, however, is that they must treat their multiple ports as
independent.  They constitute separate SCSI "Service Deliver Ports"
(transport end points), and it's not possible to uphold SCSI ordering
constraints across SCSI commands delivered on these separate ports.  This is
fine for load balancing of disk IOs, which frequently are unordered anyway.

But it doesn't work for two important applications: writes to sequential
devices (tape) and asynchronous (but ordered) disk mirroring.  These are two
important MAN/WAN applications.  We could exclude them from the iSCSI
requirements if we chose; but I'd like to see some debate on that point at
the requirements level.

iSCSI command numbering permits command ordering to be maintained across the
several TCP connections in a session.  This allows bandwidth aggregation
through the technique of "ordered command striping".

Command striping has one other desirable property for aggregating even
commands that have no ordering requirement: Striping round-robin across N
connections means that the aggregate will proceed at N times the rate of the
slowest connection.  Yes, this is a desirable property, because it means
that if one connection experiences congestion, the entire aggregate will
back off.  It probably is not appropriate simply to shift load to a faster
(not-yet-congested) connection, as that practice likely defeats TCP's
congestion control.  This result is conter-intuitive to the max, and
contradicts the practice of current path-balancing software; but I think it
deserves further consideration.

R


Randy Haagens
Director, Networked Storage Architecture
Storage Organization
Hewlett-Packard Co.
tel. +1 916 785 4578
e-mail: Randy_Haagens@hp.com


-----Original Message-----
From: Black_David@emc.com [mailto:Black_David@emc.com]
Sent: Tuesday, September 19, 2000 12:22 PM
To: hufferd@us.ibm.com; Black_David@emc.com
Cc: ips@ece.cmu.edu
Subject: iSCSI: Session Partial Resolution


> Perhaps I am wrong but it seems we are drifting and not focusing on the
> search for concurrence regarding Symmetric vs Asymmetric.  What is your
> view?

John's acquiring the prescient talent of causing the
co-chair to appear and take action ... as long as he
doesn't start calling himself a "wizard" based on these
summoning powers, I won't object :-).  In any case ...

In the past week, I have seen at most one objection
to each of the following two proposed points of consensus:

(1) An iSCSI session containing a single TCP connection
	should not be required to use the currently specified
	iSCSI command reference numbers and sliding window
	mechanism because TCP will deliver commands in order.
(2) Use of more than one TCP connection per iSCSI session
	is OPTIONAL.

Therefore I declare these to be the WG rough consensus on
these issues, and the next version of the iSCSI draft
should remove the command reference numbers and sliding
window mechanism from the iSCSI header.  Somesh Gupta's
objection to (1) and Matt Wakeley's continued objection to
(2) are noted as part of declaring these to be the WG rough
consensus.  Anyone else who objects to this declaration
of rough consensus should email me directly with the
reasons for the objection.

OTOH, I do not see consensus on the session model for
multiple connection sessions among the Asymmetric model,
the Symmetric model, and Pierre Labat's proposal.  In
order to make progress on iSCSI, I see no alternative to
separating multi-connection sessions from the main iSCSI
spec.  Significant effort and email traffic has been
invested in this topic for at least 6 weeks and the issue
is not settled -- I don't think holding up the iSCSI spec
for another 6+ weeks in hopes of settling this issue on
the list is an effective way to make progress, but I'm
prepared to listen to dissenting opinions (e.g., if
someone thinks there is rough consensus, and I've missed
it); please send such opinions directly to me rather than
using the list.  I've already had one offline comment from
an outside observer expressing amazement at the willingness
of this community to discuss multi-connection sessions
"ad nauseum".

Therefore, I would ask that the authors of the next
version of the iSCSI draft delete all specification
of multiple connection sessions from the next version
of the except for a note that they will be handled in 
a separate document.

Producing that separate document is going to require
an offline design team.  The design team can either be
chartered to write a compromise session specification
or to evaluate competing specifications and choose one.
My current inclination is to do the latter, which would
involve having the design team produce a set of requirements
and guidelines for session specifications in consultation
with the co-chairs, evaluate Internet-Drafts documenting
the specifications, and recommend an approach to the WG.
Comments on this process are solicited - either on the
list or to me directly.  Further discussion of multi-connection
sessions on the list is probably not a good use of list
bandwidth.

--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
black_david@emc.com       Mobile: +1 (978) 394-7754
---------------------------------------------------



Randy Haagens
Director, Networked Storage Architecture
Storage Organization
Hewlett-Packard Co.
e-mail: Randy_Haagens@hp.com
tel: +1 916 785 4578
fax: +1 916 785 1911
 <<Haagens, Randy (E-mail).vcf>>

Haagens, Randy (E-mail).vcf