Re: iSCSI Autosense Consensus, Connection next steps

["Randall R. Stewart" <randall@stewart.chicago.il.us>]

John:

Here are some of mine...(interspersed with Joshua's email)

John Hufferd/San Jose/IBM wrote:
> 
> I think Joshua has made some interesting points, do the rest of you have
> some supportive, or contrary opinions about what he has said?
> .
> .
> .
> John L. Hufferd
> 
> Joshua Tseng <jtseng@NishanSystems.com> on 09/05/2000 06:52:54 PM
> 
> To:   John Hufferd/San Jose/IBM@IBMUS, Joshua Tseng
>       <jtseng@NishanSystems.com>
> cc:   ips@ece.cmu.edu
> Subject:  RE: iSCSI Autosense Consensus, Connection next steps
> 
> John,
> 
> >My question then is: does Data Balancing on multiple
> >connections cause a similar need for additional memory as does the
> >synchronous connections with Multiple Commands on different connections.
> 
> In the symmetric model, the target must be prepared to buffer all of the
> commands and associated unsolicited data PDU's that may be sent between
> ExpCmdRN and MaxCmdRN.  This can be a lot of memory.  Either that or
> discard
> commands.  But in order to realize the performance benefits of command
> windowing, the target must expand the window in order to let the initiator
> to "let her rip!" and put multiple commands in flight.  It seems to me that
> the existing symmetric model adds complexity which would only be of benefit
> if the target has lots of memory.  So my question is this:  The command
> windowing mechanism seems to require lots of resources (including memory)
> and four 32-bit fields (CmdRN, ExpStatRN, StatRN, and MaxCmdRN) to
> implement, but does all this really buy very much?  In the end, symmetric
> implementations may end up discarding commands anyway, as in the asymmetric
> case, unless they want to install globs of memory.
> 
> WRT the load balancing algorithm implemented by iSCSI, anything other than
> round robin will require more complexity to sense which connection is
> over-utilized or poorly performing.  Once again, what does the added
> complexity buy us?  I'm not an development engineer, but my guess is
> getting
> iSCSI to talk to TCP to get this info and use it isn't cheap.  In
> asymmetric, commands are sent in-order down a relatively less-utilized
> command TCP connection.  A slow TCP connection only affects unfortunate
> exchanges whose data PDU's happen to use that connection, and not all
> sessions between initiator and target.
> 

It will be highly stack dependant on WHAT TCP information is available
(if ANY) and how you access it. In many cases the ONLY way you can get
at the types of information this would require is by reading the
/dev/memory and gloming through the kernel.. el.yucko.. and even
this may not be able to get all the information you need.

R


> Josh
> 
> -----Original Message-----
> From: John Hufferd/San Jose/IBM [mailto:hufferd@us.ibm.com]
> Sent: Tuesday, September 05, 2000 4:53 PM
> To: Joshua Tseng
> Cc: ips@ece.cmu.edu
> Subject: RE: iSCSI Autosense Consensus, Connection next steps
> 
> Joshua,
> I think some of your points are correct.  But we also need to examine a few
> statements.
> 
> First, the round robin approach of balancing, if only done in the manor you
> specified,  would of course cause every thing to slow to the speed of the
> slowest link.  But this is a well known problem, and general managed by a
> modified round robin so that it continues to other connections if one of
> the connections is still busy (several different way of determining this).
> In fact this would be a problem in the current wedge drivers today used
> with Fibre Channels if only a blind round robin approach was used.
> 
> Your point about the target perhaps needing additional memory, is probably
> correct.  This is also solvable on the initiator side (by a number of
> techniques)  however, the target can not be sure that the initiator will
> have the right techniques, so it must assume that it needs additional
> memory.  This seems like a correct point.  If this is a worry, however, the
> Target can just reject additional conversations per Session.  The counter I
> would make to my last point, however, would be if Asynchronous was used, I
> could still could handle multiple data ports and at least get that kind of
> balancing.  My question then is: does Data Balancing on multiple
> connections cause a similar need for additional memory as does the
> synchronous connections with Multiple Commands on different connections.
> 
> On the other hand, think that your point about link recovery not being
> (significantly) more difficult with Async then Sync is essentially correct.
> 
> .
> .
> .
> John L. Hufferd
> 
> Joshua Tseng <jtseng@NishanSystems.com>@ece.cmu.edu on 09/05/2000 02:54:56
> PM
> 
> Sent by:  owner-ips@ece.cmu.edu
> 
> To:   Black_David@emc.com, Julian Satran/Haifa/IBM@IBMIL, ips@ece.cmu.edu
> cc:
> Subject:  RE: iSCSI Autosense Consensus, Connection next steps
> 
> David,
> 
> If I may give my input on the TCP connection model discussion, I'd like
> to advocate the asymmetric model for the following reasons.  Using
> multiple TCP connections, each TCP connection will likely have
> different performance characteristics.  If this is so, then some commands
> may reach the target out of order from other commands sent down different
> TCP connections.  Commands sent down the faster connection will then need
> to be buffered until commands sent down slower connections arrive.  This
> will lead to the following problems that I can think of:
> 
> 1)  There will be increased memory requirements for the iSCSI target,
> which must buffer commands and related data and reorder them before passing
> them on to the SCSI controller.
> 
> 2)  If the initiator cannot sense the performance characteristics of
> each TCP connection, then interactions between initiator and target will
> be dependent on the slowest TCP connection.  For example, if there are
> five TCP connections and one of them is extraordinarily slow, and the
> initiator merely round-robins the commands down each of the five
> connections,
> all interactions between initiator and target will be only as fast as the
> slowest connection.
> 
> In addition, it has been stated that detecting and recovering from a failed
> asymmetric control connection is more complicated and difficult than
> detecting and recovering from a failed symmetric data/command TCP
> connection.
> I do not understand why this is so.  In both cases, iSCSI initiator will
> need
> to abort the lost command AND those following the lost command, and resend
> all
> commands in a new or different TCP connection.  In the symmetric case,
> iSCSI
> commands received by a target following a lost command will not be
> acknowledged,
> since ExpStatRN is a cummulative windowing mechanism.
> 
> Josh Tseng
> 
> -----Original Message-----
> From: Black_David@emc.com [mailto:Black_David@emc.com]
> Sent: Sunday, September 03, 2000 8:20 AM
> To: julian_satran@il.ibm.com; ips@ece.cmu.edu
> Cc: Black_David@emc.com
> Subject: iSCSI Autosense Consensus, Connection next steps
> 
> > It looks like we have consensus - but the chairmen have the call.
> 
> Indeed we do, and I apologize for the delay, due to an inability to
> get connected.
> 
> First, on Autosense, I believe that rough consensus exists for
> iSCSI to require Autosense.  I have seen only one objection;
> If anyone other than Doug Otis disagrees with this, please
> say so on the list.
> 
> Second, on connections, I haven't seen enough discussion to call
> consensus, but I am going to try to narrow the option space and
> structure the discussion.  Four models for sessions have been
> proposed:
> 
> (1) Symmetric - all connections usable for command and data.
> (2) Asymmetric - single command connection, others are data.
> (3) Split - assign LUN sets to specific connections or pools of
>      connections.
> (4) SCTP - use SCTP's support for multiple connections.
> 
> I have only seen one message suggesting/supporting (3) Split, so
> I believe rough consensus exists to not pursue that model
> any further.  If anyone other than Paul von Stamwitz disagrees,
> and believes that the Split model should be pursued, please
> say so on the list with technical rationale.
> 
> I haven't seen enough discussion on use of SCTP instead of
> TCP to call consensus.  I am concerned that this list is not
> going to produce a consensus on the issue phrased in that
> fashion (i.e., pick exactly one).  The one point on which there
> does seem to be consensus is that SCTP is a considerably
> younger protocol that TCP, and hence the likely timelines
> to availability of hardware acceleration seem to favor TCP.
> OTOH, this sort of future prediction can easily miss the mark.
> 
> I can see two possible ways to make progress in this area:
> - An off-line design team to do an intensive evaluation of SCTP
>      vs. TCP for iSCSI.
> - Recognize the merits of SCTP as well as TCP, plan for both
>      with the anticipation that TCP will be used first.
> The latter makes more sense to me, because it appears to
> be more amenable to consensus, avoids the investment
> of effort in the design team, and avoids having the prospect
> of abandoning TCP hanging over the heads of those engaged
> in TCP-specific work.  The practical implication of proceeding
> in this fashion is that SCTP-friendliness becomes an additional
> criteria to use in evaluating iSCSI design proposals.  Proceeding
> in this fashion is only a proposal at this juncture -- please
> comment on whether this is the right way to proceed (to me
> or on the list).
> 
> The specification of sessions for iSCSI over TCP needs to
> proceed, so under the assumption that either the "plan for
> both" path will be pursued, or that we can't wait for the
> design team's conclusions, and hence need to work on TCP
> in case it is selected, I want to summarize the tradeoffs
> between the Symmetric and Asymmetric session models,
> in the aim of simulating more discussions so that we can
> get to consensus.
> 
> [X] The Symmetric model imposes additional work on implementations
>      that do not use multiple connections because they will
>      have to implement the command numbering.  The
>      Asymmetric model does not do this.
> [Y] The Asymmetric model makes dealing with a failed control
>      connection more complicated than the symmetric model
>      because agreement between the two sides is required
>      to establish a new control connection (or use an existing
>      data connection for control).
> 
> I have seen the discussion of [X] favoring Asymmetric for systems
> that will not support multiple TCP connections per iSCSI session.
> I also note (as an example of applying the SCTP-friendliness
> from above) that [X] favoring Asymmetric also applies to SCTP, as
> there seems to be no point in using multiple SCTP sessions in a
> single iSCSI session.
> 
> We need to give the folks working on revising the iSCSI draft some
> direction on this issue in fairly short order, so comments are hereby
> solicited ...
> 
> Thanks,
> --David
> 
> ---------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 42 South St., Hopkinton, MA  01748
> +1 (508) 435-1000 x75140, FAX: +1 (508) 497-6909
> black_david@emc.com  Cellular: +1 (978) 394-7754
> ---------------------------------------------------

-- 
Randall R. Stewart
randall@stewart.chicago.il.us or rrs@cisco.com
815-342-5222 (cell) 815-477-2127 (work)