RE: IP Storage Framework document

[Dave_Lee@3com.com]

My position is that we still need to consider TCP, UDP, and other options in the
initial discussions.  Otherwise, you give up on even knowing about possible
advances through an arbitrary decision without hearing them.  Perhaps some of
those advances may even help a TCP based approach -- who knows until you see
them.  This is one of the purposes of the pre-BOF meeting to try to settle some
of this out before the BOF.  We don't know if Sun, HP, Seagate, etc. have other
proposals that may be better than anything this group has.

To stimulate more discussion, let me approach the issue from another vector.
Isn?t one of the nice features about IP is that it can be diverse and support
multiple transports?  If we accept your arguments that TCP is the solution, why
not do away with IP and create TCP flows through the network?  What if we
identify features or requirements for IP storage that would benefit from
capabilities that TCP does not provide, such as multicast?  Should we ignore
that and still stick with TCP?  Not at all clear to me.

I just don?t know if everybody will buy argument that TCP is the single cure-all
for all the network?s ills.  That slow start thing is a particular concern for
me.  The additional cost to do MTU discovery (which everybody doesn?t do
currently) is a concern to me -- why bother if it?s always 1500?  And so forth.
I can find an argument for TCP and one against TCP, which seems to say that we
should be flexible in our thinking.

I?ll grant you that some simplistic TCP accelerations do exist today.  Doing
minor things to the transmit path is easy -- the receive processing,
retransmits, and other things that make TCP what it is and account for a nice
part of the overhead are hard.  There are a slew of vendors (many startups)
promising us network processor solutions that will solve all our problems.  I?m
sure they?re promising you guys as well.  But they?re vaporware at this point.
Are you going to commit your products on the mere possibility that somebody
unproven will provide a critical component?

It?s just not clear to me that TCP in silicon is cost-effective, especially at
10/100 speeds.  For ultra-low cost devices, the cost of the extra silicon may be
a concern to some.  Initially, the cost will be high and even then, if I?m price
sensitive and space sensitive, then it would be a very serious issue to me.
It?ll be awhile before we see fully integrated, single chip solutions.  Bottom
line, focusing solely on the WAN solution as the only solution before we see the
alternatives and betting on unproven technology may not be the best strategy.

One might argue that if it is a simple UDP based protocol, it will be
significantly easier to implement the *complete* protocol in hardware than
something based ontop of TCP.   I hope UDP is not new fangled protocol, at any
rate. :-)  Some of the UDP-related problems that you are mentioning, we can
clearly provide optional mechanisms to cure it for those who want to cure it.
Key word here is optional.  Those who do not want to use these options, they
don?t -- and they still interoperate.

It is probably not in the IETF?s interest to be dictating to vendors and
customers on how they build their networks and storage systems.  We want to
enable them to be able to provide and to use interoperable storage solutions as
they see fit.  To enable the most broadly interoperable solution would be the
best answer.

Dave




Andreas Bechtolsheim <avb@cisco.com> on 02/04/2000 06:48:19 AM

Sent by:  Andreas Bechtolsheim <avb@cisco.com>


To:   cheriton@cisco.com, Dave Lee/HQ/3Com, avb@cisco.com,
      mark_bradley@btc.adaptec.com
cc:   See Below
Subject:  RE: IP Storage Framework document



Mark,

I think this whole argument comes down to the question
"is it easier to put TCP/IP into hardware or is it easier
to invent a new IP protocol to replace TCP functionality
because such a protocol is easier to put into hardware?"

Along these lines and in response to your points below,
I would like to offer the following observations:

1. It appears straightforward to implement the "standard" TCP/IP control
and data path in hardware, leaving more complex options to software.
Both the latest 3COM and Intel Gigabit Ethernet NICs have
added hardware support for the TCP/IP transmit path.
There are several startups building full TCP/IP hardware implementations,
and people are busy adding hardware functions such as IPSEC security.
Seems to me the time has arrived to take advantage of contemporary
silicon technology for the network interface controller.

2. The estimated cost for a TCP/IP hardware stack is 200K gates.
This is about 2 mm^2 in 0.15um CMOS or less than $1 worth of silicon.
To believe that TCP/IP hardware is a cost issue is a fallacy.

3. While one can always build separate networks for storage,
just like Fibrechannel is today, doing so of course removes the
strongest motivation to converge storage on IP in the first place.

4. One can implement Storage over TCP/IP in software today with
off-the shelf 100 Mbit and Gigabit Ethernet NICs and switches.
People are reporting close to 1 Gbps wirespeed TCP/IP performance with
the latest off-the-shelf Gigabit Ethernet adaptors which is
more than enough for small-office/home-office storage I/O.

Bottomline, while designing TCP/IP hardware NICs requires some effort,
the most important enabling factor is that TCP/IP has been a stable
architecture that has achieved extremely wide world-wide deployment
measured in the 100s of Millions and soon Billions of end nodes.

Thus IMHO the opportunity is to address and leverage this large market,
rather than inventing yet another network layer and building hardware
for one particular purpose that today has an installed base of zero.



> From mark_bradley@btc.adaptec.com Fri Feb  4 04:45:51 2000
> From: "Bradley, Mark" <mark_bradley@btc.adaptec.com>
> To: "'David R. Cheriton'" <cheriton@cisco.com>, Dave_Lee@3com.com,
>         Andreas Bechtolsheim <avb@cisco.com>
> Cc: "Bradley, Mark" <mark_bradley@btc.adaptec.com>, Carl_Madison@3com.com,
>         Arun_Verma@3com.com, kquick@iphase.com, bassoon@ece.cmu.edu,
>         "VonStamwitz, Paul" <paulv@corp.adaptec.com>,
>         "Wilson, Andrew"
>     <dwilson@corp.adaptec.com>, raz@emc.com,
>         avb@cisco.com, rcampbel@cisco.com, Jgw@cisco.com, ddecapit@cisco.com,
>         sob@harvard.edu, vern@aciri.org, csapuntz@cisco.com
> Subject: RE: IP Storage Framework document
> Date: Fri, 4 Feb 2000 05:46:51 -0700
> MIME-Version: 1.0
> X-SMTP-HELO: magic.adaptec.com
> X-SMTP-MAIL-FROM: mark_bradley@btc.adaptec.com
> X-SMAP-Received-From: outside
> X-SMTP-PEER-INFO: magic.adaptec.com [208.236.45.80]
>
> Hi, David.  Thank you for your thoughts.  In answer to your
> question, I think that:
> 1)TCP in h/w is probably possible, but as others have found, it
>   is very difficult to do.  A simpler protocol in hardware is
>   highly desirable
> 2)Cost.  TCP h/w implementations are likely to be complex silicon
>   designs that one could expect are costly to provide.
> 3)While it is critical that IPS be cohabitable and non-destructive
>   to network performance, it is entirely possible that a separate
>   physical layer would be used for performance reasons.  I can
>   envision many scenarios where we do this today, sans IPS.
> 4)Performance.  Both in CPU utilization and for pure bandwidth, it
>   is desirable to go light to minimize impact for those who might
>   not even implement in H/W.
> Lastly, one envisions large networks in considering your points
> below.  For small networks (SOHO, small business), your points
> are likely less important for many reasons, including network
> loading.  Small networks with generall light traffic are less
> sensitive to the issues you bring up.  They also tend to be more
> cost sensitive.
>
> These are the points that it I think you may not be sensitive to.
>   --  markb
>
> -----Original Message-----
> From: David R. Cheriton [mailto:cheriton@cisco.com]
> Sent: Friday, February 04, 2000 12:06 AM
> To: Dave_Lee@3com.com; Andreas Bechtolsheim
> Cc: mark_bradley@btc.Adaptec.Com; Carl_Madison@3com.com;
> Arun_Verma@3com.com; kquick@iphase.com; bassoon@ece.cmu.edu;
> paulv@corp.adaptec.com; dwilson@corp.adaptec.com; raz@emc.com;
> avb@cisco.com; rcampbel@cisco.com; Jgw@cisco.com; ddecapit@cisco.com;
> sob@harvard.edu; vern@aciri.org; csapuntz@cisco.com
> Subject: Re: IP Storage Framework document
>
>
> I actually dont think there is any other transport protocol to
> consider other than TCP.  Here's why.
> 1) TCP has been developed, refined, evaluated and deployed over a
>    period of 20 years, with some of the smartest folks in the
>    Internet research community spending years of effort.
>    There is NO hope in duplicating this effort, IMHO,
>    and plenty of evidence to suggest you'll screw up or end up
>    with TCP with gratutious differences if you dont.
> 2) TCP has been the most successful transport protocol in the history
>    of mankind, and continues to grow dramatically in use.
>    There are numerous other attempts now and in the past to create
>    a new transport protocol for one reason or another, but the
>    success level has been almost 0 i.e. they're not all dead YET.
>    (I was even responsible for one old attempt, blush!)
> 3) The TCP fast path is reasonably simple, has been identified
>    in software implementations and used successfully for years.
>    A hardware implementation of that fast path seems feasible to me,
>    and, even if a little more difficult than what one might imagine
>    for some new fangled protocol, the pay-off for a fast hardware
>    TCP is certain, and huge - not so with any other protocol.
>
> I think NFS/UDP is problematic.  Even in a campus network, you can
> have systematic loss of IP fragments that UDP too often generates,
> causing communication failures and network overload.
> And, handling IP fragment reassembly is one of the more complex
> aspects of hardware IP.  With a hardware TCP, I think one could
> rely on MTU discovery and punt IP reassembly to software as
> an unusual and unwarranted case.
> Finally, my perception is that many netadmins are trying to reduce
> and restrict non-TCP traffic on their networks because it does not
> adapt predictably to network loads, causing failures.  A high-rate
> UDP or random new protocol load on a network would not be welcomed.
> I'd also mention that the same forces that are causing networks
> to converge on IP are pushing for convergence on TCP as well.
> Diversity is not nice in a mission-critical 7x24 enterprise network
> that is struggling with rapid scaling in users, bandwidth, services.
>
> So, the only thing that would convince me to consider another
> transport protocol would be some major flaw in the above reasoning,
> or proof that you just cant do hardware TCP at the required rates.
> The latter would really, really surprise me.
>
> What am I missing?
>
> At 04:54 PM 2/3/00 -0800, Dave_Lee@3com.com wrote:
> >
> >
> >
> >Andy -
> >
> >You point out excellent issues that must be considered.  Let me provide
> some
> >additional comments for all to ponder.  Before I start, I believe the
> document
> >authors are using the term IP in its most generic sense and not meant to
> imply a
> >particular transport such as TCP, UDP, or some other transport.  Drew/Paul
> --
> >feel free to correct me if I'm wrong.  There are certainly many
> possibilities
> >and I think they all need to be considered at this point.
> >
> >While it may be theoretically possible to implement TCP in silicon, it is
> not at
> >all clear if TCP in silicon will occur anytime soon -- if at all.  Betting
> on
> >that fact may not sit well with others, especially those concerned with
> higher
> >performance campus-based solutions that they want available now.  Nothing,
> >however, precludes us from taking advantage of TCP in silicon in a future
> >revision of the protocol(s).
> >
> >For WAN traffic, I think most people would agree with you on your remark
> on the
> >need for bandwidth management.  Further, I believe most people would
> probably
> >agree with you that if we were looking for a pure L2 solution (e.g.,
> directly
> >attached LAN), the IETF is *not* the right place to take this.  I
> definitely do.
> >
> >However, a customer may want to have a storage solution that can be
> accessed
> >over a campus network.  In this case, a L2 solution is not sufficient and
> IP is
> >required.  Further, it does not seem to be at all clear if TCP bandwidth
> >management is a requirement in this environment.  The NFS version 4
> requirements
> >document has some commentary on this discontinuity.  In a campus network,
> there
> >are other possible ways to do bandwidth management, including traffic
> >engineering, differential services, etc.
> >
> >In light of the NFS discussion, perhaps looking to NFS may not be a bad
> thing to
> >do?  NFS can operate over both TCP and UDP.  Perhaps this model is also a
> good
> >model for IP storage.
> >
> >Dave
> >
> >
> >
> >
> >Andreas Bechtolsheim <avb@cisco.com> on 02/03/2000 09:14:54 AM
> >
> >Sent by:  Andreas Bechtolsheim <avb@cisco.com>
> >
> >
> >To:   See Below
> >cc:   See Below
> >Subject:  Re: IP Storage Framework document
> >
> >
> >
> >While I will not be able to attend this meeting in person, I would
> >like to propose an alternative framework for Storage over IP.
> >We will discuss this in more detail at the 2/16 Meeting in San Jose.
> >
> >The key characteristics of our proposal are as follows:
> >
> >1. It is based on TCP/IP, so all the issues of a reliable
> >   end-to-end byte stream including error recovery and
> >   congestion management are addressed without inventing
> >   a new protocol to do the same functions.
> >
> >2. It works over any network topology and any distance,
> >   addressing the needs of both local and remote connectivity
> >   between servers and storage devices.
> >
> >3. The protocol can be implemented efficiently in hardware
> >   inside network interface controllers allowing for
> >   high-performance wirespeed network operation.
> >
> >While in the past several new protocols have been proposed to
> >accelerate certain types of data transfers, typically with
> >the argument that they are easier to implement in hardware,
> >experience has shown that such protocols have failed to gain
> >market acceptance. With the plummenting cost of silicon,
> >at this point in history the cost of implementing TCP/IP in
> >NIC silicon is in the order of a few $ worth of silicon
> >so there is simply no economic reason not to do it.
> >
> >In addition, proposing protocols that do not adhere to the
> >TCP/IP bandwidth management algorithms pose a serious risk
> >for overall network stability, in particular if they are used
> >for high bandwidth traffic such as disk I/O.
> >
> >Finally, designing new protocols for limited topologies such
> >as local area networks does not appear consistent with the
> >goals of the IETF.
> >
> >
> >
> >To: Dave Lee/Hq/3Com               Raz @Emc.Com
> >    Mark_Bradley @Btc.Adaptec.Com  Avb @Cisco.Com
> >    Carl Madison/Us/3Com           Rcampbel @Cisco.Com
> >    Arun Verma/Hq/3Com             Jgw @Cisco.Com
> >    Kquick @Iphase.Com             Ddecapit @Cisco.Com
> >    Bassoon @Ece.Cmu.Edu           Sob @Harvard.Edu
> >    Paulv @Corp.Adaptec.Com        Vern @Aciri.Org
> >    Dwilson @Corp.Adaptec.Com
> >
> >cc: Avb @Cisco.Com                 Csapuntz @Cisco.Com
> >    Cheriton @Cisco.Com
> >
> >
>
>


cc: Carl Madison/Us/3Com           Rcampbel @Cisco.Com
    Arun Verma/Hq/3Com             Jgw @Cisco.Com
    Kquick @Iphase.Com             Ddecapit @Cisco.Com
    Bassoon @Ece.Cmu.Edu           Sob @Harvard.Edu
    Paulv @Corp.Adaptec.Com        Vern @Aciri.Org
    Dwilson @Corp.Adaptec.Com      Csapuntz @Cisco.Com
    Raz @Emc.Com