RE: IP Storage Framework document

[Andreas Bechtolsheim <avb@cisco.com>]

Mark,

I think this whole argument comes down to the question
"is it easier to put TCP/IP into hardware or is it easier
to invent a new IP protocol to replace TCP functionality
because such a protocol is easier to put into hardware?"

Along these lines and in response to your points below, 
I would like to offer the following observations:

1. It appears straightforward to implement the "standard" TCP/IP control 
and data path in hardware, leaving more complex options to software.
Both the latest 3COM and Intel Gigabit Ethernet NICs have
added hardware support for the TCP/IP transmit path.
There are several startups building full TCP/IP hardware implementations, 
and people are busy adding hardware functions such as IPSEC security.
Seems to me the time has arrived to take advantage of contemporary
silicon technology for the network interface controller.

2. The estimated cost for a TCP/IP hardware stack is 200K gates.
This is about 2 mm^2 in 0.15um CMOS or less than $1 worth of silicon.
To believe that TCP/IP hardware is a cost issue is a fallacy.

3. While one can always build separate networks for storage,
just like Fibrechannel is today, doing so of course removes the
strongest motivation to converge storage on IP in the first place.

4. One can implement Storage over TCP/IP in software today with
off-the shelf 100 Mbit and Gigabit Ethernet NICs and switches.
People are reporting close to 1 Gbps wirespeed TCP/IP performance with 
the latest off-the-shelf Gigabit Ethernet adaptors which is
more than enough for small-office/home-office storage I/O.

Bottomline, while designing TCP/IP hardware NICs requires some effort,
the most important enabling factor is that TCP/IP has been a stable
architecture that has achieved extremely wide world-wide deployment
measured in the 100s of Millions and soon Billions of end nodes.

Thus IMHO the opportunity is to address and leverage this large market,
rather than inventing yet another network layer and building hardware
for one particular purpose that today has an installed base of zero.



> From mark_bradley@btc.adaptec.com Fri Feb  4 04:45:51 2000
> From: "Bradley, Mark" <mark_bradley@btc.adaptec.com>
> To: "'David R. Cheriton'" <cheriton@cisco.com>, Dave_Lee@3com.com,
>         Andreas Bechtolsheim <avb@cisco.com>
> Cc: "Bradley, Mark" <mark_bradley@btc.adaptec.com>, Carl_Madison@3com.com,
>         Arun_Verma@3com.com, kquick@iphase.com, bassoon@ece.cmu.edu,
>         "VonStamwitz, Paul" <paulv@corp.adaptec.com>,
>         "Wilson, Andrew"
> 	 <dwilson@corp.adaptec.com>, raz@emc.com,
>         avb@cisco.com, rcampbel@cisco.com, Jgw@cisco.com, ddecapit@cisco.com,
>         sob@harvard.edu, vern@aciri.org, csapuntz@cisco.com
> Subject: RE: IP Storage Framework document
> Date: Fri, 4 Feb 2000 05:46:51 -0700 
> MIME-Version: 1.0
> X-SMTP-HELO: magic.adaptec.com
> X-SMTP-MAIL-FROM: mark_bradley@btc.adaptec.com
> X-SMAP-Received-From: outside
> X-SMTP-PEER-INFO: magic.adaptec.com [208.236.45.80]
> 
> Hi, David.  Thank you for your thoughts.  In answer to your
> question, I think that:
> 1)TCP in h/w is probably possible, but as others have found, it
>   is very difficult to do.  A simpler protocol in hardware is
>   highly desirable
> 2)Cost.  TCP h/w implementations are likely to be complex silicon
>   designs that one could expect are costly to provide.
> 3)While it is critical that IPS be cohabitable and non-destructive
>   to network performance, it is entirely possible that a separate
>   physical layer would be used for performance reasons.  I can
>   envision many scenarios where we do this today, sans IPS.
> 4)Performance.  Both in CPU utilization and for pure bandwidth, it
>   is desirable to go light to minimize impact for those who might
>   not even implement in H/W.
> Lastly, one envisions large networks in considering your points
> below.  For small networks (SOHO, small business), your points
> are likely less important for many reasons, including network 
> loading.  Small networks with generall light traffic are less
> sensitive to the issues you bring up.  They also tend to be more
> cost sensitive.
> 
> These are the points that it I think you may not be sensitive to.
>   --  markb 
> 
> -----Original Message-----
> From: David R. Cheriton [mailto:cheriton@cisco.com]
> Sent: Friday, February 04, 2000 12:06 AM
> To: Dave_Lee@3com.com; Andreas Bechtolsheim
> Cc: mark_bradley@btc.Adaptec.Com; Carl_Madison@3com.com;
> Arun_Verma@3com.com; kquick@iphase.com; bassoon@ece.cmu.edu;
> paulv@corp.adaptec.com; dwilson@corp.adaptec.com; raz@emc.com;
> avb@cisco.com; rcampbel@cisco.com; Jgw@cisco.com; ddecapit@cisco.com;
> sob@harvard.edu; vern@aciri.org; csapuntz@cisco.com
> Subject: Re: IP Storage Framework document
> 
> 
> I actually dont think there is any other transport protocol to
> consider other than TCP.  Here's why.
> 1) TCP has been developed, refined, evaluated and deployed over a
>    period of 20 years, with some of the smartest folks in the
>    Internet research community spending years of effort.
>    There is NO hope in duplicating this effort, IMHO,
>    and plenty of evidence to suggest you'll screw up or end up
>    with TCP with gratutious differences if you dont.
> 2) TCP has been the most successful transport protocol in the history
>    of mankind, and continues to grow dramatically in use.
>    There are numerous other attempts now and in the past to create
>    a new transport protocol for one reason or another, but the
>    success level has been almost 0 i.e. they're not all dead YET.
>    (I was even responsible for one old attempt, blush!)
> 3) The TCP fast path is reasonably simple, has been identified
>    in software implementations and used successfully for years.
>    A hardware implementation of that fast path seems feasible to me,
>    and, even if a little more difficult than what one might imagine
>    for some new fangled protocol, the pay-off for a fast hardware
>    TCP is certain, and huge - not so with any other protocol.
> 
> I think NFS/UDP is problematic.  Even in a campus network, you can
> have systematic loss of IP fragments that UDP too often generates,
> causing communication failures and network overload.
> And, handling IP fragment reassembly is one of the more complex
> aspects of hardware IP.  With a hardware TCP, I think one could
> rely on MTU discovery and punt IP reassembly to software as
> an unusual and unwarranted case.
> Finally, my perception is that many netadmins are trying to reduce
> and restrict non-TCP traffic on their networks because it does not
> adapt predictably to network loads, causing failures.  A high-rate
> UDP or random new protocol load on a network would not be welcomed.
> I'd also mention that the same forces that are causing networks
> to converge on IP are pushing for convergence on TCP as well.
> Diversity is not nice in a mission-critical 7x24 enterprise network
> that is struggling with rapid scaling in users, bandwidth, services.
> 
> So, the only thing that would convince me to consider another
> transport protocol would be some major flaw in the above reasoning,
> or proof that you just cant do hardware TCP at the required rates.
> The latter would really, really surprise me.
> 
> What am I missing?
> 
> At 04:54 PM 2/3/00 -0800, Dave_Lee@3com.com wrote:
> >
> >
> >
> >Andy -
> >
> >You point out excellent issues that must be considered.  Let me provide
> some
> >additional comments for all to ponder.  Before I start, I believe the
> document
> >authors are using the term IP in its most generic sense and not meant to
> imply a
> >particular transport such as TCP, UDP, or some other transport.  Drew/Paul
> --
> >feel free to correct me if I'm wrong.  There are certainly many
> possibilities
> >and I think they all need to be considered at this point.
> >
> >While it may be theoretically possible to implement TCP in silicon, it is
> not at
> >all clear if TCP in silicon will occur anytime soon -- if at all.  Betting
> on
> >that fact may not sit well with others, especially those concerned with
> higher
> >performance campus-based solutions that they want available now.  Nothing,
> >however, precludes us from taking advantage of TCP in silicon in a future
> >revision of the protocol(s).
> >
> >For WAN traffic, I think most people would agree with you on your remark
> on the
> >need for bandwidth management.  Further, I believe most people would
> probably
> >agree with you that if we were looking for a pure L2 solution (e.g.,
> directly
> >attached LAN), the IETF is *not* the right place to take this.  I
> definitely do.
> >
> >However, a customer may want to have a storage solution that can be
> accessed
> >over a campus network.  In this case, a L2 solution is not sufficient and
> IP is
> >required.  Further, it does not seem to be at all clear if TCP bandwidth
> >management is a requirement in this environment.  The NFS version 4
> requirements
> >document has some commentary on this discontinuity.  In a campus network,
> there
> >are other possible ways to do bandwidth management, including traffic
> >engineering, differential services, etc.
> >
> >In light of the NFS discussion, perhaps looking to NFS may not be a bad
> thing to
> >do?  NFS can operate over both TCP and UDP.  Perhaps this model is also a
> good
> >model for IP storage.
> >
> >Dave
> >
> >
> >
> >
> >Andreas Bechtolsheim <avb@cisco.com> on 02/03/2000 09:14:54 AM
> >
> >Sent by:  Andreas Bechtolsheim <avb@cisco.com>
> >
> >
> >To:   See Below
> >cc:   See Below
> >Subject:  Re: IP Storage Framework document
> >
> >
> >
> >While I will not be able to attend this meeting in person, I would
> >like to propose an alternative framework for Storage over IP.
> >We will discuss this in more detail at the 2/16 Meeting in San Jose.
> >
> >The key characteristics of our proposal are as follows:
> >
> >1. It is based on TCP/IP, so all the issues of a reliable
> >   end-to-end byte stream including error recovery and
> >   congestion management are addressed without inventing
> >   a new protocol to do the same functions.
> >
> >2. It works over any network topology and any distance,
> >   addressing the needs of both local and remote connectivity
> >   between servers and storage devices.
> >
> >3. The protocol can be implemented efficiently in hardware
> >   inside network interface controllers allowing for
> >   high-performance wirespeed network operation.
> >
> >While in the past several new protocols have been proposed to
> >accelerate certain types of data transfers, typically with
> >the argument that they are easier to implement in hardware,
> >experience has shown that such protocols have failed to gain
> >market acceptance. With the plummenting cost of silicon,
> >at this point in history the cost of implementing TCP/IP in
> >NIC silicon is in the order of a few $ worth of silicon
> >so there is simply no economic reason not to do it.
> >
> >In addition, proposing protocols that do not adhere to the
> >TCP/IP bandwidth management algorithms pose a serious risk
> >for overall network stability, in particular if they are used
> >for high bandwidth traffic such as disk I/O.
> >
> >Finally, designing new protocols for limited topologies such
> >as local area networks does not appear consistent with the
> >goals of the IETF.
> >
> >
> >
> >To: Dave Lee/Hq/3Com               Raz @Emc.Com
> >    Mark_Bradley @Btc.Adaptec.Com  Avb @Cisco.Com
> >    Carl Madison/Us/3Com           Rcampbel @Cisco.Com
> >    Arun Verma/Hq/3Com             Jgw @Cisco.Com
> >    Kquick @Iphase.Com             Ddecapit @Cisco.Com
> >    Bassoon @Ece.Cmu.Edu           Sob @Harvard.Edu
> >    Paulv @Corp.Adaptec.Com        Vern @Aciri.Org
> >    Dwilson @Corp.Adaptec.Com
> >
> >cc: Avb @Cisco.Com                 Csapuntz @Cisco.Com
> >    Cheriton @Cisco.Com
> >
> >
> 
>