Re: IETF mailing list question on Storage over Ethernet/IP

[narakamath@lightel.com]

Thank you all for some excellent discussions on the subject.  For all the reasons mentioned, I am mapping TCP/IP to DWDM to create very high performance SANs.  I hope to share more on this as we progress and launch products.  You can see more on this at www.lightel.com.

On Fri, 26 May 2000, Dave Nagle wrote:

> 
> 
> A few comments about this one.
> 
> 1. FC does not provide reliable transmission.  It provides for error
> detection, but escalates recovery to "upper level protocol".  FCP-2 has
> improved this situation, but is not widely implemented yet.  One of the
> advantages of using a transport such as TCP is that link errors will be
> corrected in a manner that is transparent to the application protocol
> (SCSI).
> 
> 2. Jumbo frames will not be necessary when TCP is implemented in hardware.
> Most FC implementations use 1024 byte frames, and performance is very
> adequate, given hardware implementation of FCP.
> 
> 3. The cost of using different transport protocols in the LAN and WAN is
> that the two will not interoperate.  Many of us believe that TCP has proven
> itself in both the LAN and WAN.  I bet your PC or UN*X workstation is using
> TCP for all its protocol needs.
> 
> 4. The IPS working group is mapping SCSI to TCP.  Another working group is
> mapping FC to IP.  These are very different approaches.  The first (ours)
> preserves SCSI, but does not include any vestige of Fibre Channel.  It is
> intended for use in the LAN, MAN and WAN.  Its best use is for connecting
> hosts computers to storage controllers using Ethernet and IP WAN technology.
> It will be possible, but non-trivial, to translate between SCSI over TCP/IP
> and SCSI over Fibrechannel.  The second is a tunneling scheme for extending
> Fibre Channel over the IP WAN.  It does not contemplate Ethernet-based hosts
> or storage controllers.
> 
> 5. Just about any reliable transport will do nicely for transporting SCSI
> commands.  We chose TCP because its implementation and behavior are
> well-known, and it is well-supported with load-balancing, QoS and security
> features.  While another protocol (such as reliable datagram) might be
> arguably better suited to storage transport applications, we'll use TCP
> "because it's there".  We'll have the benefit of all the other investment
> that's going into improving TCP for internet uses.
> 
> Randy Haagens
> Networked Storage Architecture
> Storage Organization
> Hewlett-Packard Co.
> e-mail: Randy_Haagens@hp.com
> tel: +1 916 785 4578
> fax: +1 916 785 1911
> 
> 
> > -----Original Message-----
> > From: Dave Nagle [mailto:bassoon@yogi.ece.cmu.edu]
> > Sent: Thursday, May 25, 2000 4:29 PM
> > To: SCSI-over-TCP List
> > Subject: IETF mailing list question on Storage over Ethernet/IP 
> > 
> > 
> > 
> > ------- Forwarded Message
> > 
> > Date:    Thu, 25 May 2000 19:27:02 -0400
> > From:    Dave Nagle <bassoon@yogi.ece.cmu.edu>
> > To:      "Jon William Toigo" <jtoigo@IntNet.net>
> > cc:      ietf@ietf.org
> > Subject: Re: Storage over Ethernet/IP 
> > 
> > Jon,
> > 
> > Original Message
> > - ----------------
> >  >> I am seeking a few points of clarification:
> >  >> 
> >  >> 1.  Fibre Channel folks have attempted to explain to me 
> > why TCP/IP could =
> >  >> NEVER be a viable interconnect for block level storage 
> > operations.  They =
> >  >> claim:
> >  >> a.  TCP is too CPU intensive and creates too much latency 
> > for storage =
> >  >> I/O operations.
> >  >> b.  The IP stack is too top heavy and processing packet 
> > headers is too =
> >  >> slow to support storage I/O operations.
> > 
> >   There is a lot of work to show that this is not true.  Check out Van
> > Meter's 1998 ASPLOS paper "VISA - Netstations virtual internet SCSI
> > adaptor."
> > 
> >  Perhaps more importantly, there are many companies that are building
> > TCP in silicon ASICs.  This should make TCP's performance comparable
> > to Fibre Channel.  Both TCP/IP and FC provide about the same
> > functionality ... reliable, in-order transmission.  
> > 
> > The bottom line is that FC is done in hardware while TCP has
> > traditionally been done in software. Therefore, previous performance
> > numbers are not going to be fair.  Once TCP is in silicon, its
> > performance should be roughly equal to FC.
> > 
> >  >> c.  The maximum throughput of a GE TCP/IP connection is 
> > 768 Mps, which =
> >  >> is too slow to support storage I/O operations.
> > 
> >  I believe there are higher numbers (especially with Jumbo
> > Frames). Alteon's web site show's 920 Mbps.  Microsoft and Duke
> > University have both shown TCP performance o 1Gb+/s performance over
> > other networks.
> > 
> >   BTW, why is 768 Mbps too slow for storage.  Many apps (e.g.,
> > transaction workloads) are I/O's per second bound, not bandwidth
> > bound.  Also, even if storage over IP/ether is a bit slower than FC,
> > the benefits of leveraging IP's infrastructure (i.e., routers,
> > switches, NICs, network management, networking people) is a huge
> > advantage.  
> > 
> >  There is also the issue of SCSI over TCP/IP in the SAN vs. the
> > LAN/WAN.  Some companies, focusing on the SAN, are building
> > SCSI/lightweight transport/IP while others, focusing on the WAN,
> > propose SCSI/TCP/IP.  It may be the case that SAN and WAN traffic use
> > different transport protocols to gain a bit of extra performance in
> > the SAN.  
> > 
> >  >> Is any of this true?
> >  >> 
> >  >> 2.  Adaptec has posited a replacement for TCP called STP 
> > for use as a =
> >  >> transport for storage.  Does anyone know anything about this?
> > 
> >     From Paul von Stamwitz's posting to the ips mailing list ...
> >    
> >       The link to the SEP draft is
> >       http://www.ietf.org/internet-drafts/draft-wilson-sep-00.txt
> >    
> >       The press release is at:
> >     http://www.adaptec.com/adaptec/press/release000504.html
> >    
> >     The demo shows a Gb ethernet controller transporting SCSI 
> > traffic to several
> >     targets through an off-the-shelf 100TX switch with a Gb  
> > uplink. The targets
> >     are ethernet to U160 SCSI bridges with one or more SCSI  
> > drives attached. The
> >     host controller runs under NT4.0 at appears to the OS as 
> > a  SCSI host bus
> >     adapter.
> >    
> >     The architecture is based on Adaptec's SCSI Encapsulation Protocol
> >     (SEP).  SEP is mapped on top of TCP/IP or a light-weight transport
> >     protocol specifically designed for SANs.
> >     
> >     An SEP overview was presented at the IPS BOF in Adelaide 
> > last  month and an
> >     internet draft on SEP was submitted to IETF this week. I 
> > will  forward the
> >     link as soon as it becomes available. This draft is informational
> >     only and intended to aid in this group's work toward an industry
> >     standard SCSI transport protocol over IP networks.
> > 
> > 
> >  >> 3.  Current discussions of the SCSI over IP protocol seem 
> > to ignore the =
> >  >> issue of TCP or any other transport protocol.  Does anyone know =
> >  >> definitively what transport is being suggested by the 
> > IBM/Cisco crowd?
> > 
> >    Current SCSI over IP discussions are not ignoring TCP ... they are
> >    definitely considering TCP as the primary transport.  See the ips
> >    web site at:
> >  
>      http://www.ece.cmu.edu/~ips
> 
>  >> 
>  >> 4.  Another storage company is looking at Reliable UDP as a substitute =
>  >> for TCP in storage data transfers.  Where can I learn more about this =
>  >> protocol, which I am told was introduced many years ago by Cisco?
> 
>   Companies to look at include:
> 
>      nishansystems.com
>      interprophet.com
>      san.com
>      arkresearch.com
> 
>   Also, I believe that the IETF IP over FC working group is now
> looking at FC over IP.
> 
> 
> 
> dave...........
> 
> David Nagle
> Director, Parallel Data Lab
> Senior Reseach Computer Scientist
> School of Computer Science
> Carnegie Mellon University
> Pittsburgh, PA 15213
> 412-268-3898 (office)
> 412-268-3890 (fax)
> http://www.ece.cmu.edu/~bassoon
> 
> 
> - ------- End of Forwarded Message