RE: iFCP vs FCIP

[Joshua Tseng <jtseng@NishanSystems.com>]

Venkat,

> 
> Josh,
> 
> It seems FC switching and networking technology is driven 
> primarily by the
> needs of back-end storage subsystems. If we look at design 
> choices that FC
> has adopted, it is very clear that they focus on smaller networks (you
> typically don't expect thousands of storage subsystems 
> connected into a
> common shared SAN with cascade of switches, behind your hosts 
> even in a
> large data center), with very low media loss rates (frame 
> corruption rates
> of 10^-16 is typical), smaller latencies (10us to 100us). Given the
> evolution from a bus architecture (SCSI), the number of 
> targets discovered,
> the in-order delivery of data frames, low latency across the 
> network are
> "assumed" by applications. Whereas it is natural to set up an 
> asynchronous
> request on a TCP socket, most applications assume low latency 
> and issue a
> synchronous request to a storage subsystem. Block-level access based
> applications have not assumed the same modes of failures as a 
> file-level
> access based applications have.

I agree that FC switching and networking technology is
driven by the need for smaller, high-performance networks
for server storage subsystems.  There will always be a need
for these smaller networks--I don't dispute that at all.  However,
with growing storage demand in many enterprise applications,
many storage networks need to scale beyond levels which Fibre
Channel can provide today.  So the relevant question is should an
implementer for such a network wait for FC networking technology
mature in its scalability, or can Fibre Channel be mapped to TCP/IP
and leverage the scalability that IP already provides?  I believe
that in many cases, the latter is the more desirable approach, and
that is why iFCP was created.

On the topic of local high-performance networks, note that there is
nothing that prevents gigabit ethernet from performing as well
or better than Fibre Channel.

> 
> An example is out-of-order delivery of exchanges. Creating an 
> FC mesh with
> FSPF (as described in FC-SW-2) can cause out-of-order delivery of FCP
> exchanges. End points are supposed to deal with out-of-order 
> delivery, but
> very few SCSI drivers and target mode devices appear to deal 
> with this. So,
> fabric vendors are forced to guarantee in-order delivery; one 
> way they do
> this is to ensure that FSPF routes remain "fixed" and a route 
> change can
> only occur after an R_A_TOV time elapses without any I/Os on the ISL.

Because iFCP uses TCP, in-order delivery is guaranteed by iFCP.
If you are thinking about mFCP/UDP, most OSPF implementations will
not chose different paths for traffic between the same two IP 
addresses.  So the only case you will get alternate route selection
is in a route-failure situation, in which case out-of-order is
not your main concern (packet loss is).

> 
> Another example is approach to flow control and congestion 
> avoidance. Flow
> control is addressed using buffer-to-buffer credits, but congestion
> avoidance in ISLs is something not really addressed. The 
> solution is to use
> Class-4 service and some form of VC_RDY to avoid this. 
> Overprovisioning ISLs
> is something that is rarely done. Once again, block storage 
> being at the
> bottom most layer in the information hierarchy of an 
> organization, requires
> robustness and predictability in performance. FC has targeted 
> these issues
> and addressed them well. It is fairly easy to achieve upwards 
> of 90% link
> utilization on an FC-AL, but much harder to do in an Ethernet segment.

Yes, optimal usage of available bandwidth is an issue for any
application built upon TCP, including iFCP, FCIP, and iSCSI.  But link
utilization is a function of the transport protocol (e.g., UDP and TCP),
not the layer-2 link technology.  Thus, it IS possible to obtain
90%+ utilization over gigabit ethernet by using UDP.

> 
> Extending storage over a wide area either with iFCP or FCIP 
> or FC-BB-2 needs
> consideration of how applications are to deal with these problems and
> issues. Without block-level applications adapting to 
> wide-area networking
> (loss rates of 10^-4, latencies of 100ms), it becomes harder 
> to take care of
> these issues in transport layer. Although iFCP and FCIP are 
> both workable
> technologies, I'm not sure if a mere address translation, or 
> setting up a
> tunnel is adequate. Again, what guarantees can we provide 
> about in-order
> delivery at the FCP Portals of iFCP?  Maybe block-level access in host
> drivers and kernel block-level I/O will take care of these 
> over time. But
> until then, FC products and fabrics will continue to have its 
> role. In that
> respect, FC over ATM (FC-BB-2) appears to be  closer to the 
> original design
> goals of FC, compared to iFCP. BTW, I assume that iFCP handles FC-AL
> correctly, by implementing an FL_Port equivalent.

I agree that iFCP and FCIP have some common issues with regard to WAN
networking which might be resolved in a similar ways.  However, note that
in the iFCP case, the iFCP gateway is a visible entity to Fibre Channel
devices.  Thus, the iFCP gateway does have the extra benefit of being
able to interact with Fibre Channel devices, such as to manipulate FC
frame sizes (to avoid fragmentation), and B-B credits (to coordinate TCP &
B-B flow control).  The FCIP approach is much simpler--just pass everything
through the tunnel.

Josh

> 
> Venkat Rangan
> Rhapsody Networks Inc.
> http://www.rhapsodynetworks.com
> 
> 
> -----Original Message-----
> From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> Joshua Tseng
> Sent: Wednesday, November 29, 2000 10:55 AM
> To: Murali Rajagopal; Ips (E-mail)
> Subject: RE: iFCP vs FCIP
> 
> 
> Murali,
> 
> >
> > With my TC hat off:
> >
> > Charles observation that FCIP's goal to maintain transparency
> > within the
> > switching FC Fabric is correct as far data transport is
> > concerned. However,
> > there is a clearly defined architecture defined in FC-SW-2
> > standards that
> > allow a device such as FCIP to connect to a border switch. In
> > other words,
> > from a routing standpoint the FC fabric is certainly aware of
> > a hierarchial
> > network and is supported jointly by the FSPF routing 
> protocol and the
> > FSPF-backbone routing protocols. This OSPF-based hierarchial
> > model provides
> > a lot of flexibility to the nature of the FC backbone 
> networks. TCP/IP
> > happens to be one of the many possabilities. (Other
> > possabilities include FC
> > directly over ATM and SONET as defined in the ANSI T11 FC-BB
> > standards)
> 
> Is the possibility of tunneling FC over ATM and SONET deemed to be
> an advantage for Fibre Channel networks?  TCP/IP has been networking
> over these transports for many years.  It seems if I can map Fibre
> Channel to IP (as iFCP does), then I can leverage all of the physical
> network infrastructure that TCP/IP currently runs on.
> 
> >
> > The second plus of this model is that it allows any type of
> > traffic and
> > allows for a very simple almost stateless (from FC
> > point-of-view) behavior.
> > This directly translates to scalability. The comment made by
> > someone in this
> > thread about FCIP being limited is inaccurate- it is in fact
> > the opposite.
> 
> I fail to understand how this changes or improves the scalability
> of FC. FC has a hard limit of 239 switches, period.  FCIP doesn't
> change this.
> 
> >
> > Finally, Joshua's comment on the small number of switches in
> > a FC SAN is an
> > observation from the past and this is rapidly changing as
> > evidenced by the
> > growing size of SANs in Data Centers.
> 
> I have worked on stable OSPF networks comprising of more than
> 800 routers and switches.  When interconnected with EGP routing
> protocols, IP is virtually unlimited in scale.  I do not think
> Fibre Channel and FSPF can ever approach this scalability.  There
> are stability problems right now with FSPF implementations.  Perhaps
> in time, they will be patched up, but why wait for FC networking
> protocols to be debugged when there are many stable and tested OSPF
> implementions out there?
> 
> Josh
> 
> >
> > -Murali Rajagopal
> > LightSand Communications
> >
> >
> > -----Original Message-----
> > From: owner-ips@ece.cmu.edu 
> [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> > Charles Monia
> > Sent: Tuesday, November 28, 2000 7:19 PM
> > To: Ips (E-mail)
> > Cc: David Robinson (E-mail)
> > Subject: RE: iFCP vs FCIP
> >
> >
> > Hi Folks:
> >
> > The issue is that the design goals and underlying network models are
> > fundamentally different. Essentially, FCIP's goal is to provide a
> > transparent conduit between Fibre Channel fabrics while
> > iFCP's goal is ULP
> > transparency between N_PORTs.
> >
> > As a result, in iFCP, the fabric-wide services provided by FC fabric
> > elements (and often implemented with proprietary protocols)
> > are replaced by
> > standard, IP-based equivalents. For that reason, an iFCP
> > gateway does not
> > need to recognize or provide facilities for servicing 
> inter-switch FC
> > protocols, such as those for zoning, naming and routing.
> >
> > Charles
> > > -----Original Message-----
> > > From: Joshua Tseng [mailto:jtseng@NishanSystems.com]
> > > Sent: Tuesday, November 28, 2000 4:08 PM
> > > To: David Robinson; Ips (E-mail)
> > > Subject: RE: iFCP vs FCIP
> > >
> > >
> > > Hi David,
> > >
> > > >
> > > > I am no FCP expert so please correct me if I am wrong. In a pure
> > > > FCP world, there is end-to-end traffic and there is
> > traffic that is
> > > > destined to go between AS's. The primary difference is 
> that there
> > > > is an explicit route to the border gateways in the latter
> > > > case. In both
> > > > the proposals, within the FCP realm the addresses are FCP
> > > based until
> > > > they hit an edge node.  In iFCP the destination is
> > > converted to an IP
> > > > address that represents the end node address (which may
> > actually be
> > > > a gateway back into FCP on the other side), in FCIP the 
> request is
> > > > routed to the other AS's FC border gateway and this request is
> > > > encapsulated
> > > > in a TCP request. Given that we are moving between AS's (I
> > > > believe that
> > > > is an assumption in FCIP) can we not use iFCP and instead of
> > > > specifying
> > > > the IP address of the end node, specify the IP address of the
> > > > other AS's
> > > > border gateway since FCP should already be doing some
> > encapsulation
> > > > to route between AS's?
> > > >
> > > > 	-David
> > >
> > > Up until recently with the creation of the DMP routing 
> protocol, the
> > > concept of AS's (Autonomous Systems, right?) was foreign to Fibre
> > > Channel networking.  Most Fibre Channel networks are
> > comprised of just
> > > a handful of switches--the largest FC network I have ever heard of
> > > being deployed is a 15 switch fabric.  Perhaps somewhere there are
> > > some fabrics which are bigger, but probably not by much.
> > > (Architecturally, a single Fibre Channel fabric has a
> > maximum capacity
> > > of 239 switches)
> > >
> > > FCIP does not do anything to improve the scalability limits of
> > > the Fibre Channel fabric.  All it does is allow extension of the
> > > FC fabric over distances using an IP network.  The FCIP gateway is
> > > completely invisible and non-intrusive to the Fibre 
> Channel switches
> > > and does not change or improve the scalability or interoperability
> > > limits of FC fabrics.
> > >
> > > On the other hand, an iFCP gateway actively participates in
> > > switching and routing traffic between FC fabrics and FC 
> devices, by
> > > mapping FC addresses to IP addresses and routing them 
> using standard
> > > IP routing protocols.  Using iFCP, a storage network has the same
> > > scalability limits as any other IP network (e.g., IPv4
> > address space,
> > > etc...).
> > >
> > > Josh
> > >
> >
>