Re: FCIP and wandering duplicates

[Vivek Kashyap <viv@sequent.com>]

Raj,

I am, after reading your and Doug's replies, still unclear whether FC handles 
duplicates or not. It appears to me that it does not.

E_D_TOV seems to be a time out that is set for retransmission between
end points. However, it is possible to introduce transient loops in
the fabric when a topology change (addtion of a new switch or a
link/switch going down) occurs, due to the re-calculation of paths
unless such changes are made atomic to the traffic. In such a case
there could be duplicates.

Of course, if the FC switches are quiesced during the switch updates
then loops won't matter. Or is there a different mechanism or an
algorithm that guarantees looplessnes?


TTL retuning will at most reduce the duplicates but can't guarantee
elimination of duplicates. So if FC somehow guarantees duplicate
free fabrics and does not handle duplicates then it appears to me that
FCIP always will have the danger of corrupting data due to duplicates
inserted by IP.


Vivek

> 
> Raj,
> 
> I do not understand what tuning TTL does to benefit any situation.  The path
> between locations is not fixed so such tuning may induce lost packets and is
> unrelated to time in transit.  Depending on the domain, a packet may be
> repeated by equipment that does not necessarily decrement TTL.  Although not
> a desired feature, duplication may happen if the packet transmit status is
> made uncertain by local signaling.
> 
> Doug
> 
> > -----Original Message-----
> > From: owner-ips@ece.cmu.edu [mailto:owner-ips@ece.cmu.edu]On Behalf Of
> > Raj Bhagwat
> > Sent: Monday, September 11, 2000 12:26 PM
> > To: Douglas Otis; Vivek Kashyap; ipfc@standards.gadzoox.com
> > Cc: ips@ece.cmu.edu
> > Subject: Re: FCIP and wandering duplicates
> >
> >
> > Doug,
> >
> > I understand how exactly TTL is used. That is why I stated that TTL should
> > be set to a value larger than the number of hops between the source and
> > destination. An IP end station may not know this and it may choose to set
> > the TTL to the max value of 255. However, an FCIP gateway, with some
> > intelligence, can perhaps know the # of hops to other FCIP
> > gateways it talks
> > to, especially in enterprise networks.
> >
> > It is not clear to me how there can be duplicate frames without the source
> > sending twice. Could you please explain that?
> >
> > Thanks,
> >
> > Raj.
> >
> > ----- Original Message -----
> > From: Douglas Otis <dotis@sanlight.net>
> > To: Raj Bhagwat <rajb@lightsand.com>; Vivek Kashyap <viv@sequent.com>;
> > <ipfc@standards.gadzoox.com>
> > Cc: <ips@ece.cmu.edu>
> > Sent: Monday, September 11, 2000 11:53 AM
> > Subject: RE: FCIP and wandering duplicates
> >
> >
> > > Raj,
> > >
> > > You can see duplications without the source sending twice.  Reducing TTL
> > > value may prevent the packet from ever arriving.  The number of
> > times TTL
> > is
> > > decremented depends on the path and not time in transit.
> > >
> > > Doug
> > >
> > > > -----Original Message-----
> > > > From: Raj Bhagwat [mailto:rajb@lightsand.com]
> > > > Sent: Monday, September 11, 2000 11:31 AM
> > > > To: Douglas Otis; Vivek Kashyap; ipfc@standards.gadzoox.com
> > > > Cc: ips@ece.cmu.edu
> > > > Subject: Re: FCIP and wandering duplicates
> > > >
> > > >
> > > > Doug,
> > > >
> > > > Your point on TTL is well taken. It is there to prevent a packet from
> > > > getting stuck in a routing loop. Duplicates of unicast packets can
> > happen
> > > > when the sender retransmits a packet and both the original and the
> > > > retransmitted packets reach the destination. If the sender does not
> > > > retransmit for an adequate amount of time, duplicates of
> > unicast packets
> > > > should not happen. This is true even when there are routing loops or
> > > > topology changes in the network.
> > > >
> > > > As you have pointed out, there is no direct correlation between
> > > > E_D_TOV and
> > > > TTL. And that could be a problem. However, setting the TTL to a
> > relatively
> > > > smaller value (larger than the # of hops between the source and
> > > > destination
> > > > FCIP gateways) can reduce the possibility of duplicates when
> > the sender
> > of
> > > > the FC frame waits for at least E_D_TOV before retransmitting
> > the frame.
> > > >
> > > > I don't think SCTP at the FCIP gateway will solve the duplicate
> > > > problem even
> > > > with a single retry. There is no synchronization between the FC device
> > and
> > > > the FCIP gateway for error recovery. So, the FC device can retransmit
> > > > independent of the SCTP at the FCIP gateway. One way to solve this
> > problem
> > > > is to fine tune the timeout values on the FC side and at the
> > SCTP level
> > on
> > > > the FCIP gateway.
> > > >
> > > > Regards,
> > > >
> > > > Raj Bhagwat
> > > > LightSand Communications
> > > >
> > > >
> > > > ----- Original Message -----
> > > > From: Douglas Otis <dotis@sanlight.net>
> > > > To: Raj Bhagwat <rajb@lightsand.com>; Vivek Kashyap <viv@sequent.com>;
> > > > <ipfc@standards.gadzoox.com>
> > > > Cc: <ips@ece.cmu.edu>
> > > > Sent: Monday, September 11, 2000 10:14 AM
> > > > Subject: RE: FCIP and wandering duplicates
> > > >
> > > >
> > > > > Raj,
> > > > >
> > > > > Unless you are willing to wrap the FC frame within some protocol,
> > > > duplicates
> > > > > can happen and this has nothing to do with TTL.  TTL
> > prevents circular
> > > > paths
> > > > > from forever sending a packet within a loop.  A modified
> > SCTP protocol
> > > > that
> > > > > will solve both of these problems, if restricted one time retry.  A
> > > > > Metro-Area-Network should induce about 5ms RTT delay such that
> > > > a 10ms skew
> > > > > becomes possible but is limited by the single retry.  TTL buys
> > > > you nothing
> > > > > with this E_D_TOV, Error_Detect Timeout.
> > > > >
> > > > > Doug
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: owner-ips@ece.cmu.edu
> > [mailto:owner-ips@ece.cmu.edu]On Behalf
> > Of
> > > > > > Raj Bhagwat
> > > > > > Sent: Sunday, September 10, 2000 10:57 PM
> > > > > > To: Vivek Kashyap; ipfc@standards.gadzoox.com
> > > > > > Cc: ips@ece.cmu.edu
> > > > > > Subject: Re: FCIP and wandering duplicates
> > > > > >
> > > > > >
> > > > > > Vivek,
> > > > > >
> > > > > > In a properly implemented FC-only fabric, duplicates should
> > > > not happen.
> > > > A
> > > > > > frame that enters a fabric should either be delivered
> > within E_D_TOV
> > > > > > (typically 2 sec) or be discarded. FC devices don't retransmit a
> > frame
> > > > > > before the E_D_TOV timer expires.
> > > > > >
> > > > > > You do bring up an interesting point when two SAN islands are
> > > > > > bridged across
> > > > > > an IP network. Once an FC frame enters the IP network, it is hard
> > > > > > to say how
> > > > > > the E_D_TOV is enforced by that part of the network. As authors of
> > the
> > > > > > FCoverIP draft, we have given some thought to the possibility of
> > > > > > delivering
> > > > > > duplicate frames when both the TCP stack at the FCIP
> > gateway and the
> > > > > > original FC device initiate recovery/retransmission
> > procedure in an
> > > > > > overlapping time period. This is one of the major reasons why we
> > > > initially
> > > > > > chose to make the FCIP gateway a stateless device with no
> > > > buffering and
> > > > no
> > > > > > retransmission.
> > > > > >
> > > > > > The good news is that the {IP encapsulated} FC frame will be
> > protected
> > > > by
> > > > > > the IP TTL once it enters the IP network. The bad news is that
> > > > > > detecting the
> > > > > > E_D_TOV timeout while the frame is inside the IP network may be
> > > > difficult.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Raj Bhagwat
> > > > > > LightSand Communications
> > > > > >
> > > > > >
> > > > > > ----- Original Message -----
> > > > > > From: Vivek Kashyap <viv@sequent.com>
> > > > > > To: <ipfc@standards.gadzoox.com>
> > > > > > Sent: Sunday, September 10, 2000 3:17 PM
> > > > > > Subject: FCIP and wandering duplicates
> > > > > >
> > > > > >
> > > > > > > I read the FCIP document recently. It does not address the issue
> > of
> > > > > > > duplicates.
> > > > > > >
> > > > > > > Does fibre channel protect itself from duplicates ? I looked at
> > the
> > > > > > > header and it does not have a time to live field. Thus
> > a duplicate
> > > > > > > caught in a loop and then released at the 'right' time could be
> > > > > > > accepted by the receiver. Even if the FC fabric path
> > algorithms do
> > > > > > > ensure loop free paths transient loops due to errors, switch
> > > > > > > misconfigurations or failures are possible. Does FC handle
> > > > such a case
> > > > ?
> > > > > > >
> > > > > > > If it does not then FCIP will exacerbate the problem because the
> > IP
> > > > > > > network could have 'transient' loops. Such a 'wandering
> > duplicate'
> > > > > > > could be re-injected into the FC island and be accepted
> > by a newer
> > > > > > > incarnation of a FC inter-island exchange. Furthermore with
> > > > no TTL one
> > > > > > > can't really provide a timout between starting two
> > conversations.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Vivek
> > > > > > > --
> > > > > > > Vivek Kashyap
> > > > > > > IBM-NumaQ
> > > > > > > viv@sequent.com
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> >
> 


-- 
Vivek Kashyap
Advisory Engineer
IBM-NumaQ
viv@sequent.com
vivk@us.ibm.com
503 578 3422 (o)