Re: Performance of iSCSI, FCIP and iFCP

["Victor Firoiu" <vfiroiu@nortelnetworks.com>]

Somesh,

our answer is beween your lines below.

Victor

Somesh Gupta wrote:
> 
> Franco,
> 
> Thanks for getting us to exercise the under-utilized math part of
> our brains :-)
> 
> I wanted to echo the comments that Stephen made. What I think
> you are saying is that
> 
>  If a single packet drop occurs no more frequently than the time
>  to recover (the rate of the connection which existed before the
>  error occured), then a quarter of the bandwidth during the
>  recovery period is lost. In the simplistic case where an error
>  occurs with just the correct frequency, you loose a quarter of
>  the available bandwidth.
> 
>  If you have N connections sharing the same bandwidth, then the
>  time to recover is reduced to 1/N and since the rate on each
>  connection is (1/N), the lost rate is proportional to (1/N)^2.
> 
>  A minor quibble with the math. I assume (considering this low
>  rate) that the error is likely due to transmission errors which
>  is proportional to the number of bytes sent, the (1/n)^2 is not
>  quite true as the number of bytes sent will be larger so errors
>  will occor somewhat more frequently as a function of time.
> 
You are right.  That is why in our numerical example, the one-TCP case
has an effective rate of 8.27Gb/s out of 10Gb/s available and not
7.5Gb/s.

In general, the ratio between the amount of data losing transmission
opportunity per loss event in one-TCP case versus the N-TCP case is
proportional to N^2.

> The really interesting equation to me is to look at the lost
> oportunity in isolation of the number of connections (i.e.
> you can scale the numbers any which way)
> 
> D = C/8*I/4 = C^2*RTT^2/(256*M)
> 
> As long as RTTs are small, and/or data rate (C) is not too large,
> we don't loose too large a percentage of the available bandwidth.
> 
> However, if both become large which is what we face, and keep
> becoming larger, then we keep getting closer and closer to the
> 1/4 loss rate, unless we keep increasing the number of connections
> to compensate.
> 
> In a perfect world, you would be able to convince the IETF to
> use the product of rate and RTT as a factor to adjust the A and
> the M part in AIMD algorithm to achieve the same result. It
> seems a little bit of waste of energy to get around the math
> by using multiple connections (BTW - are there any published
> papers on experiences with multiple connections for a single
> data stream in a high performance environment - would love to
> learn from someone else's mistakes).
> 
> Somesh
> 

I agree that we can mitigate bandwidth loss through another congestion
control algorithm.  But we have no control whatsoever
of TCP, and thus we are committed to using whatever a vanilla TCP looks
like.  Even if super-TCP would be used, the FCIP storage gateways would
still need to be aware of the identity and number of storage
connections, and thus their processing overhead would be close to the
TCP-connection-for-storage-connection solution as in iFCP or iSCSI. 
Therefore, the solution with one TCP connection per storage connection
seems to be more appropriate than one TCP connection per PHY.


Victor