Re: FCIP link: Severe underutilization problem?

[Stephen Bailey <steph@cs.uchicago.edu>]

> Has anybody else confirmed these calculations?

I don't think they are correct.

When you are within the FC network, credits are link-layer.  That
means, each LINK must have enough credits to satisfy only its
bandwidth * delay product.  You will not get a 200 MS delay on a
single FC link.  You will get a speed-of-light in a fibre (or twinax)
+ some small overhead type delay.

When you are in TCP, YES, you need enough buffering to cover the
bandwidth * delay of the entire TCP connection.  If THAT has a 200 MS
RTT and a bandwidth of 10 Gbits/s, you need 200 MB of buffering
(analogous to FC credits) in the FCIP gateways to `guarantee' it works
at line rate under ideal circumstances.  The FCP HBAs do not need 200
MB worth of BB credits.  For the HBA's first hop FC link, a number
like Tachyon's 4 BB credits is more than enough.

> And the number-of-SCSI commans/R2Ts/Data in flow has to be also
> large to fill the pipe.

Julian makes the vastly more important point here.  This applies no
matter WHAT the link technology.

Even if you have flow control buffering to saturate your links at 10
Gb/s * 200 MS, most file systems don't generate nearly enough
concurrent demand (e.g. 200 MB of total outstanding transfers AT ALL
TIMES in this case), to maintain performance at this bandwidth *
delay.

You can sing the aggregation song in this case, though.  Each client
may only generate O(10 MB) of concurrent demand, but if you have 20
clients, you could fully use your 10 Gb/s * 200 MS.

Even though present hosts may not be tuned for the next bandwidth *
delay step, that's never been a reason not to take the step.  The
single-stream performance of FCP @ > 10 km was pretty lackluster
initially on many platforms.  Make sure you appropriately calibrate
your initial expecations on FCIP.

Steph