RE: switch latency (was Command Queue Depth, Asymmetric/Symmetric)

["Merhar, Milan" <mmerhar@pirus.com>]

Michael,

You're quite right about the actual crossbars introducing 
minimal delays - typically, only a few tens of bit times 
at 2-3x the port bit rate.

But, there's often a store-and-forward delay at the input port, 
where the first bits received sit until the last bits of the 
packet arrive (introducing one packet transmission time's delay,
or about 12.5 usec for a 1500 byte Gigabit Ethernet packet.)
This is also where packet inspection and classification
delays can appear, as you correctly point out.

There is a similar store-and-forward delay coming out of 
the crossbar, where the first bits switched may wait until 
the last bits clear the crossbar, before the actual 
packet transmission is scheduled to the outgoing port. 
As the crossbar may run at 2-3 times the input bit rate, 
this delay is less, often on the order of 4-6 usec for 
a maximum-size packet.

Added together, this gives you 16-20 usec "first bit in to first
bit out" delay for a maximum sized Gigabit Ethernet packet, 
which would be reported in the usual data-sheet convention
as about 5 usec "last bit in to first bit out."
Some of the big switches introduce a little more store and 
forward delay, as they use two levels of internal switching,
one at the module level, and another at the chassis level.
 
"Sub 1 usec" numbers may indicate the switch selectively 
uses cut-through forwarding rather than store-and-forward 
(tricky to get right, but certainly faster,) or may merely be 
the result of doing the measurements with a minimum-sized packet. 

As for delay numbers exceeding 1 millisecond, all I can suggest 
is that someone was looking at the specs for a 10 Megabit 
Ethernet product (where an 1518 byte packet takes 1.2 msec 
to transmit,) or for a software-based bridge or router, 
which can easily build up 1 msec of queuing time.

And, of course, when you start considering large WAN networks,
with multiple router hops and substantial speed-of-light delays, 
30-50 msec starts to look mighty fast.  Add in congestion queuing, 
and you can easily build up to the often quoted 150-300 msec 
"Internet latency."

- milan

-----Original Message-----
From: Michael Krause [mailto:krause@cup.hp.com]
Sent: Tuesday, September 12, 2000 11:21 AM
To: Scott Bradner
Cc: ips@ece.cmu.edu
Subject: Re: Command Queue Depth (was asymmetric/Symmetric)


At 07:14 AM 9/8/00 -0400, Scott Bradner wrote:
> > I don't know what the
> > latency through ethernet switches is, but I'd hope it wasn't in the ms 
> range.
>
>processing time is in the 10 to 50 microsecond range

Don't know whose switches you are evaluating.  They range from sub 1-usec 
to perhaps 5-10 usec - it is all a matter of how much additional processing 
is going on within the switch.  For example, many switches examine not just 
the ethernet headers/tags but also look at the IP headers and provide 
additional functionality based on their content.  This is where many switch 
vendors try to differentiate and is not a function of whether they can 
switch fast or not. The actual switching elements are usually crossbars 
that operate in the 100-500ns range - chip process improvements will lower 
that by quite a bit over the next couple of years.

Mike