TCP speed

[Scott Bradner <sob@harvard.edu>]

the topic of TCP speeds came up - here is a note from Craig Partridge
(someone who should know) on the topic.

Scott

From craig@aland.bbn.com Sun Sep 10 15:04:54 2000
To: Scott Bradner <sob@harvard.edu>
Subject: Re: tcp speed 
In-reply-to: Your message of "Sun, 10 Sep 2000 09:15:25 EDT."
             <200009101315.JAA10411@newdev.harvard.edu> 
Date: Sun, 10 Sep 2000 13:26:59 -0400
From: Craig Partridge <craig@aland.bbn.com>


[Scott -- feel free to recirculate as appropriate.]

This note is a brief summary of what is achievable in TCP performance.

First off, let's talk about processing.  Not including interrupt costs (more
on those in a minute), it costs about 20 or 30 instructions to handle a TCP
segment that is received in order, plus time for copying and checksumming.
Copying and checksumming can be done at memory speed, and for large packets
20 or 30 instruction times is trivial, so effectively TCP can transfer data
at memory rate.  For more on how to achieve these rates, see Van Jacobson's
work on TCP Header Prediction, my book on Gigabit Networking, and the
performance notes in Rich Stevens' TCP-IP Illustrated.

Three things, however, slow TCP down.

The first problem is interrupts.  Most fast CPUs these days really really
don't like interrupts -- they may burn 1000s of instruction cycle to handle
them.  So having a network interface that minimizes interrupts (e.g., like
the ones Jonathan Smith pioneered in the mid-90s) is critical or you'll spend
all your time in the interrupt driver.

The second issue is TCP window size.  If you don't use TCP Large Windows,
and actually set a large window size, you're limited to what is typically
shipped with your system.  In most systems the standard window size is
about 16 to 32 KB, which limits you to about 250 Mb/s on a typical LAN.
(Moral here: tune up your TCP).

The third issue is TCP congestion control.  TCP takes a while to get to send
at full speed, as it is probing the network to determine available network
capacity.  To date, there's no agreed upon solution to allow TCP (or any
other protocol for that matter) to send fast from the start.  However, this
time delay is only a problem over long distances -- you won't see problems
of note on your LAN.

In summary, in a local environment, with TCP big windows, you should go
as fast as your processors let you.  In a wide area environment, you'll
bump against congestion control issues, but then again so will any other
protocol.  As a final comment, it is worth noting that some commercial
TCPs were running at 1 Gbps nearly a decade ago.

Craig