17th USENIX Symposium on Operating Systems Design and Implementation. July 10–12, 2023. Boston, MA.
Hugo Sadok, Nirav Atre, Zhipeng Zhao*, Daniel S. Berger*^, James C. Hoe, Aurojit Panda+, Justine Sherry, Ren Wang†
Carnegie Mellon University
* Microsoft
^ Univesrity of Washington
+ New York University
† Intel
Today, most communication between the NIC and software involves exchanging fixed-size packet buffers. This packetized interface was designed for an era when NICs implemented few offloads and software implemented the logic for translating between application data and packets. However, both NICs and networked software have evolved: modern NICs implement hardware offloads, e.g., TSO, LRO, and serialization offloads that can more efficiently translate between application data and packets. Furthermore, modern software increasingly batches network I/O to reduce overheads. These changes have led to a mismatch between the packetized interface, which assumes that the NIC and software exchange fixed-size buffers, and the features provided by modern NICs and used by modern software. This incongruence between interface and data adds software complexity and I/O overheads, which in turn limits communication performance.
This paper proposes Enso̅, a new streaming NIC-to-software interface designed to better support how NICs and software interact today. At its core, Enso̅ eschews fixed-size buffers, and instead structures communication as a stream that can be used to send arbitrary data sizes. We show that this change reduces software overheads, reduces PCIe bandwidth requirements, and leads to fewer cache misses. These improvements allow an Enso̅-based NIC to saturate a 100 Gbps link with minimum-sized packets (forwarding at 148.8 Mpps) using a single core, improve throughput for high-performance network applications by 1.5– 6×, and reduce latency by up to 43%.
FULL PAPER: pdf