PDL Talk Series

July 30, 2026

TIME: 12:00 noon to 1:00 pm EDT
PLACE: Virtual - a zoom link will be emailed closer to the seminar

Davis Haupt, Jane Street

Supercharging Metadata for Jane Street's Data Warehouse
Superstore is Jane Street's internal data warehouse. It currently stores over 500PB of data used for everything from developing new trading strategies to analyzing build times in our continuous integration pipeline. Superstore ingests hundreds of terabytes of data per day, and this talk will discuss the tradeoffs that we've made to build a system that scales out so effectively. Superstore is eventually consistent, but relies on a strongly consistent metadata layer to keep track of the history of writes along with the current listings of data files for each dataset, which today takes up 20TB of storage. Since launch, we've used a transactional, relational database for this metadata, but today, metadata transaction processing is one of our largest bottlenecks. We'll spend the rest of the talk diving deeper into our solutions here.

Our new metadata system brings together some interesting internal technologies with open source offerings. Powering our new metadata layer is Aria, a low-latency persistent message bus originally designed for exchange-like infrastructure at Jane Street which is now used for application development throughout the company. Superstore uses Aria as a leader election mechanism and replication log to provide durable and consistent updates to RocksDB key/value stores. We've taken the consistency and transaction semantics that we need and stripped out the ones we can live without to build a system with less coordination overhead and dramatically lower latencies.

BIO: Davis Haupt is a Software Engineer at Jane Street based in New York City. He's spent most of his time there working on Superstore, focusing on LSM compaction and selective read performance over very large datasets. He has a B.S.E in Computer Science from the University of Pennsylvania. Previously he worked at MongoDB on query processing and static analysis, including some applications of Structured Encryption.

CONTACTS

, PDL Co-Director
RMCIC 2208

, PDL Co-Director
(412) 268-3064
GHC 9109

Executive Director, Parallel Data Lab
(412) 268-5485

PDL Administrative Manager
(412) 268-6716