PDL Talk Series

August 6, 2026

TIME: 12:00 noon to 1:00 pm EDT
PLACE: Virtual - a zoom link will be emailed closer to the seminar

Stephen Mell, Canegie Mellon University

Opportunistic Evaluation for Compound AI Systems
AI systems have shown impressive performance across a wide range of tasks. Part of this has been driven by a shift toward compound AI systems, composed of one or more AI models and programmatic components. Latency is a key concern for these systems, and there are many opportunities to parallelize and stream model calls. However, manually doing so greatly increases code complexity, intertwining high-level algorithms with low-level performance optimizations. In this dissertation, we propose providing parallelization and streaming automatically for high-level, sequential code via an approach called opportunistic evaluation. First, we develop its theory, introducing a novel calculus, λO, providing sound, out-of-order execution of general-purpose programs that make external calls, such as those to AI models. We then describe a language, Opal, which offers automatic parallelization and streaming, and we demonstrate its ability to express and optimize a wide range of compound AI systems, improving performance by up to 12.7×. Next, we build a system, PopPy, which automatically parallelizes Python programs, and we show that it can express 5 compound AI systems from the literature with minimal modifications, yielding speedups of up to 6.4×. Finally, we build an extensible language, Quasar, for code generated by AI models and show that it can provide latency, security, and reliability improvements, without accuracy degradation. Together, these show how opportunistic evaluation can provide significant benefits to compound AI systems written in general-purpose programming languages.

It consists of the following papers:

• https://arxiv.org/abs/2405.11361
• https://arxiv.org/abs/2605.18697
• https://arxiv.org/abs/2506.12202
  

BIO: Stephen Mell recently joined CMU CSD as a postdoc, advised by Zhihao Jia. Previously, he completed his PhD at the University of Pennsylvania, advised by Osbert Bastani and Steve Zdancewic. His interests are broadly at the intersection of programming languages, machine learning, and systems.

CONTACTS

, PDL Co-Director
RMCIC 2208

, PDL Co-Director
(412) 268-3064
GHC 9109

Executive Director, Parallel Data Lab
(412) 268-5485

PDL Administrative Manager
(412) 268-6716