PARALLEL DATA LAB 

PDL Abstract

OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving

arXiv:2609.14237v1 [cs.DC] 13 Sep 2026.

Zikun Li, Yixuan Mei, Shiqi Pan, Zixuan Chen, Xiaowen Zhang, Mengdi Wu, Shuhuai Lin, Yutong Yang, Zhihao Zhang, Xupeng Miao†, Rashmi Vinayak‡, Zhihao Jia

Carnegie Mellon University
† Peking University
‡ CMU and Google

http://www.pdl.cmu.edu/

LLM serving systems increasingly disaggregate inference into finer-grained stages, with recent approaches separating attention from FFN or MoE execution during decode. This operator-level disaggregated serving (ODS) can improve hardware matching and enable independent scaling, particularly across heterogeneous devices. However, existing systems fix operator boundaries and lack a unified characterization of when disaggregation reduces serving cost. We present Op- Weave, an end-to-end framework for heterogeneous ODS. OpWeave provides an analytical cost model that bounds the gains of homogeneous and heterogeneous ODS over colocated serving. It jointly optimizes operator partitioning and deployment configuration through a regularity-aware planner that keeps the search tractable even for hybrid-attention models. A vLLM-based runtime executes the synthesized plans with flexible operator stages across heterogeneous device groups. In our evaluation, OpWeave reduces serving cost by up to 1.78× on homogeneous and 1.89× on heterogeneous GPU clusters relative to the best feasible baseline, while meeting latency SLOs.

FULL TR: pdf