SC ’21, November 14–19, 2021, St. Louis, MO, USA. Supercedes Carnegie Mellon University Parallel Data Lab Technical Report CMU-PDL-21-101, July 2021.
Qing Zheng, Chuck Cranor, Greg Ganger, Garth Gibson, George Amvrosiadis, Brad Settlemyer†, Gary Grider†
Carnagie Mellon University
† Los Alamos National Laboratory
High-Performance Computing (HPC) is known for its use of massive concurrency. But it can be challenging for a parallel filesystem’s control plane to utilize cores when every client process must globally synchronize and serialize its metadata mutations with those of other clients. We present DeltaFS, a new paradigm for distributed filesystem metadata.
DeltaFS allows jobs to self-commit their namespace changes to logs, avoiding the cost of global synchronization. Followup jobs selectively merge logs produced by previous jobs as needed, a principle we term No Ground Truth which allows for efficient data sharing. By avoiding unnecessary synchronization of metadata operations, DeltaFS improves metadata operation throughput up to 98× leveraging parallelism on the nodes where job processes run. This speedup grows as job size increases. DeltaFS enables efficient inter-job communication, reducing overall workflow runtime by significantly improving client metadata operation latency up to 49× and resource usage up to 52×.
KEYWORDS: Distributed filesystem metadata, massively-parallel computing, data storage
CONFERENCE PAPER: pdf
FULL TR: pdf