PDL Abstract Runtime Estimation and Resource Allocation for Concurrency Testing Carnegie Mellon Univsersity Parallel Data Laboratory Technical Report CMU-PDL-12-115, November 2012. Chuck Cranor, Milo Polte, Garth A. Gibson Parallel Data Laboratory Carnegie Mellon University Pittsburgh, PA 15213 http://www.pdl.cmu.edu/ In this report we describe how we adapted the Parallel Log Structured Filesystem (PLFS) to enable HPC applications to be able read and write data from the HDFS cloud storage subsystem. Our enhanced version of PLFS provides HPC applications with the ability to concurrently write from multiple compute nodes into a single file stored in HDFS, thus allowing HPC applications to checkpoint. Our results show that HDFS combined with our PLFS HDFS I/O Store module is able to handle a concurrent write checkpoint workload generated by a benchmark with good performance. KEYWORDS: HPC Computing, Cloud Storage, PLFS, HDFS, PVFS FULL PAPER: pdf Parallel Data Laboratory

PARALLEL DATA LAB 

PDL Publications

PDL Abstract

Runtime Estimation and Resource Allocation for Concurrency Testing

Carnegie Mellon Univsersity Parallel Data Laboratory Technical Report CMU-PDL-12-115, November 2012.

Chuck Cranor, Milo Polte, Garth A. Gibson

Parallel Data Laboratory
Carnegie Mellon University
Pittsburgh, PA 15213

http://www.pdl.cmu.edu/

In this report we describe how we adapted the Parallel Log Structured Filesystem (PLFS) to enable HPC applications to be able read and write data from the HDFS cloud storage subsystem. Our enhanced version of PLFS provides HPC applications with the ability to concurrently write from multiple compute nodes into a single file stored in HDFS, thus allowing HPC applications to checkpoint. Our results show that HDFS combined with our PLFS HDFS I/O Store module is able to handle a concurrent write checkpoint workload generated by a benchmark with good performance.

KEYWORDS: HPC Computing, Cloud Storage, PLFS, HDFS, PVFS

FULL PAPER: pdf