Large-scale data-analytics applications and database services are moving to the public cloud to benefit from its high elasticity and ease of resource management. Unfortunately, it is not always clear how to optimally design a cloud storage system given a particular set of application requirements. Cloud providers offer a large variety of storage services with different performance guarantees. But various applications demand different Service-Level Objectives (SLOs), such as throughput and latency, that require a unique combination of these services. Since users lack a deep understanding of these trade-offs, they typically choose instance and storage types sub-optimally, under-provisioning cloud resources, which can cause disruptions in operations, or over-provisioning, which results in higher costs.
To address this problem, we have develop Mimir, a system that helps users to make optimal decisions when composing distributed storage systems in the public cloud. Mimir lets users enter a set of SLOs and outputs the most cost-efficient cloud resources configuration that minimizes the overall cost paid by the user. While Mimir relies on measurements or specifications of cloud resource performance to solve this optimization problem, it is often the case that the effective application performance of the resulting system may differ, e.g., due to interference or resource contention. To resolve this, Mimir monitors SLO metrics while running the storage system software, and proposes adjustments as needed.
While working on building the Mimir system, we are also looking for exciting ideas of using public cloud resources in cost-efficient ways. For example, aggressively exploiting burstable storage service in public clouds, it is possible to increase storage throughput by up to 100x at a cost increase of only 10-40%. This new way of exploiting burstable storage service exposes opportunities for additional exploration of storage application designs on public clouds.
FACULTY
George Amvrosiadis
Greg Ganger
GRAD STUDENTS
We thank the members and companies of the PDL Consortium: Amazon, Bloomberg, Datadog, Google, Honda, Intel Corporation, IBM, Jane Street, Meta, Microsoft Research, Oracle Corporation, Pure Storage, Salesforce, Samsung Semiconductor Inc., Two Sigma, and Western Digital for their interest, insights, feedback, and support.