Parallel file system with tunable consistency

Wang, Chen

Permalink

https://hdl.handle.net/2142/116191 Copy

Description

Title

Parallel file system with tunable consistency

Author(s)

Wang, Chen

Issue Date

2022-07-08

Director of Research (if dissertation) or Advisor (if thesis)

• Snir, Marc
• Mohror, Kathryn

Doctoral Committee Chair(s)

Snir, Marc

Committee Member(s)

• Huang, Jian
• Winslett, Marianne
• Gropp, William

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Parallel file system
• Consistency model
• Tunable consistency
• Parallel Storage
• I/O Tracing
• MPI Tracing

Abstract

In high performance computing (HPC) systems, the I/O demands of applications are supported by parallel file systems such as Lustre and GPFS. Most general-purpose parallel file systems (PFS) support the POSIX I/O interface and its consistency model. However, the POSIX standard was defined decades ago for use by a single machine with a single storage device. It is not fit for highly concurrent applications typically seen on modern HPC systems. The major impediment to the PFS performance is the strict adherence to POSIX consistency semantics, which requires sequential consistency in general and atomicity for many operations. The strict enforcement of these requirements impedes caching, generates significant additional traffic, and results in congestion in situations of high sharing, especially for small block reads and writes. The use of the POSIX consistency model for I/O has plagued the HPC community for many years, but it is becoming more problematic due to two key reasons: (1) the rapid increase in the scale of HPC systems; (2) the emergence of the new storage techniques such as persistent memory. They make the overhead of maintaining POSIX consistency relatively higher. Many efforts have been made toward PFSs with relaxed consistency semantics. However, different applications have different consistency requirements. A PFS providing a static consistency model will not work ideally for every application. Moreover, the correctness is not guaranteed if the provided consistency model is weaker than the desired one. In this dissertation, we first propose a multi-level I/O and MPI tracing tool. We then collect detailed traces from 17 representative HPC applications and I/O benchmarks. Next, we employ a trace-driven analysis approach to study the consistency requirements of these applications. And finally, based on this study, we propose and design a parallel file system that provides tunable consistency models.

Graduation Semester

2022-08

Type of Resource

Thesis

Copyright and License Information

Copyright 2022 Chen Wang

Owning Collections