Design and implementation of learning-based storage systems: a holistic approach

Sun, Jinghan

Permalink

https://hdl.handle.net/2142/129928 Copy

Description

Title

Design and implementation of learning-based storage systems: a holistic approach

Author(s)

Sun, Jinghan

Issue Date

2025-07-13

Director of Research (if dissertation) or Advisor (if thesis)

• Huang, Jian
• Snir, Marc

Doctoral Committee Chair(s)

• Huang, Jian
• Snir, Marc

Committee Member(s)

• Gropp, William
• Patel, Sanjay
• Swift, Michael
• Chang, Jichuan

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Storage Systems
• Machine Learning For Systems
• Hardware Software Co-design
• Solid-state Drive
• Cloud Storage

Language

eng

Abstract

Storage systems have evolved over decades into a complicated ecosystem that includes storage hardware, system software, storage infrastructure, and data applications. This growing complexity poses significant challenges for storage development and deployment. Traditional human-driven, heuristic-based approaches to building storage systems cannot rapidly meet the ever-increasing demands for storage performance and efficiency. As we embrace the advancement of machine learning (ML), it is the golden age today to invent new approaches to building storage systems. This dissertation focuses on the design and implementation of learning-based storage systems across the entire storage stack, which develops learning-based approaches to optimize storage performance, resource efficiency, and management. The storage performance of traditional hardware and software cannot keep up with the increasing demands of applications. For example, in solid-state drives (SSDs), the flash translation layer (FTL) manages performance-critical metadata structures using human-driven heuristics, but it fails to adapt to different workload patterns and results in severe performance loss. To address this performance challenge, this dissertation proposes the first learning-based flash translation layer, LeaFTL, that can dynamically capture data access patterns of storage workloads at runtime. It significantly reduces the memory footprint of the address mapping table by grouping a large set of mapping entries into a learned segment. The saved memory space further benefits data caching and improves the overall storage performance. Storage resource efficiency is also critical as it directly affects the operational costs of storage infrastructure. For instance, cloud platforms manage storage resources at scale and it is beneficial for them to achieve high storage utilization. However, our study reveals that storage resources in modern cloud platforms are severely underutilized. To address this resource efficiency challenge, this dissertation proposes a learning-based storage harvesting framework named BlockFlex, that can dynamically harvest idle storage capacity and bandwidth to improve cloud storage utilization. BlockFlex leverages a lightweight online learning approach to predict resource utilization and storage resource demands, based on which it enables accurate storage resource harvesting. Storage management is becoming complicated with the rapid development of the software and hardware over the past decades. This is especially true for cloud platforms, where the cloud storage resource is shared by multiple tenants with complex storage states and dynamic workload characteristics. Cloud platform enforces isolation mechanisms as it manages collocated tenants, but weak isolation incurs high performance interference and strong isolation causes low storage utilization. This dissertation explores reinforcement learning (RL) techniques to combat this fundamental tussle, leveraging its unique advantages of optimizing decisions in the complex and dynamic environment. It employs multi-agent reinforcement learning to dynamically manage resource scheduling across multiple tenants. Our experiments show that it can achieve both high storage utilization and performance isolation.

Graduation Semester

2025-08

Type of Resource

Text

Handle URL

https://hdl.handle.net/2142/129928

Copyright and License Information

Copyright 2025 Jinghan Sun

Owning Collections