Chasing the “tail at scale”: toward cloud-native architectures

Stojkovic, Jovan

Permalink

https://hdl.handle.net/2142/130086 Copy

Description

Title

Chasing the “tail at scale”: toward cloud-native architectures

Author(s)

Stojkovic, Jovan

Issue Date

2025-07-08

Director of Research (if dissertation) or Advisor (if thesis)

Torrellas, Josep

Doctoral Committee Chair(s)

Torrellas, Josep

Committee Member(s)

• Xu, Tianyin
• Marinov, Darko
• Huang, Jian
• Franke, Hubertus
• Delimitrou, Christina
• Skarlatos, Dimitrios
• Hughes, Christopher J

Department of Study

Siebel School Comp & Data Sci

Discipline

Computer Science

Degree Granting Institution

University of Illinois Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• cloud computing
• serverless computing
• microservices
• computer architecture

Abstract

Cloud computing is undergoing a radical transformation with the emergence of lightweight cloud-native computing paradigms, such as microservices and serverless computing. Users build their applications by combining services, benefiting from a simplified programming model and fine-grained billing. At the same time, providers consolidate many services into a smaller number of servers, improving the utilization of their infrastructure. However, the detailed characterization of cloud-native environments presented in this thesis shows that these workloads differ significantly from traditional monolithic applications. They execute services that run for short times, exhibit bursty invocation patterns, and have frequent I/O operations that cause context switches. In addition to their core logic, services also execute many auxiliary operations known as datacenter tax, such as data serialization and encryption. Finally, services have stringent tail latency bounds, requiring the slowest requests to complete within a strict deadline. These characteristics result in significant inefficiencies in performance, energy, and resource utilization when cloud-native workloads run on conventional servers with conventional software stacks, negating the paradigm’s potential benefits. The goal of this thesis is to design hardware platforms and software stacks that enable the execution of cloud-native workloads with orders of magnitude better efficiency. The first part of the thesis designs a new hardware stack for cloud-native services. It introduces µManycore, a CPU architecture that minimizes the tail latency of cloud-native services. The thesis then extends the architecture with HardHarvest to boost utilization via hardware-based core harvesting, and refines the microarchitecture with Mosaic for better performance under frequent context switches. Finally, this thesis integrates on-package accelerators into the architecture and proposes AccelFlow, a framework that enables fine-grained, low-overhead orchestration of accelerators to reduce the datacenter tax in cloud-native environments. To maximize the efficiency of the proposed hardware architecture, the second part of the thesis builds a full software stack that is tightly co-designed with the hardware. It begins with MXFaaS, a mechanism that improves resource utilization by efficiently multiplexing resources during bursts of same-function invocations. Then, it integrates the novel Concord distributed caching system for FaaS environments, and uses SpecFaaS to accelerate end-toend application workflows through speculative service execution. Finally, this thesis improves the energy efficiency of cloud-native environments with two frameworks: EcoFaaS, which uses fine-grained scheduling and dynamic frequency scaling, and SmartOClock, which underprovisions resources and selectively overclocks cores during load spikes.

Graduation Semester

2025-08

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/130086

Copyright and License Information

Copyright 2025 Jovan Stojkovic

Owning Collections