Enhancing hybrid cloud ray clusters: automated data management and innovative networking for efficient HPC cloud bursting
Zhu, Zhongbo
Loading…
Permalink
https://hdl.handle.net/2142/124427
Description
Title
Enhancing hybrid cloud ray clusters: automated data management and innovative networking for efficient HPC cloud bursting
Author(s)
Zhu, Zhongbo
Issue Date
2024-04-30
Director of Research (if dissertation) or Advisor (if thesis)
Kindratenko, Volodymyr
Department of Study
Electrical & Computer Eng
Discipline
Electrical & Computer Engr
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
M.S.
Degree Level
Thesis
Keyword(s)
Distributed Systems
Cloud Computing
High Performance Computing
Distributed Machine Learning
Language
eng
Abstract
The increasing reliance on hybrid systems that combine High-Performance Computing (HPC) and Cloud computing reflects their ability to manage workload surges and reduce users' queuing time. This thesis presents the development of an innovative HPC-Cloud bursting system, which leverages the Ray open-source distributed framework. Our system features an advanced automated data management mechanism and a unique dynamic label-based scheduling algorithm that intelligently manages data dependencies and minimizes data transmission overheads. My personal contributions to this project were critical in several areas: I designed and implemented network solutions and infrastructure as code, significantly simplifying the deployment and operational management of the hybrid system while also reducing costs. Additionally, I played an important role in optimizing the system's scheduler to improve stability and contributed directly to a performance boost. Our benchmarks, which focus on machine learning model training and image processing tasks, demonstrate our system's enhanced efficiency, achieving up to 20\% improvement when compared to traditional methods.
Use this login method if you
don't
have an
@illinois.edu
email address.
(Oops, I do have one)
IDEALS migrated to a new platform on June 23, 2022. If you created
your account prior to this date, you will have to reset your password
using the forgot-password link below.
