2026-09-19T09:02:47Z https://www.ideals.illinois.edu/oai-pmh

oai:www.ideals.illinois.edu:2142/117774 2023-07-11 col_2142_5131 col_2142_8888 com_2142_5130 com_2142_8887 com_2142_234

Kim, Nam Sung Kim, Nam Sung Hwu, Wen-mei Torrellas, Josep Chen, Deming 2022-12 application/pdf en text Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-04-12 without embargo terms The student, Youjie Li, accepted the attached license on 2022-11-22 at 20:06. The student, Youjie Li, submitted this Dissertation for approval on 2022-11-22 at 21:09. This Dissertation was approved for publication on 2022-11-23 at 13:35. DSpace SAF Submission Ingestion Package generated from Vireo submission #18624 on 2023-04-12 at 07:31:54 Communication-centric cross-stack acceleration for distributed machine learning Li, Youjie 2022-11-23 Distributed Machine Learning Distributed Training Parallel Computing In-network Computing Smart Nics Programmable Switches Pipelined Sgd Machine Learning Framework Distributed training has been the Holy Grail in machine learning systems, as it is an indispensable technique for addressing the ever-growing computation demands and memory requirements due to the unprecedented scaling of model sizes and data volumes. However, even distributed training takes inordinate time, of which a large fraction is paid for communication overhead in either inter-server networks or intra-server interconnects. In this dissertation, we propose cross-stack solutions that span hardware, software, and algorithms to accelerate and scale distributed training systems. First, we propose in-network computing by leveraging novel network hardware in modern data-centers, such as programmable network interface cards and switches, for not only compressing traffic volume in real time but also reducing network hops on the fly. Second, we present new algorithms surrounding efficient communication, such as gradient compression and pipelined computation with communication, for further shrinking the network overhead while maintaining the training convergence and model accuracy. Third, we develop a next-generation machine learning framework by novel schemes of task decomposition and late binding, to train massive models even without sufficient memory while drastically reducing communication overhead within a server's interconnects. Thesis eng https://hdl.handle.net/2142/117774 In reference to IEEE copyrighted material which is used with permission in this thesis, the IEEE does not endorse any of [university/educational entity's name goes here]'s products or services. Internal or personal use of this material is permitted. If interested in reprinting/republishing IEEE copyrighted material for advertising or promotional purposes or for creating new collective works for resale or redistribution, please go to http://www.ieee.org/publications_standards/publications/rights/rights_link.html to learn how to obtain a License from RightsLink. Ph.D. Dissertation Electrical & Computer Engr University of Illinois at Urbana-Champaign Electrical & Computer Eng