2026-09-18T19:14:23Z https://www.ideals.illinois.edu/oai-pmh

oai:www.ideals.illinois.edu:2142/106434 2023-07-11 col_2142_5131 col_2142_8888 com_2142_5130 com_2142_8887 com_2142_234

Kim, Nam Sung Yuan, Yifan 2020-03-02T22:38:39Z 2020-03-02T22:38:39Z 2022-03-03T10:15:22Z 2019-10-17 2019-12 Distributed training of Deep Neural Networks (DNN) is an important technique to reduce the training time of large DNNs for a wide range of applications. In existing distributed training approaches, however, the communication time to periodically exchange parameters (i.e., weights) and gradients among computer nodes over the network constitutes a large fraction of the total training time. To reduce the communication time, we propose an algorithm/hardware co-design, INCEPTIONN. More specifically, observing that gradients are much more tolerant to precision loss than parameters, we first propose a gradient-centric distributed training algorithm. As designed to exchange only gradients among nodes in a distributed manner, it can transfer less information, better overlap communication with computation, and apply a more aggressive lossy compression algorithm to all the information exchanged among nodes than traditional distributed algorithms. Second, exploiting unique characteristics of gradient values, we propose a lossy compression algorithm, optimized for compressing gradients. It accomplishes high compression ratios for compressing gradients without notably affecting the accuracy of trained DNNs. Lastly, we demonstrate that compression algorithms consume a large amount of CPU time, which in turn increases total training time albeit reduced communication time. To tackle this, we propose an in-network computing approach that delegates the lossy compression task to hardware integrated with a Network Interface Card (NIC). Our experiments show that INCEPTIONN can reduce a large portion of the communication time and thus the training time of DNNs, with little degradation in accuracy of trained DNNs. Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2021-12-01 The student, Yifan Yuan, accepted the attached license on 2019-10-16 at 11:28. The student, Yifan Yuan, submitted this Thesis for approval on 2019-10-17 at 11:04. This Thesis was approved for publication on 2019-10-17 at 14:53. DSpace SAF Submission Ingestion Package generated from Vireo submission #14499 on 2020-02-28 at 17:35:46 Made available in DSpace on 2020-03-02T22:38:39Z (GMT). No. of bitstreams: 2 YUAN-THESIS-2019.pdf: 697509 bytes, checksum: 04c13b288846b999c63ebe53c4453fe7 (MD5) LICENSE.txt: 4207 bytes, checksum: 2f7056da77d43a35b3abbba998cb115c (MD5) Previous issue date: 2019-10-17 Embargo set by: Seth Robbins for item 113978 Lift date: 2022-03-02T22:39:04Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system Limited Restriction Lifted for Item 113978 on 2022-03-03T10:15:22Z. application/pdf http://hdl.handle.net/2142/106434 en Copyright 2019 Yifan Yuan distributed training accelerator Accelerating distributed neural network training with network-centric approach text text Electrical & Computer Eng Electrical & Computer Engr University of Illinois at Urbana-Champaign Thesis M.S.