2026-09-21T05:04:09Z https://www.ideals.illinois.edu/oai-pmh

oai:www.ideals.illinois.edu:2142/105916 2023-07-11 col_2142_10761 col_2142_5131 com_2142_10755 com_2142_234 com_2142_5130

Snir, Marc Snir, Marc Gropp, William Hwu, Wen-mei Van Essen, Brian Schwing, Alexander Dryden, Nikoli Joseph 2019-11-26T20:59:36Z 2019-11-26T20:59:36Z 2021-11-27T10:15:23Z 2019-07-09 2019-08 Accelerating and scaling the training of deep neural networks (DNNs) is critical to keep up with growing datasets, reduce training times, and enable training on memory-constrained problems where parallelism is necessary. In this thesis, I present a set of techniques that can leverage large high-performance computing systems for fast training of DNNs. I first introduce a suite of algorithms to exploit additional parallelism in convolutional layers when training, expanding beyond the standard sample-wise data-parallel approach to include spatial parallelism and channel and filter parallelism. Next, I present optimizations to communication frameworks to reduce communication overheads at large scales. Finally, I discuss communication quantization, which can directly reduce communication volumes. In concert, these methods allow rapid training and enable training on problems that were previously infeasible. Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2021-08-01 The student, Nikoli Dryden, accepted the attached license on 2019-07-09 at 14:44. The student, Nikoli Dryden, submitted this Dissertation for approval on 2019-07-09 at 14:44. This Dissertation was approved for publication on 2019-07-09 at 15:16. DSpace SAF Submission Ingestion Package generated from Vireo submission #14222 on 2019-11-26 at 14:01:58 Made available in DSpace on 2019-11-26T20:59:36Z (GMT). No. of bitstreams: 2 DRYDEN-DISSERTATION-2019.pdf: 1517701 bytes, checksum: e62a499063edb4ed5333cccdef479754 (MD5) LICENSE.txt: 4210 bytes, checksum: 62bebd64df1dccf769c6f4d6b266b4b5 (MD5) Previous issue date: 2019-07-09 Embargo set by: Seth Robbins for item 113063 Lift date: 2021-11-26T20:59:54Z Reason: Author requested closed access (OA after 2yrs) in Vireo ETD system Limited Restriction Lifted for Item 113063 on 2021-11-27T10:15:23Z. application/pdf http://hdl.handle.net/2142/105916 en Copyright 2019 Nikoli Joseph Dryden High-performance computing deep learning convolutional neural network parallel computing machine learning Large-scale training of deep neural networks text text Computer Science Computer Science University of Illinois at Urbana-Champaign Dissertation Ph.D.