2026-09-22T12:45:11Z https://www.ideals.illinois.edu/oai-pmh

oai:www.ideals.illinois.edu:2142/108055 2023-07-11 col_2142_5131 col_2142_8888 com_2142_5130 com_2142_8887 com_2142_234

machine translation A translation framework for discovering word-like units from visual scenes and spoken descriptions text Thesis Hasegawa-Johnson, Mark A Wang, Liming 2020-08-26T21:58:07Z 2020-08-26T21:58:07Z 2020-05-14 2020-05 In the absence of dictionaries, translators, or grammars, it is still possible to learn some of the words of a new language by listening to spoken descriptions of images. If several images, each containing a particular visually salient object, each co-occur with a particular sequence of speech sounds, we can infer that those speech sounds are a word whose definition is the visible object. A multimodal word discovery system accepts, as input, a database of spoken descriptions of images (or a set of corresponding phone transcriptions) and learns a mapping from waveform segments (or phone strings) to their associated image concepts. In this thesis, we propose a novel framework for multimodal word discovery systems based on statistical machine translation (SMT) and neural machine translation (NMT). We extend the existing theoretical frameworks on unsupervised word discovery and demonstrate a class of effective models for end-to-end word discovery from image regions and spoken descriptions. Finally, we provide a careful ablation study on components of my system and present some of the challenges in multimodal spoken word discovery. Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2020-08-25 without embargo terms The student, Liming Wang, accepted the attached license on 2020-05-13 at 10:26. The student, Liming Wang, submitted this Thesis for approval on 2020-05-13 at 10:28. This Thesis was approved for publication on 2020-05-14 at 09:24. DSpace SAF Submission Ingestion Package generated from Vireo submission #15375 on 2020-08-25 at 17:14:33 Made available in DSpace on 2020-08-26T21:58:07Z (GMT). No. of bitstreams: 2 WANG-THESIS-2020.pdf: 1955236 bytes, checksum: d466468eb8c76cf7cef7185d035e1cb1 (MD5) LICENSE.txt: 4208 bytes, checksum: 404bb44630b880d96166085d5cde918c (MD5) Previous issue date: 2020-05-14 application/pdf http://hdl.handle.net/2142/108055 en Copyright 2020 Liming Wang multimodal learning low-resource speech technology Electrical & Computer Eng Electrical & Computer Engr University of Illinois at Urbana-Champaign Thesis M.S.