2026-09-19T01:05:49Z https://www.ideals.illinois.edu/oai-pmh

oai:www.ideals.illinois.edu:2142/110569 2023-07-11 col_2142_5131 col_2142_10761 com_2142_5130 com_2142_10755 com_2142_234

Zhai, Chengxiang Cheng, Xiang 2021-09-17T01:11:17Z 2021-09-17T01:11:17Z 2021-04-27 2021-05 Structure prediction (SP) tasks are important in natural language understanding in the sense that they provide complex and structured knowledge of the text. Recently, some unified text-to-text transformer models like T5 and TANL have produced competitive results on SP tasks. These models convert SP tasks into a seq2seq problem, where a transformer is used to generate sequences with special tokens representing the extracted spans, labels, and relationships. Compared to many popular Natural Language Understanding models that are designed specifically for the task, the output of the text-to-text transformer is more flexible. With proper format, it could be trained on multiple tasks together and take advantage of the shared knowledge between tasks. To better understand how these models achieve better performance by multi-task learning, we designed several experiments to measure the knowledge transfer ability of a recently proposed model, TANL. In our experiments, we found that the multi-head attention in the decoder can capture the relationship between tasks which leads to performance improvement. Another finding is that TANL may produce many outputs with invalid format when trained from scratch, and starting from a T5 pre-trained model helps to mitigate this problem. Based on these observations and some new intuitions, we proposed an improved version of TANL called SDCT5 (step decomposed and constrained text-to-text Transformer). Preliminary experiment results show that our model can achieve better performance on SP tasks compared to TANL and benefit more from multi-task learning. Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2021-09-16 without embargo terms The student, Xiang Cheng, accepted the attached license on 2021-04-24 at 10:19. The student, Xiang Cheng, submitted this Thesis for approval on 2021-04-24 at 10:28. This Thesis was approved for publication on 2021-04-27 at 09:34. DSpace SAF Submission Ingestion Package generated from Vireo submission #16544 on 2021-09-16 at 16:47:31 Made available in DSpace on 2021-09-17T01:11:17Z (GMT). No. of bitstreams: 2 CHENG-THESIS-2021.pdf: 1242117 bytes, checksum: c62d59918b73c76eb5fc07f0b10311e2 (MD5) LICENSE.txt: 4208 bytes, checksum: d01d4fe5f48c513e9a750014751422ec (MD5) Previous issue date: 2021-04-27 application/pdf http://hdl.handle.net/2142/110569 en Copyright 2021 Xiang Cheng natural language processing structure prediction multi-task learning A deeper look into multi-task learning ability of unified text-to-text transformer text Thesis Computer Science Computer Science University of Illinois at Urbana-Champaign Thesis M.S.