2026-09-20T01:50:37Z https://www.ideals.illinois.edu/oai-pmh

oai:www.ideals.illinois.edu:2142/124357 2024-09-16 col_2142_5131 col_2142_10761 com_2142_5130 com_2142_10755 com_2142_234

Hoiem, Derek 2024-05 application/pdf en text Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms The student, Joshua Levine, accepted the attached license on 2024-04-22 at 09:10. The student, Joshua Levine, submitted this Thesis for approval on 2024-04-22 at 09:26. This Thesis was approved for publication on 2024-04-29 at 09:06. DSpace SAF Submission Ingestion Package generated from Vireo submission #20529 on 2024-09-16 at 00:35:42 Methods for generating visual programs with optimizable vision models Levine, Joshua 2024-04-29 Visual Programming Visual Question Answering Large Language Models Computer Vision Program Generation End-to-end vision-language models often fail to handle compositional tasks, necessitating alternative approaches for more complex problem-solving. Leveraging the visual programming paradigm, we propose a novel method for composing foundational vision models through program generation to tackle compositional tasks effectively. We investigate prompting and execution strategies that enable the synthesis of fine-tunable code by trainable large language models aimed at improving the effectiveness of the programs in solving vision-language tasks. Capitalizing on the robust compositional reasoning capabilities of large language models (LLMs), we employ pre-trained LLMs to architect programs constructed using a catalog of pre-defined atomic functions. These atomic functions, implemented with pre-trained vision models, serve as the building blocks for the visual programs generated by our system. Our methodology supports programs in various formats, always offering the flexibility to fine-tune the constituent vision models and the LLM code generator. This study concentrates on image-based question-answering. This focus underscores the critical need for advanced compositional reasoning in interpreting and responding to complex visual queries. Our evaluation encompasses the executability and correctness of the produced programs, providing a comprehensive assessment of our approach's effectiveness. This paper lays the groundwork for a subsequent investigation into the joint training of the LLMs and atomic functions, setting the stage for significant advancements in program generation and compositional reasoning in computer vision. Text eng https://hdl.handle.net/2142/124357 Copyright 2024 Joshua Levine M.S. Thesis Computer Science University of Illinois at Urbana-Champaign Computer Science