Fine-grained error analysis in machine translation

Dell, Brennan

Permalink

https://hdl.handle.net/2142/129863 Copy

Description

Title

Fine-grained error analysis in machine translation

Author(s)

Dell, Brennan

Issue Date

2025-07-16

Director of Research (if dissertation) or Advisor (if thesis)

Dunn, Jonathan

Doctoral Committee Chair(s)

Dunn, Jonathan

Committee Member(s)

• Yoon, James
• Maskharashvili, Aleksandre
• Tang, Yan

Department of Study

Linguistics

Discipline

Linguistics

Degree Granting Institution

University of Illinois Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

Machine Translation, Computational Linguistics, Turkish, Finnish, Evidentiality

Abstract

This dissertation focuses on the fine-grained evaluation method for machine translation. We define fine-grained tests as methods which evaluate translation quality with respect to a particular phenomenon. These methods contrast with one-dimensional evaluation methods, which quantify translation quality as a single number. The BLEU score, and more recent neural translation metrics like XCOMET, are examples of such metrics. In this work we demonstrate, from our own experiments and from a review of related work, that fine-grained evaluation methods are more useful tools for translation evaluation. We argue that this is the case because translation quality is inherently multidimensional. A correct translation must not only convey the meaning of the source language, but must also be grammatical in the target language. These two factors themselves are complex phenomena, with many orthogonal components. Fine-grained tests are thus well-suited to the fundamentally multidimensional nature of translation quality. In our first experiment, we demonstrate that fine-grained tests are uniquely useful for understanding how the training neural machine translation system unfolds. Additionally, through a preliminary experiment targeting the translation of the Turkish evidentiality morpheme, we observe translation system behaviors that suggest they may not have acquired evidentiality. These findings, which we were not able to explain with linguistic or sentence complexity features, motivate further research on this phenomenon. We present this work as a review of related work on fine-grained error analysis in machine translation, supplemented with our own findings, analysis, and recommendations. We do this to aid the future development of such tests, which we believe will be crucial for navigating our current era which is increasingly dominated by large language models.

Graduation Semester

2025-08

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/129863

Copyright and License Information

Copyright 2025 Brennan Dell

Owning Collections