Effects of transfer learning and fine-tuning in domain-specific tasks: A case study of Word2Vec models in materials science

Wang, Xin; Su, Yanqing; Xu, Shuozhi; Lu, Kun

Permalink

https://hdl.handle.net/2142/126236 Copy

Description

Title

Effects of transfer learning and fine-tuning in domain-specific tasks: A case study of Word2Vec models in materials science

Author(s)

• Wang, Xin
• Su, Yanqing
• Xu, Shuozhi
• Lu, Kun

Issue Date

2025-03-11

Keyword(s)

• Fine-tuning
• Transfer learning
• Domain-specific tasks

Abstract

Fine-tuning and transfer learning are widely used to help models adapt to specific domain tasks and new domains with limited training data. However, their effectiveness in particular tasks within specific domains is not always clear. This study examines the effects of fine-tuning and transfer learning by using “creep” as a representative property and “gas turbine” as a representative application to search relevant materials. Three Word2Vec models are compared: Pei et al.’s model trained on 6.4 million abstracts in materials science (Model 1), a model trained on abstracts and titles from 1 million articles in a specific subarea in materials science: “metallic materials for extreme environments” (Model 2), and a fine-tuned version of Model 1 using Model 2’s corpus (Model 3). The models’ comparison reveals the effects of fine-tuning (Model 1 vs. Model 3) and transfer learning (Model 2 vs. Model 3). Model 3 outperforms Model 2 in identifying materials with the “creep.” This shows that transfer learning from general materials science literature improves performance. However, for the “gas turbine,” Models 2 and 3 achieve 96% accuracy, surpassing Model 1, indicating that fine-tuning improves performance while transfer learning does not. Principal Component Analysis (PCA) confirmed Model 2’s specialized understanding of extreme materials and Model 3’s benefits from transfer learning. The PCA visualization indicates Model 3 shows greater semantic differentiation, while Model 2 displays more semantic similarity. These initial findings revealed the effects of fine-tuning and transfer learning in domain-specific tasks. Future work will expand the scope to fully evaluate both techniques.

Publisher

iSchools

Series/Report Name or Number

iConference 2025 Proceedings

Type of Resource

Other

Genre of Resource

Conference Poster

Language

eng

Handle URL

https://hdl.handle.net/2142/126236

Copyright and License Information

Copyright 2025 is held by Xin Wang, Yanqing Su, Shuozhi Xu, and Kun Lu. Copyright permissions, when appropriate, must be obtained directly from the authors.

Owning Collections