University of Illinois Urbana-Champaign

Beyond words: Understanding emotional shifts in maternal vocalizations through speech emotion recognition models

Tin, Alara

This item is only available for download by members of the University of Illinois community. Students, faculty, and staff at the U of I may log in with your NetID and password to view the item. If you are trying to access an Illinois-restricted dissertation or thesis, you can request a copy through your library's Inter-Library Loan office or purchase a copy directly from ProQuest.

Permalink

https://hdl.handle.net/2142/129636

Description

Title
Beyond words: Understanding emotional shifts in maternal vocalizations through speech emotion recognition models
Author(s)
Tin, Alara
Issue Date
2025-05-07
Director of Research (if dissertation) or Advisor (if thesis)
Hasegawa-Johnson, Mark
Department of Study
Electrical & Computer Eng
Discipline
Electrical & Computer Engr
Degree Granting Institution
University of Illinois Urbana-Champaign
Degree Name
M.S.
Degree Level
Thesis
Keyword(s)
  • speech emotion recognition
  • CNN-BiLSTM
  • acoustic features
  • domain adaptation
  • mother-infant interaction
Abstract
Automatic Speech Emotion Recognition (SER) has significant potential to provide insights into our understanding of dyadic communications. This study focuses on maternal vocalizations within mother-infant dyadic interactions, examining how mothers’ happy and neutral emotional tones shift in response to varying infant stress levels. To achieve this, we employ multiple models: our hybrid CNN-BiLSTM architecture, alongside pre-trained transformer-based models such as wav2vec 2.0 and HuBERT. Our evaluation demonstrates that the hybrid model outperforms these transformer-based approaches after fine-tuning, achieving a minimum improvement of 3.94 percentage points in the test accuracy and 11 percentage points in the weighted average of F1 scores in the IDP dataset. Using our fine-tuned model, we analyze maternal vocalizations in different age groups of infants (3, 6, and 9 months) and classify infants into low-, mid-, and high-stress categories based on the Root Mean Square (RMS) energy features of their vocalizations during stress-inducing events. Our findings reveal a moderate effect size (Cohen’s d) of associations between high stress levels and pronounced vocalization changes in mothers of 3-month-olds, more nuanced responses in mothers of 9-month-olds, and a balanced distribution of vocalization shifts in mothers of 6-month-olds. The novel application of SER in mother-infant studies underscores emotional adaptation in maternal vocalizations and its potential to expand analyses to bidirectional influences, providing deeper insights into emotional communication dynamics.
Graduation Semester
2025-05
Type of Resource
Thesis
Handle URL
https://hdl.handle.net/2142/129636
Copyright and License Information
Copyright 2025 Alara Tin

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois