A data-augmented approach to evaluating teamwork using log data from digital collaboration tools

Shi, Wenxuan

Permalink

https://hdl.handle.net/2142/130030 Copy

Description

Title

A data-augmented approach to evaluating teamwork using log data from digital collaboration tools

Author(s)

Shi, Wenxuan

Issue Date

2025-07-10

Director of Research (if dissertation) or Advisor (if thesis)

Bailey, Brian

Doctoral Committee Chair(s)

Bailey, Brian

Committee Member(s)

• Sundaram, Hari
• Mercier, Emma
• Sterman, Sarah
• Wang, Dakuo

Department of Study

Siebel School Comp & Data Sci

Discipline

Computer Science

Degree Granting Institution

University of Illinois Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• human-computer interaction
• collaboration
• teamwork
• team assessment
• peer evaluations
• log data
• data-driven assessment
• interface design

Abstract

Teamwork is a critical skill in both educational and professional settings, yet evaluating individual contributions to collaborative work remains a persistent challenge. Instructors often rely on peer evaluations to assess teamwork, but these evaluations are prone to social and cognitive biases, including memory limitations, inconsistent standards, and interpersonal pressures. At the same time, the increasing use of digital collaboration tools such as Google Docs, GitHub, and Slack presents a new opportunity: these tools generate rich log data that can reveal how teams interact, coordinate, and contribute over time. This dissertation presents the design and evaluation of a data-augmented peer evaluation system that extracts, analyzes, and visualizes log data from digital collaboration tools to support students' evaluations of teamwork. By reviewing the visualization prior to rating their teammates, students can ground their evaluations in concrete evidence of each team member's behaviors. My approach centers students as active participants in interpreting and annotating their own data, thus also contributing knowledge of how students understand and value contributions within teams. This system is the culmination of a multi-stage investigation into how peer evaluations of teamwork can be augmented with log data from digital collaboration tools. I begin with a mixed-methods study examining current and potential uses of log data to support students' and instructors' evaluation practices. Through interviews and surveys, I find that while both groups already refer to log data to validate or challenge peer evaluations, this usage is often informal, retrospective, and shaped by uncertainty about how to interpret the data. Next, I conduct a between-subjects experiment to test whether reviewing log data during the peer evaluation process improves evaluation quality. The results show that log data increased consistency and perceived accuracy of the evaluations, motivating the design of a system that incorporates log data into the peer evaluation process. Grounded in the empirical findings of these two studies, I design and implement the data-augmented peer evaluation system through an iterative user-centered design process. The system captures multiple components of the teamwork: content, technical, and communication contributions in Google Drive, GitHub, and Slack. In the interface, the student reflects on an interactive timeline visualization of the team’s log data prior to evaluating their teammates, allowing them to ground their evaluations in the data. The student can also contextualize the data by marking contributions as valuable and tagging behaviors displayed by their teammates. I evaluate this system in a semester-long field study in a user interface design course. Results show that the data-augmented peer evaluation system helped students recall specific contributions, reflect on work patterns, and justify their ratings. At the same time, students expressed concerns about the representativeness of the data, with some even demonstrating resistance towards the system. Together, these contributions demonstrate the value of integrating log data into peer evaluations and advance knowledge of how data can be used not just to monitor teams, but to support fair and meaningful assessments of teamwork that can guide learning. This work also contributes design principles for creating data-driven evaluation systems that balance fairness, student agency, and meaningful reflection. This dissertation progresses toward a future in which every person working in a team can compose and receive meaningful feedback that helps them to feel valued for their contributions and learn effective teamwork behaviors.

Graduation Semester

2025-08

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/130030

Copyright and License Information

Copyright 2025 Wenxuan Shi

Owning Collections