Designing a conversational agent for promoting participation of under-contributing members in a small group chat discussion

Do, Hyo Jin

Permalink

https://hdl.handle.net/2142/120494 Copy

Description

Title

Designing a conversational agent for promoting participation of under-contributing members in a small group chat discussion

Author(s)

Do, Hyo Jin

Issue Date

2023-03-07

Director of Research (if dissertation) or Advisor (if thesis)

Bailey, Brian P

Doctoral Committee Chair(s)

• Bailey, Brian P
• Karahalios, Karrie

Committee Member(s)

• Chandrasekharan, Eshwar
• Sung, Chul

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Group Discussion
• Collaborative Task
• Conversational Agent
• Participation Balance

Abstract

Balanced participation is one of the crucial aspects of teamwork that leads to high satisfaction and performance. However, low participating behavior of one or a few members is common in group discussions including online group chat discussions. This dissertation addresses this challenge of low participating behavior by building a conversational agent (CA) that plays the role of a facilitator in group chat discussions and promotes participation of under-contributing members. I present the results from three experiments that aim to advance the design of the CA by using effective communication strategies, repairing imperfect interventions, and helping users to make appropriate expectations of the CA performance. In the first experiment, I tested five communication strategies of a CA when asking under-contributing members to participate more in a small group synchronous chat discussion. The five strategies include messages sent to two types of recipients (@username vs. @everyone) crossed by two separate channels (public vs. private), and a peer-mediated strategy where the CA asks a peer to address an under-contributing member. Through an online study with 185 crowdworkers, results show that the CA sending a private message specifying an individual is the most effective and preferred way to increase the participation of under-contributing members. Participants also expressed that the peer-mediated strategy promotes inclusiveness and teaches members how to involve others in the conversation. However, every strategy has trade-offs therefore my results suggest designing different communication strategies depending on group dynamics and situations. When detecting an under-contributing member, two types of detection errors can occur: 1) false-positive (FP) errors happen when the CA falsely identifies a member as under-contributing and 2) false-negative (FN) errors occur when the CA fails to identify an under-contributing member. The second experiment demonstrates how these inaccurate interventions impact user acceptance of the CA, members' participation in the discussion, and the perceived discussion experience between the members. I also tested a conversational repair strategy in which the agent sends a correctional message if an error is detected. Through an online study with 175 crowdworkers, the results demonstrated that the participants who encountered FN errors reported higher acceptance of the CA and had a more positive perceived discussion experience, but participated less compared to those who encountered FP errors. The conversational repair strategy moderated the effect of errors such as improving the perceived discussion experience of participants who received FP error messages. The results offer design implications for selecting a model between high precision (i.e., fewer FP errors) and high recall (i.e., fewer FN errors) models. When frequent FP errors are expected, the conversational repair strategy would be useful to improve the perceived discussion experience. In the third experiment, I designed three techniques to help users make appropriate expectations about how well the CA identifies under-contributing members: 1) the information technique explicitly communicates the agent's accuracy value (e.g., ``how accurate the algorithm detects whether someone is under-contributing or not is 77\%''), 2) the explanation technique gives an overview of the algorithmic decision process with actual data (e.g., ``you have sent 2 messages and 5 unique words, which is the least in the group''), and 3) the adjustment technique enables users to gain a feeling of control over the sensitivity of the detection algorithm (e.g., ``how would you like to adjust the performance of the detection algorithm''). Through an online study with 163 crowdworkers, the data show that participants who experienced the explanation technique felt a better understanding of the algorithm, were less embarrassed and perceived the CA as more intelligent compared to other techniques. The adjustment technique, however, led to a more negative perceived discussion experience and perceived group performance than other techniques. The information technique reduced members' participation because it reduced quality of contributions. The findings and discussions of the three experiments contribute to the design of a CA that opens a floor for more voices to participate and gets people to accept and adapt the facilitation techniques from the CA despite the technology being imperfect. I encourage teams to adopt a facilitator CA with improved designs in their discussion to balance members' participation, improve decision outcomes, and enhance the collaboration experience between members. This dissertation progresses toward a future of an online group discussion where more individuals make meaningful contributions and have positive group experiences through AI-team collaboration.

Graduation Semester

2023-05

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/120494

Copyright and License Information

Copyright 2023 Hyo Jin Do

Owning Collections