Shifting paradigms in the UX evaluation of human-AI interaction: From dyadic to monadic designs

Zheng, Qingxiao

Permalink

https://hdl.handle.net/2142/125801 Copy

Description

Title

Shifting paradigms in the UX evaluation of human-AI interaction: From dyadic to monadic designs

Author(s)

Zheng, Qingxiao

Issue Date

2024-07-10

Director of Research (if dissertation) or Advisor (if thesis)

Huang, Yun

Doctoral Committee Chair(s)

Huang, Yun

Committee Member(s)

• Yao, Mike
• Wang, Yang
• Bosch, Nigel

Department of Study

Information Sciences

Discipline

Information Sciences

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Human--AI interaction
• Evaluation
• Generative AI
• User Experience

Abstract

With generative AI, we are entering a new era where individuals without coding backgrounds can become creators of AI-based solutions for delivering different services. For example, business owners can define AI chatbots to answer customers’ inquiries, and legal professionals can tailor AI to draft, review, and manage legal documentation. However, there is a lack of a systematic framework to evaluate the UX of human--AI collaboration in these interactions. This thesis identifies a paradigm shift in the UX evaluation of human--AI interaction. Prior to generative AI, UX research began with examining dyadic interactions between end users and AI, progressively expanding to polyadic interactions, where AI mediates between end users adopting multi-stakeholder perspectives (e.g., service providers and service consumers). The evaluation of these interaction modalities often considers user satisfaction, linguistic features, and task completion rate, and the challenges that mainly arise as a result of the uncertainty and complexity of AI. With the advent of generative AI, individuals with minimal AI literacy can become creators, introducing a new modality---monadic interaction---which emphasizes the unity and feedback loop between AI systems and their Creator-Users. These creator-users actively participate in defining and refining the AI's functions, allowing both to adapt and evolve. Although humans and AI systems can leverage their respective strengths to achieve better outcomes than either could independently, assessing how well AI aligns with creators’ values and intentions poses significant challenges, particularly when creators' norms deviate from broader societal standards. This dissertation further introduces a UX evaluation framework called EvalignUX (evaluating alignment of UX), designed to guide the evaluation of the three interaction modalities while maintaining the alignment focus. First, I synthesized the literature on UX research in conversational human-AI interaction before the popularity of generative AI and proposed four dimensions for evaluating dyadic and polyadic interactions: linguistics and prosodics quality, task performance, socio-emotional intelligence, and ethics and risks. Each dimension includes several evaluation metrics, such as word variation, sense of control, perceived warmth and competence, and security perception. These evaluation metrics can serve as a design material to enhance UX effectively. I then extended the EvalignUX framework to include monadic interactions through two case studies. The first case demonstrates the creator-AI interaction within a multistakeholder service, where public service providers customize their AI to address community members’ queries. New evaluation metrics such as "pronominal comprehension" and "identity-based emotive expression" prove crucial when AI is designed to play the role of human creators as service providers. The second case illustrates the creator-AI interaction for self-defined services, where creators possess diverse domain expertise and require AI to provide customized documentation for individual use. In this context, a new evaluation dimension, "normative fluidity," is particularly significant in assessing AI's ability to adapt to varying normative expectations. The term "fluidity" underscores the smoothness and ease of transition, when creators’ task definitions differ from the existing AI models. The proposed EvalignUX framework can assist UX researchers and design tool makers in addressing the challenges of evaluating AI systems in a responsive fashion, from dyadic to monadic interactions. The thesis concludes with future research towards more aligned AI with boundary awareness.

Graduation Semester

2024-08

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/125801

Copyright and License Information

Copyright 2024 Qingxiao Zheng

Owning Collections