Discovering, autogenerating, and evaluating distractors for Parsons problems in CS1

Smith IV, David Hamilton

Permalink

https://hdl.handle.net/2142/129414 Copy

Description

Title

Discovering, autogenerating, and evaluating distractors for Parsons problems in CS1

Author(s)

Smith IV, David Hamilton

Issue Date

2025-04-23

Director of Research (if dissertation) or Advisor (if thesis)

Zilles, Craig

Doctoral Committee Chair(s)

Zilles, Craig

Committee Member(s)

• Lewis, Colleen
• Ericson, Barbara
• Cunningham, Kathryn

Department of Study

Siebel School Comp & Data Sci

Discipline

Computer Science

Degree Granting Institution

University of Illinois Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Computing Education
• Parsons problems
• CS1
• Distractors
• Item Response Theory
• Classical Test Theory
• Measurement Theory

Abstract

This dissertation presents work on evaluating the use of distractor blocks in Parsons problems, a problem type aimed at supporting novice programmers in learning to write code. Since their introduction by Parsons & Hayden [1], distractors have become common place in studies investigating Parsons problems. However, prior to the work presented in this dissertation, there had been a limited number of studies investigating their utility. In filling this gap, my work has focused on investigating methods of developing distractors and the impact of distractors in both summative and formative contexts. I first introduce a method of creating distractor templates from analysis of code writing errors and using those templates to support the autogeneration of distractors [2], a process that has been integrated into CodeSpec, a programming practice and eTextbook platform [3]. I present a series of studies comparing Parsons problems on exams that include distractors to those that do not. These studies have also included a comparison between different methods of including distractors, specifically comparing the use of distractors that are jumbled among the other options to distractors that are visually-grouped with their correct alternative [4, 5]. Additionally, I present a study comparing the impact on learning gains when practicing with Parsons problems that include distractors versus those without distractors. Learning gains are measured through pre- and post-assessment performance. In support of this work, I have also added the ability to associate feedback with distractors in the PrairieLearn assessment platform. The findings presented suggest that in summative contexts the presence of a single distractor group does not meaningfully increase question difficulty or item quality while causing students to spend significantly more time on the problems, a concern in a time constrained context such as an exam. However, as more distractor groups are added difficulty continues to increase with minimal impact—either positively or negatively—to item quality. This suggests that when distractors are added in sufficient quantities they can be useful for reducing the ceiling effect on score we observed in Parsons problems used in this study that do not include distractors. In formative contexts, the findings I present suggest that the presence of distractors causes students to engage more deeply with the code presented in blocks they are grouped with. The results of a randomized controlled trial indicate that students practicing with questions including distractors had measurable learning gains as measured through the post-test.

Graduation Semester

2025-05

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/129414

Copyright and License Information

Copyright 2025 David Smith IV

Owning Collections