A multidimensional approach for analyzing variants of code writing questions in a CS1 course

Abstract

To defend against collaborative cheating in code writing questions, instructors of CS1 courses with asynchronous exams can use the strategy of question variants, being manually written questions to be selected at random to assess the same learning goal. In order to create these variants, currently the instructors have to rely on intuition to accomplish the competing goals of ensuring variants are different enough to defend against collaborative cheating, and yet similar enough where students are assessed fairly. In this paper, we propose a multidimensional approach of analyzing these variants. We apply our approach on a dataset of 3 midterm exams from a large CS1 course. Our results show that (1) observable inequalities exist between variants and (2) these differences are not just limited to score. Our results also show that the information gathered from our analysis approach can be used to provide recommendations for improving design of future variants.