Regression test prioritization for modern software

Cheng, Runxiang

Permalink

https://hdl.handle.net/2142/129204 Copy

Description

Title

Regression test prioritization for modern software

Author(s)

Cheng, Runxiang

Issue Date

2025-04-16

Director of Research (if dissertation) or Advisor (if thesis)

Marinov, Darko

Doctoral Committee Chair(s)

Marinov, Darko

Committee Member(s)

• Zhang, Lingming
• Xu, Tianyin
• Yang, Wei

Department of Study

Siebel School Comp & Data Sci

Discipline

Computer Science

Degree Granting Institution

University of Illinois Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Software engineering
• Software testing
• Regression testing
• Test prioritization

Abstract

Continuous Integration (CI) is a common practice in software development for increasing code quality. CI runs regression test suite on each code change to help software developers find faults in the change. As codebase size and code change frequency have grown rapidly in modern software development, regression test suite runtime has also increased, thereby preventing timely debugging feedback to developers. Regression Test Prioritization (RTP) aims to expose faults in code change sooner by reordering tests in the test suite, so that the ones likely to fail are run earlier. Despite that RTP has been extensively studied for nearly three decades, research results of RTP are rarely put into practice. To facilitate the adoption of RTP for modern software systems, RTP must be demonstrated effective in speeding up regression fault detection in relevant and practical testing settings. Moreover, robust and efficient RTP tooling must be available for practitioners to easily apply RTP research results in popular software ecosystems. This dissertation takes a step forward in addressing these challenges via three lines of work. First, this dissertation examines key research findings from prior RTP studies on a newly-proposed dataset of long-running test suites with real test failures and up-to-date CI practices. Through extensive evaluation, this dissertation identifies the most effective and efficient RTP techniques under the impact of realistic CI issues such as flaky test failures. Second, this dissertation applies traditional RTP techniques for configuration testing, an emerging but critical testing scenario; it further proposes novel and effective RTP techniques for configuration testing. This dissertation assesses various factors that influence the effectiveness of RTP in speeding up misconfiguration detection, and provides detailed analyses and guidelines in applying RTP to configuration tests. Third, this dissertation presents pytest-ranking, a readily-usable open-source RTP tool for Python and its most popular testing framework Pytest. This dissertation realistically evaluates the efficacy of pytest-ranking, demonstrating that pytest-ranking is well integrated with Python/Pytest ecosystems, can be conveniently deployed into CI, has low overhead, and can detect faults substantially faster than baselines.

Graduation Semester

2025-05

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/129204

Copyright and License Information

Copyright 2025 Runxiang Cheng

Owning Collections