Empirically revisiting and enhancing IR-based test-case prioritization

Peng, Qianyang

Permalink

https://hdl.handle.net/2142/105272 Copy

Description

Title

Empirically revisiting and enhancing IR-based test-case prioritization

Author(s)

Peng, Qianyang

Issue Date

2019-04-26

Director of Research (if dissertation) or Advisor (if thesis)

• Marinov, Darko
• Zhang, Lingming

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Software Testing
• Test Case Prioritization
• Flaky Test
• Information Retrieval

Abstract

Regression testing is widely used to check code changes during software evolution. How- ever, regression testing can also incur huge cost since real-world software systems can accumulate huge and time-consuming test suites. Therefore, a large number of approaches have been proposed to speed up regression testing, including test-case prioritization, regression test selection, and test-suite minimization/reduction. Test-case prioritization (TCP) aims to detect regression bugs faster via reordering regression tests. To date, various TCP techniques have been proposed in the literature, including both change-aware ones that consider program changes between the old and new versions for better prioritization and change-unaware ones that simply use the dynamic or static in- formation from the old version. Although various studies have investigated the effectiveness of the existing TCP techniques, they suffer from the following threats: (1) they are usually evaluated on a small dataset; (2) they are usually performed on seeded artificial or real bugs, and not on real evolution with real bugs; (3) they are usually evaluated using cost-unaware metrics. In this work, we study the recent program changes based information retrieval (IR) approach for TCP, which has been claimed to perform better than state-of-the-art coverage based techniques but only evaluated on a small dataset by cost-unaware metrics. We reduce the threats of and further enhance the prior work by conducting a better evaluation and proposing potential improvements. To do so, we evaluate the original technique on a large-scale, real-world software-evolution dataset containing 123 projects and 2,980 program builds using both cost-aware and cost-unaware metrics under various configurations, and we design and evaluate several hybrid techniques combining the IR features and historical test execution times and failure frequencies. As a result, we confirm the effectiveness of program changes based IR approach for TCP and find an ideal configuration to maximize the effectiveness. Also, we successfully improve the performance of the original technique with our hybrid techniques. Moreover, we show that flaky tests have a substantial impact on the program changes based TCP techniques.

Graduation Semester

2019-05

Type of Resource

text

Permalink

http://hdl.handle.net/2142/105272

Copyright and License Information

Copyright 2019 Qianyang Peng

Owning Collections