Files in this item

FilesDescriptionFormat

application/pdf

application/pdfELOUSSI-THESIS-2015.pdf (311kB)
(no description provided)PDF

Description

Title:Determining flaky tests from test failures
Author(s):Eloussi, Lamyaa
Department / Program:Computer Science
Discipline:Computer Science
Degree Granting Institution:University of Illinois at Urbana-Champaign
Degree:M.S.
Genre:Thesis
Subject(s):Software Engineering
Testing
flaky tests
non-determinism
Abstract:Automated regression testing is widely used in modern software development. Whenever a developer pushes some changes to a repository, tests are run to check whether the changes broke some functionality. When previously passing tests fail, the most recent changes are typically suspected, and developers invest time and effort to debug those changes. Unfortunately, new test failures may not be due to the latest changes but due to non-deterministic tests, popularly called flaky tests, that can pass or fail even without changes to the code under test. Many projects have such flaky tests, which can cause developers to lose confidence in test results. Therefore, developers need techniques that can help them determine whether a test failure is due to their latest changes and warrants their debugging, or whether it is due to a flaky test that should be potentially debugged by someone else. The most widely used technique for determining whether a test failure is due to a flaky test is to rerun the failing test multiple times immediately after it fails: if some rerun does pass, the test is definitely flaky, but if all reruns still fail, the status is unknown. This thesis proposes three improvements to this basic technique: (1) postponing the reruns, (2) rerunning in a new runtime environment (e.g., a new JVM for Java tests), and (3) intersecting the test coverage with the latest changes. The thesis evaluates the cost of (1) and (2) and evaluates the applicability of (3) on 15 projects with a total of 2715 test classes, 10 of which contain previously known flaky tests. The results show that the proposed improvements are highly applicable and would be able to determine that more failures are due to flaky tests for the same or somewhat higher cost as rerunning failing tests immediately after failure.
Issue Date:2015-04-29
Type:Thesis
URI:http://hdl.handle.net/2142/78543
Rights Information:Copyright 2015 Lamyaa Eloussi
Date Available in IDEALS:2015-07-22
Date Deposited:May 2015


This item appears in the following Collection(s)

Item Statistics