Files in this item



application/pdfGYORI-DISSERTATION-2017.pdf (804kB)
(no description provided)PDF


Title:Proactively detecting unreliable tests
Author(s):Gyori, Alex
Director of Research:Marinov, Darko
Doctoral Committee Chair(s):Marinov, Darko
Doctoral Committee Member(s):Parthasarathy, Madhusudan; Xie, Tao; Lahiri, Shuvendu K
Department / Program:Computer Science
Discipline:Computer Science
Degree Granting Institution:University of Illinois at Urbana-Champaign
Subject(s):Unreliable tests
Flaky tests
Proactive detection
Abstract:Regression testing is the most wide-spread method to ensure the quality of software systems. Whenever a change is made to the software, tests are run to ensure bugs are not introduced: if all tests pass, the change is merged into the codebase; otherwise, the developer needs to identify the bug that was introduced by the change. Developers assume that the outcomes of the tests in the regression testing process are reliable, i.e., that the failure indicates a bug introduced by the change. Unfortunately, unreliable tests manage to get into the test suite, slowing down the developers’ workflow and having developers debug not their software but rather the test code or infrastructure. This dissertation presents two techniques to enable developers to more easily and proactively detect and debug unreliable tests early, before they become a problem and slow down the development process. Developers write unreliable tests, which may pass on their machine but may at a later point fail because the environment changes. This dissertation presents a technique to detect when code makes wrong assumptions on underdetermined APIs. While these assumptions may hold in the current environment, they may not hold in the future, causing tests to fail. The technique, NonDex, detects such wrong assumptions by exploring different behaviors that may not manifest in the current implementation, but are allowed by the specification. Furthermore, the dissertation presents PolDet, a technique to detect when tests pollute their environment; such polluting tests can cause other tests to fail or pass seemingly nondeterministically because of the environment pollution rather than changes in the code. The two techniques enable developers to identify when they are introducing unreliability in their test suite and help identify the root causes of the unreliable tests. The results of the evaluation on open-source projects show the techniques are effective at identifying issues in open-source code and also that developers are eager to fix the issues and adopt the tools.
Issue Date:2017-07-10
Rights Information:Copyright 2017 Alex Gyori
Date Available in IDEALS:2017-09-29
Date Deposited:2017-08

This item appears in the following Collection(s)

Item Statistics