Reusing software tests for configuration testing: A case study of the Hadoop project

Abstract

Configuration is an inseparable piece of today’s software development. Due to its dynamic nature and lack of standardized checking procedure, misconfiguration has been one of the dominant factors contributing to software failures in production and wide spread outages. Existing techniques (static validation, system tests, canaries, etc.) for detecting misconfigurations have limitations, either being too coarse grained and unable to precisely exercise the changed configuration value, or not considering the interaction between configuration and software source code. Configuration testing takes pages from traditional software testing practices to test software configurations in a similar vein as how software code is tested. This thesis makes a first step exploration towards configuration testing, focusing on analyzing feasibility and effectiveness of reusing existing source code tests to test configuration values. Through building a semi-automated infrastructure associating unit tests with configuration parameters, and later run the associated tests against injected correct and incorrect configuration values for each parameter, this thesis shows that reusing existing software test for configuration testing directly could yield an effective rate of 41.7% and 48.8%, for Hadoop Common and HDFS, respectively. Further, this thesis conducts in-depth analysis on tests that cannot be effectively reused, categorizes code patterns of these tests that have false positives or negatives, and provides examples on rewriting these tests.