Empirically evaluating Greedy-based test suite reduction methods at different levels of test suite complexity
| Autor: | Chu-Ti Lin, Jiun-Shiang Wang, Gregory M. Kapfhammer, Kai-Wei Tang |
|---|---|
| Rok vydání: | 2017 |
| Předmět: |
Computer science
business.industry Suite 020207 software engineering 02 engineering and technology Test method Machine learning computer.software_genre Test (assessment) Test case Regression testing 0202 electrical engineering electronic engineering information engineering Test suite 020201 artificial intelligence & image processing Artificial intelligence Test Management Approach business computer Software Simulation Test data |
| Zdroj: | Science of Computer Programming. 150:1-25 |
| ISSN: | 0167-6423 |
| Popis: | Test suite reduction is an important approach that decreases the cost of regression testing. A test suite reduction technique operates based on the relationship between the test cases in the regression test suite and the test requirements in the program under test. Thus, its effectiveness should be closely related to the complexity of a regression test suite the product of the number of test cases and the number of test requirements. Our previous work has shown that cost-aware techniques (i.e., the test suite reduction techniques that aim to decrease the regression test suite's execution cost) generally outperform the others in terms of decreasing the cost of running the regression test suite. However, the previous empirical studies that evaluated cost-aware techniques did not take into account test suite complexity. That is, prior experiments do not reveal if the cost-aware techniques scale and work effectively on test suites with more test cases and more test requirements. This means that prior experiments do not appropriately shed light on how well test suite reduction methods work with large programs or test suites. Therefore, this paper focuses on the Greedy-based techniques and empirically evaluates the additional Greedy and two cost-aware Greedy techniques at different levels of test suite complexity from various standpoints including the cost taken to run the regression test suite, the time taken to reduce the test suites, the total regression testing costs, the fault detection capability, the fault detection efficiency, and the common rates of the representative sets. To the best of our knowledge, none of the previous empirical studies classify a considerable number of test suites according to their complexity. Nor do any prior experiments evaluate the test suite reduction techniques, in terms of the aforementioned criteria, at different levels of test suite complexity. This paper represents the first such attempt to carry out this important task. Based on the empirical results, we confirm the strengths and weaknesses of the cost-aware techniques and develop insights into how the cost-aware techniques' effectiveness varies as the test suite complexity increases. We evaluate test reduction methods at various levels of test suite complexity.Few previous studies evaluated the test reduction methods in the way that we did.The cost-aware methods generally attain the lowest total regression testing costs.The cost-aware methods generally realize higher fault detection efficiency.The benefits of using cost-aware methods increase as test suite complexity grows. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect