Know You Neighbor: Fast Static Prediction of Test Flakiness
Context: Flaky tests plague regression testing in Continuous Integration environments by slowing down change releases and wasting testing time and effort. Despite the growing interest in mitigating the burden of test flakiness, how to efficiently and effectively detect flaky tests is still an open p...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9437181/ |
| _version_ | 1818645423821160448 |
|---|---|
| author | Roberto Verdecchia Emilio Cruciani Breno Miranda Antonia Bertolino |
| author_facet | Roberto Verdecchia Emilio Cruciani Breno Miranda Antonia Bertolino |
| author_sort | Roberto Verdecchia |
| collection | DOAJ |
| description | Context: Flaky tests plague regression testing in Continuous Integration environments by slowing down change releases and wasting testing time and effort. Despite the growing interest in mitigating the burden of test flakiness, how to efficiently and effectively detect flaky tests is still an open problem. Objective: In this study, we present and evaluate FLAST, an approach designed to statically predict test flakiness. FLAST leverages vector-space modeling, similarity search, dimensionality reduction, and <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-Nearest Neighbor classification in order to timely and efficiently detect test flakiness. Method: In order to gain insights into the efficiency and effectiveness of FLAST, we conduct an empirical evaluation of the approach by considering 13 real-world projects, for a total of 1,383 flaky and 26,702 non-flaky tests. We carry out a quantitative comparison of FLAST with the state-of-the-art methods to detect test flakiness, by considering a balanced dataset comprising 1,402 real-world flaky and as many non-flaky tests. Results: From the results we observe that the effectiveness of FLAST is comparable with the state-of-the-art, while providing considerable gains in terms of efficiency. In addition, the results demonstrate how by tuning the threshold of the approach FLAST can be made more conservative, so to reduce false positives, at the cost of missing more potentially flaky tests. Conclusion: The collected results demonstrate that FLAST provides a fast, low-cost and reliable approach that can be used to guide test rerunning, or to gate the inclusion of new potentially flaky tests. |
| first_indexed | 2024-12-17T00:30:31Z |
| format | Article |
| id | doaj.art-5d13dbc34c0a4727a5a861a4bf2d52ad |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-17T00:30:31Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-5d13dbc34c0a4727a5a861a4bf2d52ad2022-12-21T22:10:20ZengIEEEIEEE Access2169-35362021-01-019761197613410.1109/ACCESS.2021.30824249437181Know You Neighbor: Fast Static Prediction of Test FlakinessRoberto Verdecchia0https://orcid.org/0000-0001-9206-6637Emilio Cruciani1https://orcid.org/0000-0002-4744-5635Breno Miranda2https://orcid.org/0000-0001-9608-9393Antonia Bertolino3https://orcid.org/0000-0001-8749-1356Department of Computer Science, Vrije Universiteit Amsterdam, Amsterdam, HV, The NetherlandsComputerwissenschaften, Universität Salzburg, Salzburg, AustriaCenter of Informatics, Federal University of Pernambuco, Recife, BrazilIstituto di Scienza e Tecnologie dell’Informazione “A. Faedo”, Consiglio Nazionale delle Ricerche, Pisa, ItalyContext: Flaky tests plague regression testing in Continuous Integration environments by slowing down change releases and wasting testing time and effort. Despite the growing interest in mitigating the burden of test flakiness, how to efficiently and effectively detect flaky tests is still an open problem. Objective: In this study, we present and evaluate FLAST, an approach designed to statically predict test flakiness. FLAST leverages vector-space modeling, similarity search, dimensionality reduction, and <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-Nearest Neighbor classification in order to timely and efficiently detect test flakiness. Method: In order to gain insights into the efficiency and effectiveness of FLAST, we conduct an empirical evaluation of the approach by considering 13 real-world projects, for a total of 1,383 flaky and 26,702 non-flaky tests. We carry out a quantitative comparison of FLAST with the state-of-the-art methods to detect test flakiness, by considering a balanced dataset comprising 1,402 real-world flaky and as many non-flaky tests. Results: From the results we observe that the effectiveness of FLAST is comparable with the state-of-the-art, while providing considerable gains in terms of efficiency. In addition, the results demonstrate how by tuning the threshold of the approach FLAST can be made more conservative, so to reduce false positives, at the cost of missing more potentially flaky tests. Conclusion: The collected results demonstrate that FLAST provides a fast, low-cost and reliable approach that can be used to guide test rerunning, or to gate the inclusion of new potentially flaky tests.https://ieeexplore.ieee.org/document/9437181/Software testingflaky testspredictionstatic analysissimilarity |
| spellingShingle | Roberto Verdecchia Emilio Cruciani Breno Miranda Antonia Bertolino Know You Neighbor: Fast Static Prediction of Test Flakiness IEEE Access Software testing flaky tests prediction static analysis similarity |
| title | Know You Neighbor: Fast Static Prediction of Test Flakiness |
| title_full | Know You Neighbor: Fast Static Prediction of Test Flakiness |
| title_fullStr | Know You Neighbor: Fast Static Prediction of Test Flakiness |
| title_full_unstemmed | Know You Neighbor: Fast Static Prediction of Test Flakiness |
| title_short | Know You Neighbor: Fast Static Prediction of Test Flakiness |
| title_sort | know you neighbor fast static prediction of test flakiness |
| topic | Software testing flaky tests prediction static analysis similarity |
| url | https://ieeexplore.ieee.org/document/9437181/ |
| work_keys_str_mv | AT robertoverdecchia knowyouneighborfaststaticpredictionoftestflakiness AT emiliocruciani knowyouneighborfaststaticpredictionoftestflakiness AT brenomiranda knowyouneighborfaststaticpredictionoftestflakiness AT antoniabertolino knowyouneighborfaststaticpredictionoftestflakiness |