Theoretical Analysis and Empirical Evaluation of Coverage Indictors for Closed Source APP Testing
The number of mobile applications has increased geometrically nowadays, but how to ensure their quality and conduct adequate and effective testing is still a challenge for developers. On the one hand, the number of mobile apps is increasing, and the update speed is faster and faster. Many small and...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2019-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/8892488/ |
| _version_ | 1830299323071463424 |
|---|---|
| author | Sen Yang Song Huang Zhanwei Hui |
| author_facet | Sen Yang Song Huang Zhanwei Hui |
| author_sort | Sen Yang |
| collection | DOAJ |
| description | The number of mobile applications has increased geometrically nowadays, but how to ensure their quality and conduct adequate and effective testing is still a challenge for developers. On the one hand, the number of mobile apps is increasing, and the update speed is faster and faster. Many small and medium-sized companies can hardly test the app adequately before each release. On the other hand, mobile apps play more and more important roles in people's life, such as financial payment. For the sake of company security and user privacy, most companies will encrypt key codes in their APP. Even third-party testers cannot get source code, which also leads to many researchers cannot carry out further research and effective testing for these widely used mainstream APPs. Code coverage is an important indicator to guide software testing, which plays a crucial role in ensuring the quality of testing. However, it is an urgent problem to find accurate coverage indictors to evaluate these tests. And when testing those existing widely used mainstream closed-source apps, we find that the existing coarse-grained coverage metrics like method coverage is bad coverage indictors for app testing that can exaggerate or minimize the actual coverage rate, which cannot obtain satisfactory results for the evaluation of test effects. To find a more reliable coverage indictor, this paper demonstrates the correctness of instruction coverage indictor and the inaccurate of method coverage in evaluating the test of closed-source APP from the perspective of probability and statistics. Then we shows how inaccurate the method coverage can be through an empirical evaluation on datasets of closed source APPs and open source APPs respectively. It is further verified that instruction coverage is a more effective evaluation indictor than methods coverage or activity coverage in the test of closed source APP for the first time. |
| first_indexed | 2024-12-19T08:07:08Z |
| format | Article |
| id | doaj.art-b7ef144c8dbb458480444a1a91704efe |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-19T08:07:08Z |
| publishDate | 2019-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-b7ef144c8dbb458480444a1a91704efe2022-12-21T20:29:44ZengIEEEIEEE Access2169-35362019-01-01716232316233210.1109/ACCESS.2019.29519418892488Theoretical Analysis and Empirical Evaluation of Coverage Indictors for Closed Source APP TestingSen Yang0https://orcid.org/0000-0002-2263-8000Song Huang1https://orcid.org/0000-0002-0599-7474Zhanwei Hui2https://orcid.org/0000-0001-8892-1979Command and Control Engineering College, Army Engineering University of PLA, Nanjing, ChinaCommand and Control Engineering College, Army Engineering University of PLA, Nanjing, ChinaCommand and Control Engineering College, Army Engineering University of PLA, Nanjing, ChinaThe number of mobile applications has increased geometrically nowadays, but how to ensure their quality and conduct adequate and effective testing is still a challenge for developers. On the one hand, the number of mobile apps is increasing, and the update speed is faster and faster. Many small and medium-sized companies can hardly test the app adequately before each release. On the other hand, mobile apps play more and more important roles in people's life, such as financial payment. For the sake of company security and user privacy, most companies will encrypt key codes in their APP. Even third-party testers cannot get source code, which also leads to many researchers cannot carry out further research and effective testing for these widely used mainstream APPs. Code coverage is an important indicator to guide software testing, which plays a crucial role in ensuring the quality of testing. However, it is an urgent problem to find accurate coverage indictors to evaluate these tests. And when testing those existing widely used mainstream closed-source apps, we find that the existing coarse-grained coverage metrics like method coverage is bad coverage indictors for app testing that can exaggerate or minimize the actual coverage rate, which cannot obtain satisfactory results for the evaluation of test effects. To find a more reliable coverage indictor, this paper demonstrates the correctness of instruction coverage indictor and the inaccurate of method coverage in evaluating the test of closed-source APP from the perspective of probability and statistics. Then we shows how inaccurate the method coverage can be through an empirical evaluation on datasets of closed source APPs and open source APPs respectively. It is further verified that instruction coverage is a more effective evaluation indictor than methods coverage or activity coverage in the test of closed source APP for the first time.https://ieeexplore.ieee.org/document/8892488/Mobile testingautomated testingcoverage criteriainstruction coverage |
| spellingShingle | Sen Yang Song Huang Zhanwei Hui Theoretical Analysis and Empirical Evaluation of Coverage Indictors for Closed Source APP Testing IEEE Access Mobile testing automated testing coverage criteria instruction coverage |
| title | Theoretical Analysis and Empirical Evaluation of Coverage Indictors for Closed Source APP Testing |
| title_full | Theoretical Analysis and Empirical Evaluation of Coverage Indictors for Closed Source APP Testing |
| title_fullStr | Theoretical Analysis and Empirical Evaluation of Coverage Indictors for Closed Source APP Testing |
| title_full_unstemmed | Theoretical Analysis and Empirical Evaluation of Coverage Indictors for Closed Source APP Testing |
| title_short | Theoretical Analysis and Empirical Evaluation of Coverage Indictors for Closed Source APP Testing |
| title_sort | theoretical analysis and empirical evaluation of coverage indictors for closed source app testing |
| topic | Mobile testing automated testing coverage criteria instruction coverage |
| url | https://ieeexplore.ieee.org/document/8892488/ |
| work_keys_str_mv | AT senyang theoreticalanalysisandempiricalevaluationofcoverageindictorsforclosedsourceapptesting AT songhuang theoreticalanalysisandempiricalevaluationofcoverageindictorsforclosedsourceapptesting AT zhanweihui theoreticalanalysisandempiricalevaluationofcoverageindictorsforclosedsourceapptesting |