Structural test coverage criteria for deep neural networks
Deep neural networks (DNNs) have a wide range of applications, and software employing them must be thoroughly tested, especially in safety-critical domains. However, traditional software test coverage metrics cannot be applied directly to DNNs. In this paper, inspired by the MC/DC coverage criterion...
| Main Authors: | , , , , , |
|---|---|
| Format: | Journal article |
| Published: |
Association for Computing Machinery
2019
|
| _version_ | 1797053648091480064 |
|---|---|
| author | Sun, Y Huang, X Kroening, D Sharp, J Hill, M Ashmore, R |
| author_facet | Sun, Y Huang, X Kroening, D Sharp, J Hill, M Ashmore, R |
| author_sort | Sun, Y |
| collection | OXFORD |
| description | Deep neural networks (DNNs) have a wide range of applications, and software employing them must be thoroughly tested, especially in safety-critical domains. However, traditional software test coverage metrics cannot be applied directly to DNNs. In this paper, inspired by the MC/DC coverage criterion, we propose a family of four novel test coverage criteria that are tailored to structural features of DNNs and their semantics. We validate the criteria by demonstrating that the generated test inputs guided via our proposed coverage criteria are able to capture undesired behaviours in a DNN. Test cases are generated using a symbolic approach and a gradient-based heuristic search. By comparing them with existing methods, we show that our criteria achieve a balance between their ability to find bugs (proxied using adversarial examples and correlation with functional coverage) and the computational cost of test case generation. Our experiments are conducted on state-of-the-art DNNs obtained using popular open source datasets, including MNIST, CIFAR-10 and ImageNet. |
| first_indexed | 2024-03-06T18:46:37Z |
| format | Journal article |
| id | oxford-uuid:0ebc8d5f-de08-4077-868f-09ca30c2de7e |
| institution | University of Oxford |
| last_indexed | 2024-03-06T18:46:37Z |
| publishDate | 2019 |
| publisher | Association for Computing Machinery |
| record_format | dspace |
| spelling | oxford-uuid:0ebc8d5f-de08-4077-868f-09ca30c2de7e2022-03-26T09:47:31ZStructural test coverage criteria for deep neural networksJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:0ebc8d5f-de08-4077-868f-09ca30c2de7eSymplectic Elements at OxfordAssociation for Computing Machinery2019Sun, YHuang, XKroening, DSharp, JHill, MAshmore, RDeep neural networks (DNNs) have a wide range of applications, and software employing them must be thoroughly tested, especially in safety-critical domains. However, traditional software test coverage metrics cannot be applied directly to DNNs. In this paper, inspired by the MC/DC coverage criterion, we propose a family of four novel test coverage criteria that are tailored to structural features of DNNs and their semantics. We validate the criteria by demonstrating that the generated test inputs guided via our proposed coverage criteria are able to capture undesired behaviours in a DNN. Test cases are generated using a symbolic approach and a gradient-based heuristic search. By comparing them with existing methods, we show that our criteria achieve a balance between their ability to find bugs (proxied using adversarial examples and correlation with functional coverage) and the computational cost of test case generation. Our experiments are conducted on state-of-the-art DNNs obtained using popular open source datasets, including MNIST, CIFAR-10 and ImageNet. |
| spellingShingle | Sun, Y Huang, X Kroening, D Sharp, J Hill, M Ashmore, R Structural test coverage criteria for deep neural networks |
| title | Structural test coverage criteria for deep neural networks |
| title_full | Structural test coverage criteria for deep neural networks |
| title_fullStr | Structural test coverage criteria for deep neural networks |
| title_full_unstemmed | Structural test coverage criteria for deep neural networks |
| title_short | Structural test coverage criteria for deep neural networks |
| title_sort | structural test coverage criteria for deep neural networks |
| work_keys_str_mv | AT suny structuraltestcoveragecriteriafordeepneuralnetworks AT huangx structuraltestcoveragecriteriafordeepneuralnetworks AT kroeningd structuraltestcoveragecriteriafordeepneuralnetworks AT sharpj structuraltestcoveragecriteriafordeepneuralnetworks AT hillm structuraltestcoveragecriteriafordeepneuralnetworks AT ashmorer structuraltestcoveragecriteriafordeepneuralnetworks |