GUI Testing Using Computer Vision
Testing a GUI's visual behavior typically requires human testers to interact with the GUI and to observe whether the expected results of interaction are presented. This paper presents a new approach to GUI testing using computer vision for testers to automate their tasks. Testers can write a vi...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | en_US |
| Published: |
Association for Computing Machinery (ACM)
2012
|
| Online Access: | http://hdl.handle.net/1721.1/72684 https://orcid.org/0000-0002-0442-691X |
| _version_ | 1826214282414522368 |
|---|---|
| author | Chang, Tsung-Hsiang Yeh, Tom Miller, Robert C. |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Chang, Tsung-Hsiang Yeh, Tom Miller, Robert C. |
| author_sort | Chang, Tsung-Hsiang |
| collection | MIT |
| description | Testing a GUI's visual behavior typically requires human testers to interact with the GUI and to observe whether the expected results of interaction are presented. This paper presents a new approach to GUI testing using computer vision for testers to automate their tasks. Testers can write a visual test script that uses images to specify which GUI components to interact with and what visual feedback to be observed. Testers can also generate visual test scripts by demonstration. By recording both input events and screen images, it is possible to extract the images of components interacted with and the visual feedback seen by the demonstrator, and generate a visual test script automatically. We show that a variety of GUI behavior can be tested using this approach. Also, we show how this approach can facilitate good testing practices such as unit testing, regression testing, and test-driven development. |
| first_indexed | 2024-09-23T11:41:05Z |
| format | Article |
| id | mit-1721.1/72684 |
| institution | Massachusetts Institute of Technology |
| language | en_US |
| last_indexed | 2024-09-23T11:41:05Z |
| publishDate | 2012 |
| publisher | Association for Computing Machinery (ACM) |
| record_format | dspace |
| spelling | mit-1721.1/726842022-10-01T05:14:21Z GUI Testing Using Computer Vision Chang, Tsung-Hsiang Yeh, Tom Miller, Robert C. Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Miller, Robert C. Chang, Tsung-Hsiang Miller, Robert C. Testing a GUI's visual behavior typically requires human testers to interact with the GUI and to observe whether the expected results of interaction are presented. This paper presents a new approach to GUI testing using computer vision for testers to automate their tasks. Testers can write a visual test script that uses images to specify which GUI components to interact with and what visual feedback to be observed. Testers can also generate visual test scripts by demonstration. By recording both input events and screen images, it is possible to extract the images of components interacted with and the visual feedback seen by the demonstrator, and generate a visual test script automatically. We show that a variety of GUI behavior can be tested using this approach. Also, we show how this approach can facilitate good testing practices such as unit testing, regression testing, and test-driven development. National Science Foundation (U.S.). (Grant number IIS-0447800) Quanta Computer (Firm) (TParty project) 2012-09-13T14:41:24Z 2012-09-13T14:41:24Z 2010-04 Article http://purl.org/eprint/type/ConferencePaper 978-1-60558-929-9 http://hdl.handle.net/1721.1/72684 Tsung-Hsiang Chang, Tom Yeh, and Robert C. Miller. 2010. GUI testing using computer vision. In Proceedings of the 28th international conference on Human factors in computing systems (CHI '10). ACM, New York, NY, USA, 1535-1544. https://orcid.org/0000-0002-0442-691X en_US http://dx.doi.org/10.1145/1753326.1753555 Proceedings of the 28th International Conference on Human Factors in Computing Systems (CHI '10) Creative Commons Attribution-Noncommercial-Share Alike 3.0 http://creativecommons.org/licenses/by-nc-sa/3.0/ application/pdf Association for Computing Machinery (ACM) MIT web domain |
| spellingShingle | Chang, Tsung-Hsiang Yeh, Tom Miller, Robert C. GUI Testing Using Computer Vision |
| title | GUI Testing Using Computer Vision |
| title_full | GUI Testing Using Computer Vision |
| title_fullStr | GUI Testing Using Computer Vision |
| title_full_unstemmed | GUI Testing Using Computer Vision |
| title_short | GUI Testing Using Computer Vision |
| title_sort | gui testing using computer vision |
| url | http://hdl.handle.net/1721.1/72684 https://orcid.org/0000-0002-0442-691X |
| work_keys_str_mv | AT changtsunghsiang guitestingusingcomputervision AT yehtom guitestingusingcomputervision AT millerrobertc guitestingusingcomputervision |