IoTFuzzBench: A Pragmatic Benchmarking Framework for Evaluating IoT Black-Box Protocol Fuzzers
High scalability and low operating cost make black-box protocol fuzzing a vital tool for discovering vulnerabilities in the firmware of IoT smart devices. However, it is still challenging to compare black-box protocol fuzzers due to the lack of unified benchmark firmware images, complete fuzzing mut...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-07-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/12/14/3010 |
| _version_ | 1797589505956380672 |
|---|---|
| author | Yixuan Cheng Wenxin Chen Wenqing Fan Wei Huang Gaoqing Yu Wen Liu |
| author_facet | Yixuan Cheng Wenxin Chen Wenqing Fan Wei Huang Gaoqing Yu Wen Liu |
| author_sort | Yixuan Cheng |
| collection | DOAJ |
| description | High scalability and low operating cost make black-box protocol fuzzing a vital tool for discovering vulnerabilities in the firmware of IoT smart devices. However, it is still challenging to compare black-box protocol fuzzers due to the lack of unified benchmark firmware images, complete fuzzing mutation seeds, comprehensive performance metrics, and a standardized evaluation framework. In this paper, we design and implement IoTFuzzBench, a scalable, modular, metric-driven automation framework for evaluating black-box protocol fuzzers for IoT smart devices comprehensively and quantitatively. Specifically, IoTFuzzBench has so far included 14 real-world benchmark firmware images, 30 verified real-world benchmark vulnerabilities, complete fuzzing seeds for each vulnerability, 7 popular fuzzers, and 5 categories of complementary performance metrics. We deployed IoTFuzzBench and evaluated 7 popular black-box protocol fuzzers on all benchmark firmware images and benchmark vulnerabilities. The experimental results show that IoTFuzzBench can not only provide fast, reliable, and reproducible experiments, but also effectively evaluate the ability of each fuzzer to find vulnerabilities and the differential performance on different performance metrics. The fuzzers found a total of 13 vulnerabilities out of 30. None of these fuzzers can outperform the others on all metrics. This result demonstrates the importance of comprehensive metrics. We hope our findings ease the burden of fuzzing evaluation in IoT scenarios, advancing more pragmatic and reproducible fuzzer benchmarking efforts. |
| first_indexed | 2024-03-11T01:08:34Z |
| format | Article |
| id | doaj.art-36fad15be6c84beaa4ec903deec32de8 |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-11T01:08:34Z |
| publishDate | 2023-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-36fad15be6c84beaa4ec903deec32de82023-11-18T19:04:30ZengMDPI AGElectronics2079-92922023-07-011214301010.3390/electronics12143010IoTFuzzBench: A Pragmatic Benchmarking Framework for Evaluating IoT Black-Box Protocol FuzzersYixuan Cheng0Wenxin Chen1Wenqing Fan2Wei Huang3Gaoqing Yu4Wen Liu5State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, ChinaState Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, ChinaState Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, ChinaState Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, ChinaState Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, ChinaState Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, ChinaHigh scalability and low operating cost make black-box protocol fuzzing a vital tool for discovering vulnerabilities in the firmware of IoT smart devices. However, it is still challenging to compare black-box protocol fuzzers due to the lack of unified benchmark firmware images, complete fuzzing mutation seeds, comprehensive performance metrics, and a standardized evaluation framework. In this paper, we design and implement IoTFuzzBench, a scalable, modular, metric-driven automation framework for evaluating black-box protocol fuzzers for IoT smart devices comprehensively and quantitatively. Specifically, IoTFuzzBench has so far included 14 real-world benchmark firmware images, 30 verified real-world benchmark vulnerabilities, complete fuzzing seeds for each vulnerability, 7 popular fuzzers, and 5 categories of complementary performance metrics. We deployed IoTFuzzBench and evaluated 7 popular black-box protocol fuzzers on all benchmark firmware images and benchmark vulnerabilities. The experimental results show that IoTFuzzBench can not only provide fast, reliable, and reproducible experiments, but also effectively evaluate the ability of each fuzzer to find vulnerabilities and the differential performance on different performance metrics. The fuzzers found a total of 13 vulnerabilities out of 30. None of these fuzzers can outperform the others on all metrics. This result demonstrates the importance of comprehensive metrics. We hope our findings ease the burden of fuzzing evaluation in IoT scenarios, advancing more pragmatic and reproducible fuzzer benchmarking efforts.https://www.mdpi.com/2079-9292/12/14/3010fuzzing evaluationblack-box fuzzingIoT smart devicebenchmark suite |
| spellingShingle | Yixuan Cheng Wenxin Chen Wenqing Fan Wei Huang Gaoqing Yu Wen Liu IoTFuzzBench: A Pragmatic Benchmarking Framework for Evaluating IoT Black-Box Protocol Fuzzers Electronics fuzzing evaluation black-box fuzzing IoT smart device benchmark suite |
| title | IoTFuzzBench: A Pragmatic Benchmarking Framework for Evaluating IoT Black-Box Protocol Fuzzers |
| title_full | IoTFuzzBench: A Pragmatic Benchmarking Framework for Evaluating IoT Black-Box Protocol Fuzzers |
| title_fullStr | IoTFuzzBench: A Pragmatic Benchmarking Framework for Evaluating IoT Black-Box Protocol Fuzzers |
| title_full_unstemmed | IoTFuzzBench: A Pragmatic Benchmarking Framework for Evaluating IoT Black-Box Protocol Fuzzers |
| title_short | IoTFuzzBench: A Pragmatic Benchmarking Framework for Evaluating IoT Black-Box Protocol Fuzzers |
| title_sort | iotfuzzbench a pragmatic benchmarking framework for evaluating iot black box protocol fuzzers |
| topic | fuzzing evaluation black-box fuzzing IoT smart device benchmark suite |
| url | https://www.mdpi.com/2079-9292/12/14/3010 |
| work_keys_str_mv | AT yixuancheng iotfuzzbenchapragmaticbenchmarkingframeworkforevaluatingiotblackboxprotocolfuzzers AT wenxinchen iotfuzzbenchapragmaticbenchmarkingframeworkforevaluatingiotblackboxprotocolfuzzers AT wenqingfan iotfuzzbenchapragmaticbenchmarkingframeworkforevaluatingiotblackboxprotocolfuzzers AT weihuang iotfuzzbenchapragmaticbenchmarkingframeworkforevaluatingiotblackboxprotocolfuzzers AT gaoqingyu iotfuzzbenchapragmaticbenchmarkingframeworkforevaluatingiotblackboxprotocolfuzzers AT wenliu iotfuzzbenchapragmaticbenchmarkingframeworkforevaluatingiotblackboxprotocolfuzzers |