A Seed Scheduling Method With a Reinforcement Learning for a Coverage Guided Fuzzing
Seed scheduling, which determines which seed is input to the fuzzer first and the number of mutated test cases that are generated for the input seed, significantly influences crash detection performance in fuzz testing. Even for the same fuzzer, the performance in terms of detecting crashes that cau...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10005088/ |
| _version_ | 1797956357924585472 |
|---|---|
| author | Gyeongtaek Choi Seungho Jeon Jaeik Cho Jongsub Moon |
| author_facet | Gyeongtaek Choi Seungho Jeon Jaeik Cho Jongsub Moon |
| author_sort | Gyeongtaek Choi |
| collection | DOAJ |
| description | Seed scheduling, which determines which seed is input to the fuzzer first and the number of mutated test cases that are generated for the input seed, significantly influences crash detection performance in fuzz testing. Even for the same fuzzer, the performance in terms of detecting crashes that cause program failure varies considerably depending on the seed-scheduling method used. Most existing coverage-guided fuzzers use a heuristic seed-scheduling method. These heuristic methods can’t properly determine the seed with a high potential to cause the crash; thus, the fuzzer detects the crash inefficiently. Moreover, the fuzzer’s crash detection performance is affected by the characteristics of target programs. To address this problem, we propose a general-purpose reinforced seed-scheduling method that not only improves the crash detection performance of fuzz testing but also remains unaffected by the characteristics of the target program. The fuzzer with the proposed method detected the most crashes in all but one of the target programs in which crashes were detected in the experimental results conducted on various programs, and showed better crash detection efficiency than the comparison targets overall. |
| first_indexed | 2024-04-10T23:47:43Z |
| format | Article |
| id | doaj.art-1b21782f709f4b8a9afb98949b3af075 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-10T23:47:43Z |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-1b21782f709f4b8a9afb98949b3af0752023-01-11T00:00:19ZengIEEEIEEE Access2169-35362023-01-01112048205710.1109/ACCESS.2022.323387510005088A Seed Scheduling Method With a Reinforcement Learning for a Coverage Guided FuzzingGyeongtaek Choi0https://orcid.org/0000-0001-5794-7439Seungho Jeon1https://orcid.org/0000-0002-7116-6062Jaeik Cho2https://orcid.org/0000-0001-6665-8604Jongsub Moon3https://orcid.org/0000-0002-6457-4316Department of Information Security, School of Cybersecurity, Korea University, Seoul, Republic of KoreaDepartment of Information Security, School of Cybersecurity, Korea University, Seoul, Republic of KoreaDepartment of Engineering, Computer Science, Lewis University, Romeoville, IL, USADepartment of Information Security, School of Cybersecurity, Korea University, Seoul, Republic of KoreaSeed scheduling, which determines which seed is input to the fuzzer first and the number of mutated test cases that are generated for the input seed, significantly influences crash detection performance in fuzz testing. Even for the same fuzzer, the performance in terms of detecting crashes that cause program failure varies considerably depending on the seed-scheduling method used. Most existing coverage-guided fuzzers use a heuristic seed-scheduling method. These heuristic methods can’t properly determine the seed with a high potential to cause the crash; thus, the fuzzer detects the crash inefficiently. Moreover, the fuzzer’s crash detection performance is affected by the characteristics of target programs. To address this problem, we propose a general-purpose reinforced seed-scheduling method that not only improves the crash detection performance of fuzz testing but also remains unaffected by the characteristics of the target program. The fuzzer with the proposed method detected the most crashes in all but one of the target programs in which crashes were detected in the experimental results conducted on various programs, and showed better crash detection efficiency than the comparison targets overall.https://ieeexplore.ieee.org/document/10005088/Coverage-guided fuzzingpower schedulereinforcement learningseed schedulingseed selection |
| spellingShingle | Gyeongtaek Choi Seungho Jeon Jaeik Cho Jongsub Moon A Seed Scheduling Method With a Reinforcement Learning for a Coverage Guided Fuzzing IEEE Access Coverage-guided fuzzing power schedule reinforcement learning seed scheduling seed selection |
| title | A Seed Scheduling Method With a Reinforcement Learning for a Coverage Guided Fuzzing |
| title_full | A Seed Scheduling Method With a Reinforcement Learning for a Coverage Guided Fuzzing |
| title_fullStr | A Seed Scheduling Method With a Reinforcement Learning for a Coverage Guided Fuzzing |
| title_full_unstemmed | A Seed Scheduling Method With a Reinforcement Learning for a Coverage Guided Fuzzing |
| title_short | A Seed Scheduling Method With a Reinforcement Learning for a Coverage Guided Fuzzing |
| title_sort | seed scheduling method with a reinforcement learning for a coverage guided fuzzing |
| topic | Coverage-guided fuzzing power schedule reinforcement learning seed scheduling seed selection |
| url | https://ieeexplore.ieee.org/document/10005088/ |
| work_keys_str_mv | AT gyeongtaekchoi aseedschedulingmethodwithareinforcementlearningforacoverageguidedfuzzing AT seunghojeon aseedschedulingmethodwithareinforcementlearningforacoverageguidedfuzzing AT jaeikcho aseedschedulingmethodwithareinforcementlearningforacoverageguidedfuzzing AT jongsubmoon aseedschedulingmethodwithareinforcementlearningforacoverageguidedfuzzing AT gyeongtaekchoi seedschedulingmethodwithareinforcementlearningforacoverageguidedfuzzing AT seunghojeon seedschedulingmethodwithareinforcementlearningforacoverageguidedfuzzing AT jaeikcho seedschedulingmethodwithareinforcementlearningforacoverageguidedfuzzing AT jongsubmoon seedschedulingmethodwithareinforcementlearningforacoverageguidedfuzzing |