Modelling penetration testing with reinforcement learning using capture‐the‐flag challenges: Trade‐offs between model‐free learning and a priori knowledge
Abstract Penetration testing is a security exercise aimed at assessing the security of a system by simulating attacks against it. So far, penetration testing has been carried out mainly by trained human attackers and its success critically depended on the available expertise. Automating this practic...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Hindawi-IET
2023-05-01
|
| Series: | IET Information Security |
| Subjects: | |
| Online Access: | https://doi.org/10.1049/ise2.12107 |
| _version_ | 1797422366463098880 |
|---|---|
| author | Fabio Massimo Zennaro László Erdődi |
| author_facet | Fabio Massimo Zennaro László Erdődi |
| author_sort | Fabio Massimo Zennaro |
| collection | DOAJ |
| description | Abstract Penetration testing is a security exercise aimed at assessing the security of a system by simulating attacks against it. So far, penetration testing has been carried out mainly by trained human attackers and its success critically depended on the available expertise. Automating this practice constitutes a non‐trivial problem because of the range and complexity of actions that a human expert may attempt. The authors focus their attention on simplified penetration testing problems expressed in the form of capture the flag hacking challenges, and analyse how model‐free reinforcement learning algorithms may help solving them. In modelling these capture the flag competitions as reinforcement learning problems the authors highlight the specific challenges that characterize penetration testing. The authors show how this challenge may be eased by relying on different forms of prior knowledge that may be provided to the agent. Since complexity scales exponentially as soon as the set of states and actions for the reinforcement learning agent is extended, the need to restrict the exploration space by using techniques to inject a priori knowledge is highlighted, thus making it possible to achieve solutions more efficiently. |
| first_indexed | 2024-03-09T07:31:17Z |
| format | Article |
| id | doaj.art-6bea086b3ef1496b835d55ccecd805b5 |
| institution | Directory Open Access Journal |
| issn | 1751-8709 1751-8717 |
| language | English |
| last_indexed | 2024-03-09T07:31:17Z |
| publishDate | 2023-05-01 |
| publisher | Hindawi-IET |
| record_format | Article |
| series | IET Information Security |
| spelling | doaj.art-6bea086b3ef1496b835d55ccecd805b52023-12-03T06:14:31ZengHindawi-IETIET Information Security1751-87091751-87172023-05-0117344145710.1049/ise2.12107Modelling penetration testing with reinforcement learning using capture‐the‐flag challenges: Trade‐offs between model‐free learning and a priori knowledgeFabio Massimo Zennaro0László Erdődi1Department of Informatics University of Oslo Oslo NorwayDepartment of Information Security and Communication Technology NTNU Trondheim NorwayAbstract Penetration testing is a security exercise aimed at assessing the security of a system by simulating attacks against it. So far, penetration testing has been carried out mainly by trained human attackers and its success critically depended on the available expertise. Automating this practice constitutes a non‐trivial problem because of the range and complexity of actions that a human expert may attempt. The authors focus their attention on simplified penetration testing problems expressed in the form of capture the flag hacking challenges, and analyse how model‐free reinforcement learning algorithms may help solving them. In modelling these capture the flag competitions as reinforcement learning problems the authors highlight the specific challenges that characterize penetration testing. The authors show how this challenge may be eased by relying on different forms of prior knowledge that may be provided to the agent. Since complexity scales exponentially as soon as the set of states and actions for the reinforcement learning agent is extended, the need to restrict the exploration space by using techniques to inject a priori knowledge is highlighted, thus making it possible to achieve solutions more efficiently.https://doi.org/10.1049/ise2.12107capture the flagimitation learningpenetration testingQ‐learningreinforcement learning |
| spellingShingle | Fabio Massimo Zennaro László Erdődi Modelling penetration testing with reinforcement learning using capture‐the‐flag challenges: Trade‐offs between model‐free learning and a priori knowledge IET Information Security capture the flag imitation learning penetration testing Q‐learning reinforcement learning |
| title | Modelling penetration testing with reinforcement learning using capture‐the‐flag challenges: Trade‐offs between model‐free learning and a priori knowledge |
| title_full | Modelling penetration testing with reinforcement learning using capture‐the‐flag challenges: Trade‐offs between model‐free learning and a priori knowledge |
| title_fullStr | Modelling penetration testing with reinforcement learning using capture‐the‐flag challenges: Trade‐offs between model‐free learning and a priori knowledge |
| title_full_unstemmed | Modelling penetration testing with reinforcement learning using capture‐the‐flag challenges: Trade‐offs between model‐free learning and a priori knowledge |
| title_short | Modelling penetration testing with reinforcement learning using capture‐the‐flag challenges: Trade‐offs between model‐free learning and a priori knowledge |
| title_sort | modelling penetration testing with reinforcement learning using capture the flag challenges trade offs between model free learning and a priori knowledge |
| topic | capture the flag imitation learning penetration testing Q‐learning reinforcement learning |
| url | https://doi.org/10.1049/ise2.12107 |
| work_keys_str_mv | AT fabiomassimozennaro modellingpenetrationtestingwithreinforcementlearningusingcapturetheflagchallengestradeoffsbetweenmodelfreelearningandaprioriknowledge AT laszloerdodi modellingpenetrationtestingwithreinforcementlearningusingcapturetheflagchallengestradeoffsbetweenmodelfreelearningandaprioriknowledge |