CipherTrace: automatic detection of ciphers from execution traces

Cryptography is defensive in nature, as it provides confidentiality, integrity, and authenticity to its consumers for data processing, storage and transmission. The employment of cryptography in software has introduced additional complexity in investigating the intent of a binary program or a runnin...

Full description

Bibliographic Details
Main Author: Hassanin, MA
Format: Thesis
Language:English
Published: 2020
Subjects:
_version_ 1824458931083149312
author Hassanin, MA
author_facet Hassanin, MA
author_sort Hassanin, MA
collection OXFORD
description Cryptography is defensive in nature, as it provides confidentiality, integrity, and authenticity to its consumers for data processing, storage and transmission. The employment of cryptography in software has introduced additional complexity in investigating the intent of a binary program or a running process. The rapid-increasing trend of using cryptography in malicious software, has re-introduced cryptography from an ”active-defense” perspective. This trend is known as cryptovirology, which was first introduced in 1996. The ever-increasing number of malware samples employing cryptography has demanded for more sophisticated analysis techniques to account for the malware and its operation(s), and if applicable, allow for neutralization and/or recovery. Generally, the detection and identification of cryptographic functions, routines, or algorithms facilitates malware analysis and forensics. However, most of the attempts presented in the literature have basically followed a top-down approach, which (by design) has limited them, as they are mainly focused on detecting certain implementations, rather than the building blocks, elements, or ingredients of cryptography in general. Nevertheless, the lack of work on automating the process makes it harder to scale in quantity and quality. In this dissertation, we explore the idea of generalizing the detection of cryptography, via dynamically analyzing Out-of-VM execution traces of binary programs. The prototype we developed (CipherTrace) is based on PANDA, the architecture-neutral dynamic analysis platform. PANDA was favored among others mainly due to its plugin architecture as well as its record-and-replay functionality, which allows for developing extensions easily and repeatable analysis. PANDA is based on QEMU, which is a whole-system emulator and virtualizer. In QEMU, the in-guest components are not visible to the guest programs, therefore it provides isolation to a certain extend, wherein the results are more true-hearted. We apply custom heuristics on the synthetic information of called functions, e.g., count of arithmetic operations, number of executions of basic blocks, memory access patterns, and more. The synthetic information can be categorized as raw (i.e., Platform-driven or Assembly-driven), or lifted (i.e., LLVM-driven). Finally, we report the findings, observations, limitations, and assert the feasibility and scalability of the concept, the dynamic analysis approach, as well as the technique applied to identify cryptographic (crypto) elements to classify a cipher. Crypto elements are the generic elementary units (or constituents) of cryptography, i.e., the operational functional blocks (e.g., a substitution or permutation step, etc).
first_indexed 2024-03-07T08:05:40Z
format Thesis
id oxford-uuid:c8f36d8c-3b3e-40bf-9b4e-7f557fef5e83
institution University of Oxford
language English
last_indexed 2025-02-19T04:33:43Z
publishDate 2020
record_format dspace
spelling oxford-uuid:c8f36d8c-3b3e-40bf-9b4e-7f557fef5e832025-01-14T09:34:54ZCipherTrace: automatic detection of ciphers from execution tracesThesishttp://purl.org/coar/resource_type/c_bdccuuid:c8f36d8c-3b3e-40bf-9b4e-7f557fef5e83CryptographyComputer securityMalware (Computer software)EnglishHyrax Deposit2020Hassanin, MACryptography is defensive in nature, as it provides confidentiality, integrity, and authenticity to its consumers for data processing, storage and transmission. The employment of cryptography in software has introduced additional complexity in investigating the intent of a binary program or a running process. The rapid-increasing trend of using cryptography in malicious software, has re-introduced cryptography from an ”active-defense” perspective. This trend is known as cryptovirology, which was first introduced in 1996. The ever-increasing number of malware samples employing cryptography has demanded for more sophisticated analysis techniques to account for the malware and its operation(s), and if applicable, allow for neutralization and/or recovery. Generally, the detection and identification of cryptographic functions, routines, or algorithms facilitates malware analysis and forensics. However, most of the attempts presented in the literature have basically followed a top-down approach, which (by design) has limited them, as they are mainly focused on detecting certain implementations, rather than the building blocks, elements, or ingredients of cryptography in general. Nevertheless, the lack of work on automating the process makes it harder to scale in quantity and quality. In this dissertation, we explore the idea of generalizing the detection of cryptography, via dynamically analyzing Out-of-VM execution traces of binary programs. The prototype we developed (CipherTrace) is based on PANDA, the architecture-neutral dynamic analysis platform. PANDA was favored among others mainly due to its plugin architecture as well as its record-and-replay functionality, which allows for developing extensions easily and repeatable analysis. PANDA is based on QEMU, which is a whole-system emulator and virtualizer. In QEMU, the in-guest components are not visible to the guest programs, therefore it provides isolation to a certain extend, wherein the results are more true-hearted. We apply custom heuristics on the synthetic information of called functions, e.g., count of arithmetic operations, number of executions of basic blocks, memory access patterns, and more. The synthetic information can be categorized as raw (i.e., Platform-driven or Assembly-driven), or lifted (i.e., LLVM-driven). Finally, we report the findings, observations, limitations, and assert the feasibility and scalability of the concept, the dynamic analysis approach, as well as the technique applied to identify cryptographic (crypto) elements to classify a cipher. Crypto elements are the generic elementary units (or constituents) of cryptography, i.e., the operational functional blocks (e.g., a substitution or permutation step, etc).
spellingShingle Cryptography
Computer security
Malware (Computer software)
Hassanin, MA
CipherTrace: automatic detection of ciphers from execution traces
title CipherTrace: automatic detection of ciphers from execution traces
title_full CipherTrace: automatic detection of ciphers from execution traces
title_fullStr CipherTrace: automatic detection of ciphers from execution traces
title_full_unstemmed CipherTrace: automatic detection of ciphers from execution traces
title_short CipherTrace: automatic detection of ciphers from execution traces
title_sort ciphertrace automatic detection of ciphers from execution traces
topic Cryptography
Computer security
Malware (Computer software)
work_keys_str_mv AT hassaninma ciphertraceautomaticdetectionofciphersfromexecutiontraces