A constant-time chunking algorithm for packet-level deduplication
Redundancy elimination or deduplication over network packets requires significant computing resources to find basic units of repeated contents, called chunks, by checking every byte in every packet. In this paper, we present the first constant-time chunking algorithm that divides every packet into a...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2019-06-01
|
| Series: | ICT Express |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2405959518302157 |
| _version_ | 1811270330541408256 |
|---|---|
| author | MyungKeun Yoon |
| author_facet | MyungKeun Yoon |
| author_sort | MyungKeun Yoon |
| collection | DOAJ |
| description | Redundancy elimination or deduplication over network packets requires significant computing resources to find basic units of repeated contents, called chunks, by checking every byte in every packet. In this paper, we present the first constant-time chunking algorithm that divides every packet into a predefined number of chunks, irrespective of the packet size. In addition, we present the best implementation practice for packet-level deduplication by selecting an optimal combination of chunking, fingerprinting, and hash table algorithms. Through experiments with real traffic, we confirm that the throughput is improved by three times, compared with even the state-of-the-art scheme. Keywords: Deduplication, Chunking algorithm, Packet capture, Network security |
| first_indexed | 2024-04-12T21:58:55Z |
| format | Article |
| id | doaj.art-e4497667d67444a2b5f08343fcc7b7e6 |
| institution | Directory Open Access Journal |
| issn | 2405-9595 |
| language | English |
| last_indexed | 2024-04-12T21:58:55Z |
| publishDate | 2019-06-01 |
| publisher | Elsevier |
| record_format | Article |
| series | ICT Express |
| spelling | doaj.art-e4497667d67444a2b5f08343fcc7b7e62022-12-22T03:15:12ZengElsevierICT Express2405-95952019-06-0152131135A constant-time chunking algorithm for packet-level deduplicationMyungKeun Yoon0Department of Computer Engineering, Kookmin University, Seoul, Republic of KoreaRedundancy elimination or deduplication over network packets requires significant computing resources to find basic units of repeated contents, called chunks, by checking every byte in every packet. In this paper, we present the first constant-time chunking algorithm that divides every packet into a predefined number of chunks, irrespective of the packet size. In addition, we present the best implementation practice for packet-level deduplication by selecting an optimal combination of chunking, fingerprinting, and hash table algorithms. Through experiments with real traffic, we confirm that the throughput is improved by three times, compared with even the state-of-the-art scheme. Keywords: Deduplication, Chunking algorithm, Packet capture, Network securityhttp://www.sciencedirect.com/science/article/pii/S2405959518302157 |
| spellingShingle | MyungKeun Yoon A constant-time chunking algorithm for packet-level deduplication ICT Express |
| title | A constant-time chunking algorithm for packet-level deduplication |
| title_full | A constant-time chunking algorithm for packet-level deduplication |
| title_fullStr | A constant-time chunking algorithm for packet-level deduplication |
| title_full_unstemmed | A constant-time chunking algorithm for packet-level deduplication |
| title_short | A constant-time chunking algorithm for packet-level deduplication |
| title_sort | constant time chunking algorithm for packet level deduplication |
| url | http://www.sciencedirect.com/science/article/pii/S2405959518302157 |
| work_keys_str_mv | AT myungkeunyoon aconstanttimechunkingalgorithmforpacketleveldeduplication AT myungkeunyoon constanttimechunkingalgorithmforpacketleveldeduplication |