A Novel Dictionary-Based Method to Compress Log Files with Different Message Frequency Distributions
In the present day, virtually every application software generates large amounts of log entries during its work. The log files that are made from these entries are a collection of information about what happened while the program was running. This report can be used for multiple purposes such as per...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-02-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/12/4/2044 |
| _version_ | 1797482992422092800 |
|---|---|
| author | Péter Marjai Péter Lehotay-Kéry Attila Kiss |
| author_facet | Péter Marjai Péter Lehotay-Kéry Attila Kiss |
| author_sort | Péter Marjai |
| collection | DOAJ |
| description | In the present day, virtually every application software generates large amounts of log entries during its work. The log files that are made from these entries are a collection of information about what happened while the program was running. This report can be used for multiple purposes such as performance monitoring, maintaining security, or improving business decision making. Log entries are usually generated in a disorganized manner. Using template miners, the different ‘event types’ can be distinguished (each log entry is an event), and the set of all entries is split into disjointed subsets according to the event types. These events consist of two parts. The first is the constant part, which is the same for all occurrences of the same event type. The second is the parameter part, which can be different for each occurrence. Since software mass-produces log files, in our previous paper, we introduced an algorithm that uses the templates mined from the data to create a dictionary, which is then used to encode the log entries, so only the ID and the parameter list would be stored. In this paper, we enhance our algorithm with the use of the frequency of the templates, by encoding the parameters and also making use of Huffman coding. With the use of these measures, compared to the previous 67.4% compression rate, a 94.98% compression rate can be achieved (where compression rate is 1 minus the ratio of the size of the compressed file to the uncompressed size). The running times of the different measures that we used to enhance our algorithm are also compared. We also analyze the difference between the compression rate of the enhanced algorithm and general compressors such as LZMA, Bzip2, and PPMd. We examine whether the size of the log files can be further decreased with the combined use of our enhanced method and the general compressors. We also generate log files that follow different distributions to examine the compression capability if the distribution does not follow the power law. Based on our experiments, we would recommend the use of the MoLFI (Multi-objective Log message Format Identification) template miner method with our enhanced algorithm together with PPMd. |
| first_indexed | 2024-03-09T22:41:30Z |
| format | Article |
| id | doaj.art-4dceae35a9ba490ba33e5d73c41ee62a |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-09T22:41:30Z |
| publishDate | 2022-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-4dceae35a9ba490ba33e5d73c41ee62a2023-11-23T18:38:15ZengMDPI AGApplied Sciences2076-34172022-02-01124204410.3390/app12042044A Novel Dictionary-Based Method to Compress Log Files with Different Message Frequency DistributionsPéter Marjai0Péter Lehotay-Kéry1Attila Kiss2Department of Information Systems, ELTE Eötvös Loránd University, 1117 Budapest, HungaryDepartment of Information Systems, ELTE Eötvös Loránd University, 1117 Budapest, HungaryDepartment of Information Systems, ELTE Eötvös Loránd University, 1117 Budapest, HungaryIn the present day, virtually every application software generates large amounts of log entries during its work. The log files that are made from these entries are a collection of information about what happened while the program was running. This report can be used for multiple purposes such as performance monitoring, maintaining security, or improving business decision making. Log entries are usually generated in a disorganized manner. Using template miners, the different ‘event types’ can be distinguished (each log entry is an event), and the set of all entries is split into disjointed subsets according to the event types. These events consist of two parts. The first is the constant part, which is the same for all occurrences of the same event type. The second is the parameter part, which can be different for each occurrence. Since software mass-produces log files, in our previous paper, we introduced an algorithm that uses the templates mined from the data to create a dictionary, which is then used to encode the log entries, so only the ID and the parameter list would be stored. In this paper, we enhance our algorithm with the use of the frequency of the templates, by encoding the parameters and also making use of Huffman coding. With the use of these measures, compared to the previous 67.4% compression rate, a 94.98% compression rate can be achieved (where compression rate is 1 minus the ratio of the size of the compressed file to the uncompressed size). The running times of the different measures that we used to enhance our algorithm are also compared. We also analyze the difference between the compression rate of the enhanced algorithm and general compressors such as LZMA, Bzip2, and PPMd. We examine whether the size of the log files can be further decreased with the combined use of our enhanced method and the general compressors. We also generate log files that follow different distributions to examine the compression capability if the distribution does not follow the power law. Based on our experiments, we would recommend the use of the MoLFI (Multi-objective Log message Format Identification) template miner method with our enhanced algorithm together with PPMd.https://www.mdpi.com/2076-3417/12/4/2044log file processingtemplate miningcompressionLZMABzip2PPMd |
| spellingShingle | Péter Marjai Péter Lehotay-Kéry Attila Kiss A Novel Dictionary-Based Method to Compress Log Files with Different Message Frequency Distributions Applied Sciences log file processing template mining compression LZMA Bzip2 PPMd |
| title | A Novel Dictionary-Based Method to Compress Log Files with Different Message Frequency Distributions |
| title_full | A Novel Dictionary-Based Method to Compress Log Files with Different Message Frequency Distributions |
| title_fullStr | A Novel Dictionary-Based Method to Compress Log Files with Different Message Frequency Distributions |
| title_full_unstemmed | A Novel Dictionary-Based Method to Compress Log Files with Different Message Frequency Distributions |
| title_short | A Novel Dictionary-Based Method to Compress Log Files with Different Message Frequency Distributions |
| title_sort | novel dictionary based method to compress log files with different message frequency distributions |
| topic | log file processing template mining compression LZMA Bzip2 PPMd |
| url | https://www.mdpi.com/2076-3417/12/4/2044 |
| work_keys_str_mv | AT petermarjai anoveldictionarybasedmethodtocompresslogfileswithdifferentmessagefrequencydistributions AT peterlehotaykery anoveldictionarybasedmethodtocompresslogfileswithdifferentmessagefrequencydistributions AT attilakiss anoveldictionarybasedmethodtocompresslogfileswithdifferentmessagefrequencydistributions AT petermarjai noveldictionarybasedmethodtocompresslogfileswithdifferentmessagefrequencydistributions AT peterlehotaykery noveldictionarybasedmethodtocompresslogfileswithdifferentmessagefrequencydistributions AT attilakiss noveldictionarybasedmethodtocompresslogfileswithdifferentmessagefrequencydistributions |