Log Data Real Time Analysis Using Big Data Analytic Framework with Storm and Hadoop
The log data real-time processing platform which is built using Storm On YARN integrated MapReduce and Storm that use MapReduce to complete large-scale off-line data global knowledge extraction, sudden knowledge extraction of small-scale data in Kafka buffers through Storm, and continuous real-time...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
EDP Sciences
2018-01-01
|
| Series: | MATEC Web of Conferences |
| Online Access: | https://doi.org/10.1051/matecconf/201824603009 |
| _version_ | 1818941152264454144 |
|---|---|
| author | Lv Jia-Ke Li Yang Wang Xuan |
| author_facet | Lv Jia-Ke Li Yang Wang Xuan |
| author_sort | Lv Jia-Ke |
| collection | DOAJ |
| description | The log data real-time processing platform which is built using Storm On YARN integrated MapReduce and Storm that use MapReduce to complete large-scale off-line data global knowledge extraction, sudden knowledge extraction of small-scale data in Kafka buffers through Storm, and continuous real-time calculation of streaming data in combination with global knowledge. We tested our technique with the well-known KDD99 CUP data set. The experimentation results prove the system to be effective and efficient. |
| first_indexed | 2024-12-20T06:50:59Z |
| format | Article |
| id | doaj.art-eca7f01b84d5447db9bb1cc71018a27f |
| institution | Directory Open Access Journal |
| issn | 2261-236X |
| language | English |
| last_indexed | 2024-12-20T06:50:59Z |
| publishDate | 2018-01-01 |
| publisher | EDP Sciences |
| record_format | Article |
| series | MATEC Web of Conferences |
| spelling | doaj.art-eca7f01b84d5447db9bb1cc71018a27f2022-12-21T19:49:33ZengEDP SciencesMATEC Web of Conferences2261-236X2018-01-012460300910.1051/matecconf/201824603009matecconf_iswso2018_03009Log Data Real Time Analysis Using Big Data Analytic Framework with Storm and HadoopLv Jia-KeLi Yang0Wang XuanCollege of computer and information science, Southwest UniversityThe log data real-time processing platform which is built using Storm On YARN integrated MapReduce and Storm that use MapReduce to complete large-scale off-line data global knowledge extraction, sudden knowledge extraction of small-scale data in Kafka buffers through Storm, and continuous real-time calculation of streaming data in combination with global knowledge. We tested our technique with the well-known KDD99 CUP data set. The experimentation results prove the system to be effective and efficient.https://doi.org/10.1051/matecconf/201824603009 |
| spellingShingle | Lv Jia-Ke Li Yang Wang Xuan Log Data Real Time Analysis Using Big Data Analytic Framework with Storm and Hadoop MATEC Web of Conferences |
| title | Log Data Real Time Analysis Using Big Data Analytic Framework with Storm and Hadoop |
| title_full | Log Data Real Time Analysis Using Big Data Analytic Framework with Storm and Hadoop |
| title_fullStr | Log Data Real Time Analysis Using Big Data Analytic Framework with Storm and Hadoop |
| title_full_unstemmed | Log Data Real Time Analysis Using Big Data Analytic Framework with Storm and Hadoop |
| title_short | Log Data Real Time Analysis Using Big Data Analytic Framework with Storm and Hadoop |
| title_sort | log data real time analysis using big data analytic framework with storm and hadoop |
| url | https://doi.org/10.1051/matecconf/201824603009 |
| work_keys_str_mv | AT lvjiake logdatarealtimeanalysisusingbigdataanalyticframeworkwithstormandhadoop AT liyang logdatarealtimeanalysisusingbigdataanalyticframeworkwithstormandhadoop AT wangxuan logdatarealtimeanalysisusingbigdataanalyticframeworkwithstormandhadoop |