Massive data mining and analysis platform design for fully mechanized working face
The current real-time and integrity of massive data acquisition in fully mechanized working faces are poor. The abnormal data cleaning takes a long time. The data mining delays are large. This leads to low utilization rate of fully mechanized working data and incapability to assist management in iss...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Editorial Department of Industry and Mine Automation
2023-05-01
|
| Series: | Gong-kuang zidonghua |
| Subjects: | |
| Online Access: | http://www.gkzdh.cn/article/doi/10.13272/j.issn.1671-251x.18088 |
| _version_ | 1827902937827901440 |
|---|---|
| author | WANG Hongwei YANG Kun FU Xiang LI Jin JIA Sifeng |
| author_facet | WANG Hongwei YANG Kun FU Xiang LI Jin JIA Sifeng |
| author_sort | WANG Hongwei |
| collection | DOAJ |
| description | The current real-time and integrity of massive data acquisition in fully mechanized working faces are poor. The abnormal data cleaning takes a long time. The data mining delays are large. This leads to low utilization rate of fully mechanized working data and incapability to assist management in issuing decision-making instructions in real-time. In order to solve the above problems, a massive data mining and analysis platform for fully mechanized working faces is designed. The platform consists of a data source layer, a data acquisition and storage layer, a data mining layer, and a front-end application layer. The data source layer is provided with raw data by various hardware devices on the working surface. The data acquisition and storage layer uses the OPC UA gateway to collect real-time monitoring information from underground sensors, and then stores the data in the InfluxDB storage engine through the MQTT protocol and RESTful interface. The data mining layer uses the Hive data engine and Yarn resource manager to filter out abnormal data caused by workplace interference during the data acquisition process. It solves the problem of local data acquisition order disorder caused by network latency. The Spark distributed mining engine is used to explore the potential value of massive working condition data in the working face device group, improving the running speed of the data mining model. The front-end application layer utilizes visual components to associate with the back-end database. It interacts with the back-end data in real-time through AJAX technology to achieve visual display of model mining results and various monitoring data. The test results show that the platform can fully ensure the real-time and integrity of data acquisition. The cleaning efficiency is 5 times better than a standalone MySQL query engine and the mining efficiency is 4 times better than a standalone Python mining engine. |
| first_indexed | 2024-03-13T00:06:14Z |
| format | Article |
| id | doaj.art-d5a2e710c05148ccbec3eb6349607c21 |
| institution | Directory Open Access Journal |
| issn | 1671-251X |
| language | zho |
| last_indexed | 2024-03-13T00:06:14Z |
| publishDate | 2023-05-01 |
| publisher | Editorial Department of Industry and Mine Automation |
| record_format | Article |
| series | Gong-kuang zidonghua |
| spelling | doaj.art-d5a2e710c05148ccbec3eb6349607c212023-07-13T03:26:14ZzhoEditorial Department of Industry and Mine AutomationGong-kuang zidonghua1671-251X2023-05-014953036, 12610.13272/j.issn.1671-251x.18088Massive data mining and analysis platform design for fully mechanized working faceWANG Hongwei0YANG KunFU XiangLI JinJIA SifengCenter of Shanxi Engineering Research for Coal Mine Intelligent Equipment, Taiyuan University of Technology, Taiyuan 030024, ChinaThe current real-time and integrity of massive data acquisition in fully mechanized working faces are poor. The abnormal data cleaning takes a long time. The data mining delays are large. This leads to low utilization rate of fully mechanized working data and incapability to assist management in issuing decision-making instructions in real-time. In order to solve the above problems, a massive data mining and analysis platform for fully mechanized working faces is designed. The platform consists of a data source layer, a data acquisition and storage layer, a data mining layer, and a front-end application layer. The data source layer is provided with raw data by various hardware devices on the working surface. The data acquisition and storage layer uses the OPC UA gateway to collect real-time monitoring information from underground sensors, and then stores the data in the InfluxDB storage engine through the MQTT protocol and RESTful interface. The data mining layer uses the Hive data engine and Yarn resource manager to filter out abnormal data caused by workplace interference during the data acquisition process. It solves the problem of local data acquisition order disorder caused by network latency. The Spark distributed mining engine is used to explore the potential value of massive working condition data in the working face device group, improving the running speed of the data mining model. The front-end application layer utilizes visual components to associate with the back-end database. It interacts with the back-end data in real-time through AJAX technology to achieve visual display of model mining results and various monitoring data. The test results show that the platform can fully ensure the real-time and integrity of data acquisition. The cleaning efficiency is 5 times better than a standalone MySQL query engine and the mining efficiency is 4 times better than a standalone Python mining engine.http://www.gkzdh.cn/article/doi/10.13272/j.issn.1671-251x.18088fully mechanized working facemassive datadata miningdata acquisitiondata storagedata cleaningdata visualization |
| spellingShingle | WANG Hongwei YANG Kun FU Xiang LI Jin JIA Sifeng Massive data mining and analysis platform design for fully mechanized working face Gong-kuang zidonghua fully mechanized working face massive data data mining data acquisition data storage data cleaning data visualization |
| title | Massive data mining and analysis platform design for fully mechanized working face |
| title_full | Massive data mining and analysis platform design for fully mechanized working face |
| title_fullStr | Massive data mining and analysis platform design for fully mechanized working face |
| title_full_unstemmed | Massive data mining and analysis platform design for fully mechanized working face |
| title_short | Massive data mining and analysis platform design for fully mechanized working face |
| title_sort | massive data mining and analysis platform design for fully mechanized working face |
| topic | fully mechanized working face massive data data mining data acquisition data storage data cleaning data visualization |
| url | http://www.gkzdh.cn/article/doi/10.13272/j.issn.1671-251x.18088 |
| work_keys_str_mv | AT wanghongwei massivedataminingandanalysisplatformdesignforfullymechanizedworkingface AT yangkun massivedataminingandanalysisplatformdesignforfullymechanizedworkingface AT fuxiang massivedataminingandanalysisplatformdesignforfullymechanizedworkingface AT lijin massivedataminingandanalysisplatformdesignforfullymechanizedworkingface AT jiasifeng massivedataminingandanalysisplatformdesignforfullymechanizedworkingface |