Locality/Fairness-Aware Job Scheduling in Distributed Stream Processing Engines
Distributed stream processing engines (DSPEs) deploy multiple tasks on distributed servers to process data streams in real time. Many DSPEs have provided locality-aware stream partitioning (LSP) methods to reduce network communication costs. However, an even job scheduler provided by DSPEs deploys t...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-11-01
|
| Series: | Electronics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-9292/9/11/1857 |
| _version_ | 1797548776134541312 |
|---|---|
| author | Siwoon Son Yang-Sae Moon |
| author_facet | Siwoon Son Yang-Sae Moon |
| author_sort | Siwoon Son |
| collection | DOAJ |
| description | Distributed stream processing engines (DSPEs) deploy multiple tasks on distributed servers to process data streams in real time. Many DSPEs have provided locality-aware stream partitioning (LSP) methods to reduce network communication costs. However, an even job scheduler provided by DSPEs deploys tasks far away from each other on the distributed servers, which cannot use the LSP properly. In this paper, we propose a Locality/Fairness-aware job scheduler (L/F job scheduler) that considers locality together to solve problems of the even job scheduler that only considers fairness. First, the L/F job scheduler increases cohesion of contiguous tasks that require message transmissions for the locality. At the same time, it reduces coupling of parallel tasks that do not require message transmissions for the fairness. Next, we connect the contiguous tasks into a stream pipeline and evenly deploy stream pipelines to the distributed servers so that the L/F job scheduler achieves high cohesion and low coupling. Finally, we implement the proposed L/F job scheduler in Apache Storm, a representative DSPE, and evaluate it in both synthetic and real-world workloads. Experimental results show that the L/F job scheduler is similar in throughput compared to the even job scheduler, but latency is significantly improved by up to 139.2% for the LSP applications and by up to 140.7% even for the non-LSP applications. The L/F job scheduler also improves latency by 19.58% and 12.13%, respectively, in two real-world workloads. These results indicate that our L/F job scheduler provides superior processing performance for the DSPE applications. |
| first_indexed | 2024-03-10T15:04:30Z |
| format | Article |
| id | doaj.art-d75f2b4e830d42b09650ef2bb945731c |
| institution | Directory Open Access Journal |
| issn | 2079-9292 |
| language | English |
| last_indexed | 2024-03-10T15:04:30Z |
| publishDate | 2020-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Electronics |
| spelling | doaj.art-d75f2b4e830d42b09650ef2bb945731c2023-11-20T19:55:01ZengMDPI AGElectronics2079-92922020-11-01911185710.3390/electronics9111857Locality/Fairness-Aware Job Scheduling in Distributed Stream Processing EnginesSiwoon Son0Yang-Sae Moon1Department of Computer Science, Kangwon National University, Chuncheon-si, Gangwon-do 24341, KoreaDepartment of Computer Science, Kangwon National University, Chuncheon-si, Gangwon-do 24341, KoreaDistributed stream processing engines (DSPEs) deploy multiple tasks on distributed servers to process data streams in real time. Many DSPEs have provided locality-aware stream partitioning (LSP) methods to reduce network communication costs. However, an even job scheduler provided by DSPEs deploys tasks far away from each other on the distributed servers, which cannot use the LSP properly. In this paper, we propose a Locality/Fairness-aware job scheduler (L/F job scheduler) that considers locality together to solve problems of the even job scheduler that only considers fairness. First, the L/F job scheduler increases cohesion of contiguous tasks that require message transmissions for the locality. At the same time, it reduces coupling of parallel tasks that do not require message transmissions for the fairness. Next, we connect the contiguous tasks into a stream pipeline and evenly deploy stream pipelines to the distributed servers so that the L/F job scheduler achieves high cohesion and low coupling. Finally, we implement the proposed L/F job scheduler in Apache Storm, a representative DSPE, and evaluate it in both synthetic and real-world workloads. Experimental results show that the L/F job scheduler is similar in throughput compared to the even job scheduler, but latency is significantly improved by up to 139.2% for the LSP applications and by up to 140.7% even for the non-LSP applications. The L/F job scheduler also improves latency by 19.58% and 12.13%, respectively, in two real-world workloads. These results indicate that our L/F job scheduler provides superior processing performance for the DSPE applications.https://www.mdpi.com/2079-9292/9/11/1857distributed processingreal-time processingdata streamlocalityfairnessjob scheduling |
| spellingShingle | Siwoon Son Yang-Sae Moon Locality/Fairness-Aware Job Scheduling in Distributed Stream Processing Engines Electronics distributed processing real-time processing data stream locality fairness job scheduling |
| title | Locality/Fairness-Aware Job Scheduling in Distributed Stream Processing Engines |
| title_full | Locality/Fairness-Aware Job Scheduling in Distributed Stream Processing Engines |
| title_fullStr | Locality/Fairness-Aware Job Scheduling in Distributed Stream Processing Engines |
| title_full_unstemmed | Locality/Fairness-Aware Job Scheduling in Distributed Stream Processing Engines |
| title_short | Locality/Fairness-Aware Job Scheduling in Distributed Stream Processing Engines |
| title_sort | locality fairness aware job scheduling in distributed stream processing engines |
| topic | distributed processing real-time processing data stream locality fairness job scheduling |
| url | https://www.mdpi.com/2079-9292/9/11/1857 |
| work_keys_str_mv | AT siwoonson localityfairnessawarejobschedulingindistributedstreamprocessingengines AT yangsaemoon localityfairnessawarejobschedulingindistributedstreamprocessingengines |