Efficient parallel processing and fault tolerance in a streaming join system
Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2018.
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | eng |
| Published: |
Massachusetts Institute of Technology
2018
|
| Subjects: | |
| Online Access: | http://hdl.handle.net/1721.1/119578 |
| _version_ | 1826189528250974208 |
|---|---|
| author | Mohan, Varun (Varun K.) |
| author2 | Sam Madden. |
| author_facet | Sam Madden. Mohan, Varun (Varun K.) |
| author_sort | Mohan, Varun (Varun K.) |
| collection | MIT |
| description | Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2018. |
| first_indexed | 2024-09-23T08:16:16Z |
| format | Thesis |
| id | mit-1721.1/119578 |
| institution | Massachusetts Institute of Technology |
| language | eng |
| last_indexed | 2024-09-23T08:16:16Z |
| publishDate | 2018 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | mit-1721.1/1195782019-04-09T17:42:36Z Efficient parallel processing and fault tolerance in a streaming join system Mohan, Varun (Varun K.) Sam Madden. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science. Electrical Engineering and Computer Science. Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2018. This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections. Cataloged from student-submitted PDF version of thesis. Includes bibliographical references (pages 91-92). Stream joins are an important component of stream processing, as they provide an online mechanism to efficiently combine multiple streams of data. In this thesis, we consider the RiverJoin system, which presents a general method for performing stream joins without relying on the ordering and timing of records in a stream. RiverJoin implements a useful set of primitives, called operands, that provide persistence and caching, allowing full-history stream joins. RiverJoin implements critical performance optimizations like batching and automatic parallelization, having performance comparable to lossy windowed joins while maintaining its strong correctness semantics. The individual contributions of the thesis come from further extending RiverJoin by adding functionality required for an end-to-end distributed system, specifically sharding and fault tolerance. In particular, we are able to leverage RiverJoin's automatic parallelization mechanism to provide a data stream sharding interface that allows for close-to linear speedup in stream joins, allowing arbitrarily fast joins. Stream processing systems require high throughput and low latency in order to provide real-time guarantees. Traditional methods of periodic global snapshots provide fault recovery at the cost of pausing the entire system during the snapshot process and persisting records in transit, leading to large snapshot sizes. In this thesis, we present a low-latency mechanism to snapshot a RiverJoin topology asynchronously, providing consistent system recovery and exactly-once delivery semantics between its operands. The method builds upon the idea of Asynchronous Barrier Snapshotting (ABS) and a novel asynchronous backup strategy, called fork-backup, that can snapshot an operand with gigabytes of state in milliseconds. Our results show that we can take snapshots without affecting system latency and throughput even with frequent snapshotting. We are also able to issue and complete a global snapshot for a complex RiverJoin topology in around 50ms, faster than a Java GC pause, and can recover from failure in less than a second. by Varun Mohan. M. Eng. 2018-12-11T20:40:50Z 2018-12-11T20:40:50Z 2018 2018 Thesis http://hdl.handle.net/1721.1/119578 1076359309 eng MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission. http://dspace.mit.edu/handle/1721.1/7582 92 pages application/pdf Massachusetts Institute of Technology |
| spellingShingle | Electrical Engineering and Computer Science. Mohan, Varun (Varun K.) Efficient parallel processing and fault tolerance in a streaming join system |
| title | Efficient parallel processing and fault tolerance in a streaming join system |
| title_full | Efficient parallel processing and fault tolerance in a streaming join system |
| title_fullStr | Efficient parallel processing and fault tolerance in a streaming join system |
| title_full_unstemmed | Efficient parallel processing and fault tolerance in a streaming join system |
| title_short | Efficient parallel processing and fault tolerance in a streaming join system |
| title_sort | efficient parallel processing and fault tolerance in a streaming join system |
| topic | Electrical Engineering and Computer Science. |
| url | http://hdl.handle.net/1721.1/119578 |
| work_keys_str_mv | AT mohanvarunvarunk efficientparallelprocessingandfaulttoleranceinastreamingjoinsystem |