A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment
The field of cloud computing has witnessed tremendous progress, with commercial cloud providers offering powerful distributed infrastructures to small and medium enterprises (SMEs) through their revolutionary pay-as-you-go model. Simultaneously, the rise of containers has empowered virtualisation, p...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-11-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/23/12635 |
| _version_ | 1827592478701649920 |
|---|---|
| author | Thalita Vergilio Ah-Lian Kor Duncan Mullier |
| author_facet | Thalita Vergilio Ah-Lian Kor Duncan Mullier |
| author_sort | Thalita Vergilio |
| collection | DOAJ |
| description | The field of cloud computing has witnessed tremendous progress, with commercial cloud providers offering powerful distributed infrastructures to small and medium enterprises (SMEs) through their revolutionary pay-as-you-go model. Simultaneously, the rise of containers has empowered virtualisation, providing orchestration technologies for the deployment and management of large-scale distributed systems across different geolocations and providers. Big data is another research area which has developed at an extraordinary pace as industries endeavour to discover innovative and effective ways of processing large volumes of structured, semi-structured, and unstructured data. This research aims to integrate the latest advances within the fields of cloud computing, virtualisation, and big data for a systematic approach to stream processing. The novel contributions of this research are: (1) MC-BDP, a reference architecture for big data stream processing in a containerised, multi-cloud environment; (2) a case study conducted with the Estates and Sustainability departments at Leeds Beckett University to evaluate an MC-BDP prototype within the context of energy efficiency for smart buildings. The study found that MC-BDP is scalable and fault-tolerant across cloud environments, key attributes for SMEs managing resources under budgetary constraints. Additionally, our experiments on technology agnosticism and container co-location provide new insights into resource utilisation, cost implications, and optimal deployment strategies in cloud-based big data streaming, offering valuable guidelines for practitioners in the field. |
| first_indexed | 2024-03-09T01:56:05Z |
| format | Article |
| id | doaj.art-c5335cfac452430abb90335c334243ae |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-09T01:56:05Z |
| publishDate | 2023-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-c5335cfac452430abb90335c334243ae2023-12-08T15:11:14ZengMDPI AGApplied Sciences2076-34172023-11-0113231263510.3390/app132312635A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud EnvironmentThalita Vergilio0Ah-Lian Kor1Duncan Mullier2School of Built Environment, Engineering and Computing, Leeds Beckett University, Leeds LS6 3QS, UKSchool of Built Environment, Engineering and Computing, Leeds Beckett University, Leeds LS6 3QS, UKSchool of Built Environment, Engineering and Computing, Leeds Beckett University, Leeds LS6 3QS, UKThe field of cloud computing has witnessed tremendous progress, with commercial cloud providers offering powerful distributed infrastructures to small and medium enterprises (SMEs) through their revolutionary pay-as-you-go model. Simultaneously, the rise of containers has empowered virtualisation, providing orchestration technologies for the deployment and management of large-scale distributed systems across different geolocations and providers. Big data is another research area which has developed at an extraordinary pace as industries endeavour to discover innovative and effective ways of processing large volumes of structured, semi-structured, and unstructured data. This research aims to integrate the latest advances within the fields of cloud computing, virtualisation, and big data for a systematic approach to stream processing. The novel contributions of this research are: (1) MC-BDP, a reference architecture for big data stream processing in a containerised, multi-cloud environment; (2) a case study conducted with the Estates and Sustainability departments at Leeds Beckett University to evaluate an MC-BDP prototype within the context of energy efficiency for smart buildings. The study found that MC-BDP is scalable and fault-tolerant across cloud environments, key attributes for SMEs managing resources under budgetary constraints. Additionally, our experiments on technology agnosticism and container co-location provide new insights into resource utilisation, cost implications, and optimal deployment strategies in cloud-based big data streaming, offering valuable guidelines for practitioners in the field.https://www.mdpi.com/2076-3417/13/23/12635multi-cloudbig datacontainersreference architecturestreamvendor lock-in |
| spellingShingle | Thalita Vergilio Ah-Lian Kor Duncan Mullier A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment Applied Sciences multi-cloud big data containers reference architecture stream vendor lock-in |
| title | A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment |
| title_full | A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment |
| title_fullStr | A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment |
| title_full_unstemmed | A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment |
| title_short | A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment |
| title_sort | unified vendor agnostic solution for big data stream processing in a multi cloud environment |
| topic | multi-cloud big data containers reference architecture stream vendor lock-in |
| url | https://www.mdpi.com/2076-3417/13/23/12635 |
| work_keys_str_mv | AT thalitavergilio aunifiedvendoragnosticsolutionforbigdatastreamprocessinginamulticloudenvironment AT ahliankor aunifiedvendoragnosticsolutionforbigdatastreamprocessinginamulticloudenvironment AT duncanmullier aunifiedvendoragnosticsolutionforbigdatastreamprocessinginamulticloudenvironment AT thalitavergilio unifiedvendoragnosticsolutionforbigdatastreamprocessinginamulticloudenvironment AT ahliankor unifiedvendoragnosticsolutionforbigdatastreamprocessinginamulticloudenvironment AT duncanmullier unifiedvendoragnosticsolutionforbigdatastreamprocessinginamulticloudenvironment |