Automated metadata-based detection of the data product consumers in data mesh
Object is distributed data mesh. The article reviews main types of data product consumers. There are a lot of them depending on technology stack, architecture, components, purpose, etc. At the same time, it’s critical to keep inventory of all consumers of data products in distributed data mesh platf...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Igor Sikorsky Kyiv Polytechnic Institute
2023-12-01
|
| Series: | Adaptivni Sistemi Avtomatičnogo Upravlinnâ |
| Subjects: | |
| Online Access: | http://asac.kpi.ua/article/view/292261 |
| _version_ | 1797351360642940928 |
|---|---|
| author | Y. Vlasiuk V. Onyshchenko |
| author_facet | Y. Vlasiuk V. Onyshchenko |
| author_sort | Y. Vlasiuk |
| collection | DOAJ |
| description | Object is distributed data mesh. The article reviews main types of data product consumers. There are a lot of them depending on technology stack, architecture, components, purpose, etc. At the same time, it’s critical to keep inventory of all consumers of data products in distributed data mesh platform as it allows to notify them about changes in data products, collect and keep requirements for the data product during its lifecycle. The aim is to collect list of data products consumers automatically. It allows to reduce manual effort which will decrease operational cost of data mesh support. To achieve the goal a couple of automated methods for detecting data product consumers were analyzed, in particular, API-based method, event-based method, metadata-based method. After deep investigation of mentioned methods, it was discovered that API-based and event-based methods require significant changes of data mesh platform architecture by adding new software components. At the same time, metadata-based method doesn’t have such limitations and provide acceptable results. A couple of approaches of adopting metadata-based method of data product consumers identification were proposed for different technologies and components.
Ref. 7, pic. 4 |
| first_indexed | 2024-03-08T12:59:24Z |
| format | Article |
| id | doaj.art-6b5ac25b9b994a63b0b742cf646c67b4 |
| institution | Directory Open Access Journal |
| issn | 1560-8956 2522-9575 |
| language | English |
| last_indexed | 2024-03-08T12:59:24Z |
| publishDate | 2023-12-01 |
| publisher | Igor Sikorsky Kyiv Polytechnic Institute |
| record_format | Article |
| series | Adaptivni Sistemi Avtomatičnogo Upravlinnâ |
| spelling | doaj.art-6b5ac25b9b994a63b0b742cf646c67b42024-01-19T12:06:17ZengIgor Sikorsky Kyiv Polytechnic InstituteAdaptivni Sistemi Avtomatičnogo Upravlinnâ1560-89562522-95752023-12-0124311312310.20535/1560-8956.43.2023.292261330548Automated metadata-based detection of the data product consumers in data meshY. Vlasiuk0V. Onyshchenko1Igor Sikorsky Kyiv Polytechnic InstituteIgor Sikorsky Kyiv Polytechnic InstituteObject is distributed data mesh. The article reviews main types of data product consumers. There are a lot of them depending on technology stack, architecture, components, purpose, etc. At the same time, it’s critical to keep inventory of all consumers of data products in distributed data mesh platform as it allows to notify them about changes in data products, collect and keep requirements for the data product during its lifecycle. The aim is to collect list of data products consumers automatically. It allows to reduce manual effort which will decrease operational cost of data mesh support. To achieve the goal a couple of automated methods for detecting data product consumers were analyzed, in particular, API-based method, event-based method, metadata-based method. After deep investigation of mentioned methods, it was discovered that API-based and event-based methods require significant changes of data mesh platform architecture by adding new software components. At the same time, metadata-based method doesn’t have such limitations and provide acceptable results. A couple of approaches of adopting metadata-based method of data product consumers identification were proposed for different technologies and components. Ref. 7, pic. 4http://asac.kpi.ua/article/view/292261artificial intelligencedata meshdata productdata lineage |
| spellingShingle | Y. Vlasiuk V. Onyshchenko Automated metadata-based detection of the data product consumers in data mesh Adaptivni Sistemi Avtomatičnogo Upravlinnâ artificial intelligence data mesh data product data lineage |
| title | Automated metadata-based detection of the data product consumers in data mesh |
| title_full | Automated metadata-based detection of the data product consumers in data mesh |
| title_fullStr | Automated metadata-based detection of the data product consumers in data mesh |
| title_full_unstemmed | Automated metadata-based detection of the data product consumers in data mesh |
| title_short | Automated metadata-based detection of the data product consumers in data mesh |
| title_sort | automated metadata based detection of the data product consumers in data mesh |
| topic | artificial intelligence data mesh data product data lineage |
| url | http://asac.kpi.ua/article/view/292261 |
| work_keys_str_mv | AT yvlasiuk automatedmetadatabaseddetectionofthedataproductconsumersindatamesh AT vonyshchenko automatedmetadatabaseddetectionofthedataproductconsumersindatamesh |