State-of-the-art AI integration methods and frameworks
Currently, the integration of AI models and ML pipelines is complex, requiring ad-hoc developments that are error-prone and repetitive. This report evaluates three state-of-the-art frameworks for integrating AI systems: Metaflow, Luigi, and Kedro. These frameworks are thoroughly analyzed based on...
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Final Year Project (FYP) |
| Language: | English |
| Published: |
Nanyang Technological University
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/175091 |
| _version_ | 1811683704110579712 |
|---|---|
| author | Samson, Sherwin |
| author2 | Arvind Easwaran |
| author_facet | Arvind Easwaran Samson, Sherwin |
| author_sort | Samson, Sherwin |
| collection | NTU |
| description | Currently, the integration of AI models and ML pipelines is complex, requiring ad-hoc developments
that are error-prone and repetitive. This report evaluates three state-of-the-art
frameworks for integrating AI systems: Metaflow, Luigi, and Kedro. These frameworks are
thoroughly analyzed based on their features, execution, and integration capabilities for a given
ML pipeline.
Building upon these state-of-the-art pipelines, an innovative approach of adopting a Function
+ Data flow (FDF) paradigm is implemented. With FDF, functions are adopted as first-class
citizens alongside data within the pipeline. As opposed to Data being the sole currency in
traditional Data flow paradigms, functions may be defined through the operations of a pipeline
and transmitted together with the data.
Subsequently, a novel functionality is introduced through dynamic generation of Functional
Mock-up Units (FMUs) from an ML pipeline. After training a model, regardless of the number
of functions or transformations used, these elements can be serialized and packaged as an FMU.
This approach is an extension of the application of Hybrid AI in smart cities, where AI models
can be abstractly simulated and made reproducible on any platform through FMU/FMI packaging.
For validation, this automated FMU integration for an FDF pipeline is tested across three
case studies using Linear Regression, MLP, and LSTM models. This demonstrates that the
dynamic simulation and integration of AI models can be effectively controlled and scaled with
this approach. |
| first_indexed | 2024-10-01T04:16:58Z |
| format | Final Year Project (FYP) |
| id | ntu-10356/175091 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2024-10-01T04:16:58Z |
| publishDate | 2024 |
| publisher | Nanyang Technological University |
| record_format | dspace |
| spelling | ntu-10356/1750912024-04-19T15:42:21Z State-of-the-art AI integration methods and frameworks Samson, Sherwin Arvind Easwaran School of Computer Science and Engineering arvinde@ntu.edu.sg Computer and Information Science Kedro Metaflow Luigi FMU Currently, the integration of AI models and ML pipelines is complex, requiring ad-hoc developments that are error-prone and repetitive. This report evaluates three state-of-the-art frameworks for integrating AI systems: Metaflow, Luigi, and Kedro. These frameworks are thoroughly analyzed based on their features, execution, and integration capabilities for a given ML pipeline. Building upon these state-of-the-art pipelines, an innovative approach of adopting a Function + Data flow (FDF) paradigm is implemented. With FDF, functions are adopted as first-class citizens alongside data within the pipeline. As opposed to Data being the sole currency in traditional Data flow paradigms, functions may be defined through the operations of a pipeline and transmitted together with the data. Subsequently, a novel functionality is introduced through dynamic generation of Functional Mock-up Units (FMUs) from an ML pipeline. After training a model, regardless of the number of functions or transformations used, these elements can be serialized and packaged as an FMU. This approach is an extension of the application of Hybrid AI in smart cities, where AI models can be abstractly simulated and made reproducible on any platform through FMU/FMI packaging. For validation, this automated FMU integration for an FDF pipeline is tested across three case studies using Linear Regression, MLP, and LSTM models. This demonstrates that the dynamic simulation and integration of AI models can be effectively controlled and scaled with this approach. Bachelor's degree 2024-04-19T05:18:36Z 2024-04-19T05:18:36Z 2024 Final Year Project (FYP) Samson, S. (2024). State-of-the-art AI integration methods and frameworks. Final Year Project (FYP), Nanyang Technological University, Singapore. https://hdl.handle.net/10356/175091 https://hdl.handle.net/10356/175091 en application/pdf Nanyang Technological University |
| spellingShingle | Computer and Information Science Kedro Metaflow Luigi FMU Samson, Sherwin State-of-the-art AI integration methods and frameworks |
| title | State-of-the-art AI integration methods and frameworks |
| title_full | State-of-the-art AI integration methods and frameworks |
| title_fullStr | State-of-the-art AI integration methods and frameworks |
| title_full_unstemmed | State-of-the-art AI integration methods and frameworks |
| title_short | State-of-the-art AI integration methods and frameworks |
| title_sort | state of the art ai integration methods and frameworks |
| topic | Computer and Information Science Kedro Metaflow Luigi FMU |
| url | https://hdl.handle.net/10356/175091 |
| work_keys_str_mv | AT samsonsherwin stateoftheartaiintegrationmethodsandframeworks |