| Summary: | Currently, the integration of AI models and ML pipelines is complex, requiring ad-hoc developments
that are error-prone and repetitive. This report evaluates three state-of-the-art
frameworks for integrating AI systems: Metaflow, Luigi, and Kedro. These frameworks are
thoroughly analyzed based on their features, execution, and integration capabilities for a given
ML pipeline.
Building upon these state-of-the-art pipelines, an innovative approach of adopting a Function
+ Data flow (FDF) paradigm is implemented. With FDF, functions are adopted as first-class
citizens alongside data within the pipeline. As opposed to Data being the sole currency in
traditional Data flow paradigms, functions may be defined through the operations of a pipeline
and transmitted together with the data.
Subsequently, a novel functionality is introduced through dynamic generation of Functional
Mock-up Units (FMUs) from an ML pipeline. After training a model, regardless of the number
of functions or transformations used, these elements can be serialized and packaged as an FMU.
This approach is an extension of the application of Hybrid AI in smart cities, where AI models
can be abstractly simulated and made reproducible on any platform through FMU/FMI packaging.
For validation, this automated FMU integration for an FDF pipeline is tested across three
case studies using Linear Regression, MLP, and LSTM models. This demonstrates that the
dynamic simulation and integration of AI models can be effectively controlled and scaled with
this approach.
|
|---|