Architectural Concept and Evaluation of a Framework for the Efficient Automation of Computational Scientific Workflows: An Energy Systems Analysis Example
Scientists and engineers involved in the design of complex system solutions use computational workflows for their evaluations. Along with growing system complexity, the complexity of these workflows also increases. Without integration tools, scientists and engineers are often highly concerned with h...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2019-02-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/9/4/728 |
| _version_ | 1818136882053120000 |
|---|---|
| author | Jianlei Liu Eric Braun Clemens Düpmeier Patrick Kuckertz D. Severin Ryberg Martin Robinius Detlef Stolten Veit Hagenmeyer |
| author_facet | Jianlei Liu Eric Braun Clemens Düpmeier Patrick Kuckertz D. Severin Ryberg Martin Robinius Detlef Stolten Veit Hagenmeyer |
| author_sort | Jianlei Liu |
| collection | DOAJ |
| description | Scientists and engineers involved in the design of complex system solutions use computational workflows for their evaluations. Along with growing system complexity, the complexity of these workflows also increases. Without integration tools, scientists and engineers are often highly concerned with how to integrate software tools and model sets, which hinders their original research or engineering aims. Therefore, a new framework for streamlining the creation and usage of automated computational workflows is introduced in the present article. It uses state-of-the-art technologies for automation (e.g., container-automation) and coordination (e.g., distributed message oriented middleware), and a microservice-based architecture for novel distributed process execution and coordination. It also supports co-simulations as part of larger workflows including additional auxiliary computational tasks, e.g., forecasting or data transformation. Using Apache NiFi, an easy-to-use web interface is provided to create, run and control workflows without the need to be concerned with the underlying computing infrastructure. Initial framework testing via the implementation of a real-world workflow underpins promising performance in the realms of parallelizability, low overheads and reliable coordination. |
| first_indexed | 2024-12-11T09:47:28Z |
| format | Article |
| id | doaj.art-e98ffa9d54c94493a3c0f8fe30c3d65a |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-12-11T09:47:28Z |
| publishDate | 2019-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-e98ffa9d54c94493a3c0f8fe30c3d65a2022-12-22T01:12:31ZengMDPI AGApplied Sciences2076-34172019-02-019472810.3390/app9040728app9040728Architectural Concept and Evaluation of a Framework for the Efficient Automation of Computational Scientific Workflows: An Energy Systems Analysis ExampleJianlei Liu0Eric Braun1Clemens Düpmeier2Patrick Kuckertz3D. Severin Ryberg4Martin Robinius5Detlef Stolten6Veit Hagenmeyer7Institute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, GermanyInstitute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, GermanyInstitute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, GermanyInstitute of Electrochemical Process Engineering (IEK-3), Forschungszentrum Jülich GmbH, Wilhelm-Johnen-Straße, 52428 Jülich, GermanyInstitute of Electrochemical Process Engineering (IEK-3), Forschungszentrum Jülich GmbH, Wilhelm-Johnen-Straße, 52428 Jülich, GermanyInstitute of Electrochemical Process Engineering (IEK-3), Forschungszentrum Jülich GmbH, Wilhelm-Johnen-Straße, 52428 Jülich, GermanyInstitute of Electrochemical Process Engineering (IEK-3), Forschungszentrum Jülich GmbH, Wilhelm-Johnen-Straße, 52428 Jülich, GermanyInstitute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, GermanyScientists and engineers involved in the design of complex system solutions use computational workflows for their evaluations. Along with growing system complexity, the complexity of these workflows also increases. Without integration tools, scientists and engineers are often highly concerned with how to integrate software tools and model sets, which hinders their original research or engineering aims. Therefore, a new framework for streamlining the creation and usage of automated computational workflows is introduced in the present article. It uses state-of-the-art technologies for automation (e.g., container-automation) and coordination (e.g., distributed message oriented middleware), and a microservice-based architecture for novel distributed process execution and coordination. It also supports co-simulations as part of larger workflows including additional auxiliary computational tasks, e.g., forecasting or data transformation. Using Apache NiFi, an easy-to-use web interface is provided to create, run and control workflows without the need to be concerned with the underlying computing infrastructure. Initial framework testing via the implementation of a real-world workflow underpins promising performance in the realms of parallelizability, low overheads and reliable coordination.https://www.mdpi.com/2076-3417/9/4/728automationmicroservicescomputing clusterenergy systemdata processingco-simulationparallelizationcoordination |
| spellingShingle | Jianlei Liu Eric Braun Clemens Düpmeier Patrick Kuckertz D. Severin Ryberg Martin Robinius Detlef Stolten Veit Hagenmeyer Architectural Concept and Evaluation of a Framework for the Efficient Automation of Computational Scientific Workflows: An Energy Systems Analysis Example Applied Sciences automation microservices computing cluster energy system data processing co-simulation parallelization coordination |
| title | Architectural Concept and Evaluation of a Framework for the Efficient Automation of Computational Scientific Workflows: An Energy Systems Analysis Example |
| title_full | Architectural Concept and Evaluation of a Framework for the Efficient Automation of Computational Scientific Workflows: An Energy Systems Analysis Example |
| title_fullStr | Architectural Concept and Evaluation of a Framework for the Efficient Automation of Computational Scientific Workflows: An Energy Systems Analysis Example |
| title_full_unstemmed | Architectural Concept and Evaluation of a Framework for the Efficient Automation of Computational Scientific Workflows: An Energy Systems Analysis Example |
| title_short | Architectural Concept and Evaluation of a Framework for the Efficient Automation of Computational Scientific Workflows: An Energy Systems Analysis Example |
| title_sort | architectural concept and evaluation of a framework for the efficient automation of computational scientific workflows an energy systems analysis example |
| topic | automation microservices computing cluster energy system data processing co-simulation parallelization coordination |
| url | https://www.mdpi.com/2076-3417/9/4/728 |
| work_keys_str_mv | AT jianleiliu architecturalconceptandevaluationofaframeworkfortheefficientautomationofcomputationalscientificworkflowsanenergysystemsanalysisexample AT ericbraun architecturalconceptandevaluationofaframeworkfortheefficientautomationofcomputationalscientificworkflowsanenergysystemsanalysisexample AT clemensdupmeier architecturalconceptandevaluationofaframeworkfortheefficientautomationofcomputationalscientificworkflowsanenergysystemsanalysisexample AT patrickkuckertz architecturalconceptandevaluationofaframeworkfortheefficientautomationofcomputationalscientificworkflowsanenergysystemsanalysisexample AT dseverinryberg architecturalconceptandevaluationofaframeworkfortheefficientautomationofcomputationalscientificworkflowsanenergysystemsanalysisexample AT martinrobinius architecturalconceptandevaluationofaframeworkfortheefficientautomationofcomputationalscientificworkflowsanenergysystemsanalysisexample AT detlefstolten architecturalconceptandevaluationofaframeworkfortheefficientautomationofcomputationalscientificworkflowsanenergysystemsanalysisexample AT veithagenmeyer architecturalconceptandevaluationofaframeworkfortheefficientautomationofcomputationalscientificworkflowsanenergysystemsanalysisexample |