Interoperable and scalable data analysis with microservices: applications in metabolomics
<p><strong>Motivation</strong></p> Developing a robust and performant data analysis workflow that integrates all necessary components whilst still being able to scale over multiple compute nodes is a challenging task. We introduce a generic method based on the microservice ar...
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
Oxford University Press
2019
|
| _version_ | 1826287053913980928 |
|---|---|
| author | Emami Khoonsari, P Moreno, P Bergmann, S Burman, J Capuccini, M Carone, M Cascante, M de Atauri, P Foguet, C Gonzalez-Beltran, A Hankemeier, T Haug, K He, S Herman, S Johnson, D Kale, N Larsson, A Neumann, S Peters, K Pireddu, L Rocca-Serra, P Roger, P Rueedi, R Ruttkies, C Sadawi, N Salek, R Sansone-Assunta, S Schober, D Selivanov, V Thévenot, E van Vliet, M Zanetti, G Steinbeck, C Kultima, K Spjuth, O |
| author_facet | Emami Khoonsari, P Moreno, P Bergmann, S Burman, J Capuccini, M Carone, M Cascante, M de Atauri, P Foguet, C Gonzalez-Beltran, A Hankemeier, T Haug, K He, S Herman, S Johnson, D Kale, N Larsson, A Neumann, S Peters, K Pireddu, L Rocca-Serra, P Roger, P Rueedi, R Ruttkies, C Sadawi, N Salek, R Sansone-Assunta, S Schober, D Selivanov, V Thévenot, E van Vliet, M Zanetti, G Steinbeck, C Kultima, K Spjuth, O |
| author_sort | Emami Khoonsari, P |
| collection | OXFORD |
| description | <p><strong>Motivation</strong></p> Developing a robust and performant data analysis workflow that integrates all necessary components whilst still being able to scale over multiple compute nodes is a challenging task. We introduce a generic method based on the microservice architecture, where software tools are encapsulated as Docker containers that can be connected into scientific workflows and executed using the Kubernetes container orchestrator. <br/> <p><strong>Results</strong></p> We developed a Virtual Research Environment (VRE) which facilitates rapid integration of new tools and developing scalable and interoperable workflows for performing metabolomics data analysis. The environment can be launched on-demand on cloud resources and desktop computers. IT-expertise requirements on the user side are kept to a minimum, and workflows can be re-used effortlessly by any novice user. We validate our method in the field of metabolomics on two mass spectrometry, one nuclear magnetic resonance spectroscopy and one fluxomics study. We showed that the method scales dynamically with increasing availability of computational resources. We demonstrated that the method facilitates interoperability using integration of the major software suites resulting in a turn-key workflow encompassing all steps for mass-spectrometry-based metabolomics including preprocessing, statistics and identification. Microservices is a generic methodology that can serve any scientific discipline and opens up for new types of large-scale integrative science. <br/> <p><strong>Availability and implementation</strong></p> The PhenoMeNal consortium maintains a web portal (https://portal.phenomenal-h2020.eu) providing a GUI for launching the Virtual Research Environment. The GitHub repository https://github.com/phnmnl/ hosts the source code of all projects. |
| first_indexed | 2024-03-07T01:52:52Z |
| format | Journal article |
| id | oxford-uuid:9ab846f0-5110-421f-90d1-baf235d0ede3 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T01:52:52Z |
| publishDate | 2019 |
| publisher | Oxford University Press |
| record_format | dspace |
| spelling | oxford-uuid:9ab846f0-5110-421f-90d1-baf235d0ede32022-03-27T00:23:25ZInteroperable and scalable data analysis with microservices: applications in metabolomicsJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:9ab846f0-5110-421f-90d1-baf235d0ede3EnglishSymplectic Elements at OxfordOxford University Press2019Emami Khoonsari, PMoreno, PBergmann, SBurman, JCapuccini, MCarone, MCascante, Mde Atauri, PFoguet, CGonzalez-Beltran, AHankemeier, THaug, KHe, SHerman, SJohnson, DKale, NLarsson, ANeumann, SPeters, KPireddu, LRocca-Serra, PRoger, PRueedi, RRuttkies, CSadawi, NSalek, RSansone-Assunta, SSchober, DSelivanov, VThévenot, Evan Vliet, MZanetti, GSteinbeck, CKultima, KSpjuth, O<p><strong>Motivation</strong></p> Developing a robust and performant data analysis workflow that integrates all necessary components whilst still being able to scale over multiple compute nodes is a challenging task. We introduce a generic method based on the microservice architecture, where software tools are encapsulated as Docker containers that can be connected into scientific workflows and executed using the Kubernetes container orchestrator. <br/> <p><strong>Results</strong></p> We developed a Virtual Research Environment (VRE) which facilitates rapid integration of new tools and developing scalable and interoperable workflows for performing metabolomics data analysis. The environment can be launched on-demand on cloud resources and desktop computers. IT-expertise requirements on the user side are kept to a minimum, and workflows can be re-used effortlessly by any novice user. We validate our method in the field of metabolomics on two mass spectrometry, one nuclear magnetic resonance spectroscopy and one fluxomics study. We showed that the method scales dynamically with increasing availability of computational resources. We demonstrated that the method facilitates interoperability using integration of the major software suites resulting in a turn-key workflow encompassing all steps for mass-spectrometry-based metabolomics including preprocessing, statistics and identification. Microservices is a generic methodology that can serve any scientific discipline and opens up for new types of large-scale integrative science. <br/> <p><strong>Availability and implementation</strong></p> The PhenoMeNal consortium maintains a web portal (https://portal.phenomenal-h2020.eu) providing a GUI for launching the Virtual Research Environment. The GitHub repository https://github.com/phnmnl/ hosts the source code of all projects. |
| spellingShingle | Emami Khoonsari, P Moreno, P Bergmann, S Burman, J Capuccini, M Carone, M Cascante, M de Atauri, P Foguet, C Gonzalez-Beltran, A Hankemeier, T Haug, K He, S Herman, S Johnson, D Kale, N Larsson, A Neumann, S Peters, K Pireddu, L Rocca-Serra, P Roger, P Rueedi, R Ruttkies, C Sadawi, N Salek, R Sansone-Assunta, S Schober, D Selivanov, V Thévenot, E van Vliet, M Zanetti, G Steinbeck, C Kultima, K Spjuth, O Interoperable and scalable data analysis with microservices: applications in metabolomics |
| title | Interoperable and scalable data analysis with microservices: applications in metabolomics |
| title_full | Interoperable and scalable data analysis with microservices: applications in metabolomics |
| title_fullStr | Interoperable and scalable data analysis with microservices: applications in metabolomics |
| title_full_unstemmed | Interoperable and scalable data analysis with microservices: applications in metabolomics |
| title_short | Interoperable and scalable data analysis with microservices: applications in metabolomics |
| title_sort | interoperable and scalable data analysis with microservices applications in metabolomics |
| work_keys_str_mv | AT emamikhoonsarip interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT morenop interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT bergmanns interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT burmanj interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT capuccinim interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT caronem interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT cascantem interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT deataurip interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT foguetc interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT gonzalezbeltrana interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT hankemeiert interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT haugk interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT hes interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT hermans interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT johnsond interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT kalen interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT larssona interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT neumanns interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT petersk interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT pireddul interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT roccaserrap interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT rogerp interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT rueedir interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT ruttkiesc interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT sadawin interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT salekr interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT sansoneassuntas interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT schoberd interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT selivanovv interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT thevenote interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT vanvlietm interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT zanettig interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT steinbeckc interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT kultimak interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics AT spjutho interoperableandscalabledataanalysiswithmicroservicesapplicationsinmetabolomics |