SeQual-Stream: approaching stream processing to quality control of NGS datasets
Abstract Background Quality control of DNA sequences is an important data preprocessing step in many genomic analyses. However, all existing parallel tools for this purpose are based on a batch processing model, needing to have the complete genetic dataset before processing can even begin. This limi...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-10-01
|
| Series: | BMC Bioinformatics |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12859-023-05530-7 |
| _version_ | 1797647134115233792 |
|---|---|
| author | Óscar Castellanos-Rodríguez Roberto R. Expósito Juan Touriño |
| author_facet | Óscar Castellanos-Rodríguez Roberto R. Expósito Juan Touriño |
| author_sort | Óscar Castellanos-Rodríguez |
| collection | DOAJ |
| description | Abstract Background Quality control of DNA sequences is an important data preprocessing step in many genomic analyses. However, all existing parallel tools for this purpose are based on a batch processing model, needing to have the complete genetic dataset before processing can even begin. This limitation clearly hinders quality control performance in those scenarios where the dataset must be downloaded from a remote repository and/or copied to a distributed file system for its parallel processing. Results In this paper we present SeQual-Stream, a streaming tool that allows performing multiple quality control operations on genomic datasets in a fast, distributed and scalable way. To do so, our approach relies on the Apache Spark framework and the Hadoop Distributed File System (HDFS) to fully exploit the stream paradigm and accelerate the preprocessing of large datasets as they are being downloaded and/or copied to HDFS. The experimental results have shown significant improvements in the execution times of SeQual-Stream when compared to a batch processing tool with similar quality control features, providing a maximum speedup of 2.7 $$\times$$ × when processing a dataset with more than 250 million DNA sequences, while also demonstrating good scalability features. Conclusion Our solution provides a more scalable and higher performance way to carry out quality control of large genomic datasets by taking advantage of stream processing features. The tool is distributed as free open-source software released under the GNU AGPLv3 license and is publicly available to download at https://github.com/UDC-GAC/SeQual-Stream . |
| first_indexed | 2024-03-11T15:12:58Z |
| format | Article |
| id | doaj.art-9bdf2f125c774fa6a400e7e755aecf09 |
| institution | Directory Open Access Journal |
| issn | 1471-2105 |
| language | English |
| last_indexed | 2024-03-11T15:12:58Z |
| publishDate | 2023-10-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Bioinformatics |
| spelling | doaj.art-9bdf2f125c774fa6a400e7e755aecf092023-10-29T12:38:06ZengBMCBMC Bioinformatics1471-21052023-10-0124112210.1186/s12859-023-05530-7SeQual-Stream: approaching stream processing to quality control of NGS datasetsÓscar Castellanos-Rodríguez0Roberto R. Expósito1Juan Touriño2Universidade da Coruña, CITIC, Computer Architecture Group, Campus de ElviñaUniversidade da Coruña, CITIC, Computer Architecture Group, Campus de ElviñaUniversidade da Coruña, CITIC, Computer Architecture Group, Campus de ElviñaAbstract Background Quality control of DNA sequences is an important data preprocessing step in many genomic analyses. However, all existing parallel tools for this purpose are based on a batch processing model, needing to have the complete genetic dataset before processing can even begin. This limitation clearly hinders quality control performance in those scenarios where the dataset must be downloaded from a remote repository and/or copied to a distributed file system for its parallel processing. Results In this paper we present SeQual-Stream, a streaming tool that allows performing multiple quality control operations on genomic datasets in a fast, distributed and scalable way. To do so, our approach relies on the Apache Spark framework and the Hadoop Distributed File System (HDFS) to fully exploit the stream paradigm and accelerate the preprocessing of large datasets as they are being downloaded and/or copied to HDFS. The experimental results have shown significant improvements in the execution times of SeQual-Stream when compared to a batch processing tool with similar quality control features, providing a maximum speedup of 2.7 $$\times$$ × when processing a dataset with more than 250 million DNA sequences, while also demonstrating good scalability features. Conclusion Our solution provides a more scalable and higher performance way to carry out quality control of large genomic datasets by taking advantage of stream processing features. The tool is distributed as free open-source software released under the GNU AGPLv3 license and is publicly available to download at https://github.com/UDC-GAC/SeQual-Stream .https://doi.org/10.1186/s12859-023-05530-7Quality controlBig dataStream processingApache SparkNext generation sequencing (NGS) |
| spellingShingle | Óscar Castellanos-Rodríguez Roberto R. Expósito Juan Touriño SeQual-Stream: approaching stream processing to quality control of NGS datasets BMC Bioinformatics Quality control Big data Stream processing Apache Spark Next generation sequencing (NGS) |
| title | SeQual-Stream: approaching stream processing to quality control of NGS datasets |
| title_full | SeQual-Stream: approaching stream processing to quality control of NGS datasets |
| title_fullStr | SeQual-Stream: approaching stream processing to quality control of NGS datasets |
| title_full_unstemmed | SeQual-Stream: approaching stream processing to quality control of NGS datasets |
| title_short | SeQual-Stream: approaching stream processing to quality control of NGS datasets |
| title_sort | sequal stream approaching stream processing to quality control of ngs datasets |
| topic | Quality control Big data Stream processing Apache Spark Next generation sequencing (NGS) |
| url | https://doi.org/10.1186/s12859-023-05530-7 |
| work_keys_str_mv | AT oscarcastellanosrodriguez sequalstreamapproachingstreamprocessingtoqualitycontrolofngsdatasets AT robertorexposito sequalstreamapproachingstreamprocessingtoqualitycontrolofngsdatasets AT juantourino sequalstreamapproachingstreamprocessingtoqualitycontrolofngsdatasets |