PMFFRC: a large-scale genomic short reads compression optimizer via memory modeling and redundant clustering
Abstract Background Genomic sequencing reads compressors are essential for balancing high-throughput sequencing short reads generation speed, large-scale genomic data sharing, and infrastructure storage expenditure. However, most existing short reads compressors rarely utilize big-memory systems and...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2023-11-01
|
| Series: | BMC Bioinformatics |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12859-023-05566-9 |
| _version_ | 1797413791997100032 |
|---|---|
| author | Hui Sun Yingfeng Zheng Haonan Xie Huidong Ma Xiaoguang Liu Gang Wang |
| author_facet | Hui Sun Yingfeng Zheng Haonan Xie Huidong Ma Xiaoguang Liu Gang Wang |
| author_sort | Hui Sun |
| collection | DOAJ |
| description | Abstract Background Genomic sequencing reads compressors are essential for balancing high-throughput sequencing short reads generation speed, large-scale genomic data sharing, and infrastructure storage expenditure. However, most existing short reads compressors rarely utilize big-memory systems and duplicative information between diverse sequencing files to achieve a higher compression ratio for conserving reads data storage space. Results We employ compression ratio as the optimization objective and propose a large-scale genomic sequencing short reads data compression optimizer, named PMFFRC, through novelty memory modeling and redundant reads clustering technologies. By cascading PMFFRC, in 982 GB fastq format sequencing data, with 274 GB and 3.3 billion short reads, the state-of-the-art and reference-free compressors HARC, SPRING, Mstcom, and FastqCLS achieve 77.89%, 77.56%, 73.51%, and 29.36% average maximum compression ratio gains, respectively. PMFFRC saves 39.41%, 41.62%, 40.99%, and 20.19% of storage space sizes compared with the four unoptimized compressors. Conclusions PMFFRC rational usage big-memory of compression server, effectively saving the sequencing reads data storage space sizes, which relieves the basic storage facilities costs and community sharing transmitting overhead. Our work furnishes a novel solution for improving sequencing reads compression and saving storage space. The proposed PMFFRC algorithm is packaged in a same-name Linux toolkit, available un-limited at https://github.com/fahaihi/PMFFRC . |
| first_indexed | 2024-03-09T05:23:59Z |
| format | Article |
| id | doaj.art-1bc98b6a90b54fee85ab4eafd3217a09 |
| institution | Directory Open Access Journal |
| issn | 1471-2105 |
| language | English |
| last_indexed | 2024-03-09T05:23:59Z |
| publishDate | 2023-11-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Bioinformatics |
| spelling | doaj.art-1bc98b6a90b54fee85ab4eafd3217a092023-12-03T12:38:19ZengBMCBMC Bioinformatics1471-21052023-11-0124111510.1186/s12859-023-05566-9PMFFRC: a large-scale genomic short reads compression optimizer via memory modeling and redundant clusteringHui Sun0Yingfeng Zheng1Haonan Xie2Huidong Ma3Xiaoguang Liu4Gang Wang5Nankai-Baidu Joint Laboratory, College of Computer Science, Nankai UniversityNankai-Baidu Joint Laboratory, College of Computer Science, Nankai UniversityInstitute of Artificial Intelligence, School of Electrical Engineering, Guangxi UniversityNankai-Baidu Joint Laboratory, College of Computer Science, Nankai UniversityNankai-Baidu Joint Laboratory, College of Computer Science, Nankai UniversityNankai-Baidu Joint Laboratory, College of Computer Science, Nankai UniversityAbstract Background Genomic sequencing reads compressors are essential for balancing high-throughput sequencing short reads generation speed, large-scale genomic data sharing, and infrastructure storage expenditure. However, most existing short reads compressors rarely utilize big-memory systems and duplicative information between diverse sequencing files to achieve a higher compression ratio for conserving reads data storage space. Results We employ compression ratio as the optimization objective and propose a large-scale genomic sequencing short reads data compression optimizer, named PMFFRC, through novelty memory modeling and redundant reads clustering technologies. By cascading PMFFRC, in 982 GB fastq format sequencing data, with 274 GB and 3.3 billion short reads, the state-of-the-art and reference-free compressors HARC, SPRING, Mstcom, and FastqCLS achieve 77.89%, 77.56%, 73.51%, and 29.36% average maximum compression ratio gains, respectively. PMFFRC saves 39.41%, 41.62%, 40.99%, and 20.19% of storage space sizes compared with the four unoptimized compressors. Conclusions PMFFRC rational usage big-memory of compression server, effectively saving the sequencing reads data storage space sizes, which relieves the basic storage facilities costs and community sharing transmitting overhead. Our work furnishes a novel solution for improving sequencing reads compression and saving storage space. The proposed PMFFRC algorithm is packaged in a same-name Linux toolkit, available un-limited at https://github.com/fahaihi/PMFFRC .https://doi.org/10.1186/s12859-023-05566-9Short reads dataData compressionFastqParallel algorithm |
| spellingShingle | Hui Sun Yingfeng Zheng Haonan Xie Huidong Ma Xiaoguang Liu Gang Wang PMFFRC: a large-scale genomic short reads compression optimizer via memory modeling and redundant clustering BMC Bioinformatics Short reads data Data compression Fastq Parallel algorithm |
| title | PMFFRC: a large-scale genomic short reads compression optimizer via memory modeling and redundant clustering |
| title_full | PMFFRC: a large-scale genomic short reads compression optimizer via memory modeling and redundant clustering |
| title_fullStr | PMFFRC: a large-scale genomic short reads compression optimizer via memory modeling and redundant clustering |
| title_full_unstemmed | PMFFRC: a large-scale genomic short reads compression optimizer via memory modeling and redundant clustering |
| title_short | PMFFRC: a large-scale genomic short reads compression optimizer via memory modeling and redundant clustering |
| title_sort | pmffrc a large scale genomic short reads compression optimizer via memory modeling and redundant clustering |
| topic | Short reads data Data compression Fastq Parallel algorithm |
| url | https://doi.org/10.1186/s12859-023-05566-9 |
| work_keys_str_mv | AT huisun pmffrcalargescalegenomicshortreadscompressionoptimizerviamemorymodelingandredundantclustering AT yingfengzheng pmffrcalargescalegenomicshortreadscompressionoptimizerviamemorymodelingandredundantclustering AT haonanxie pmffrcalargescalegenomicshortreadscompressionoptimizerviamemorymodelingandredundantclustering AT huidongma pmffrcalargescalegenomicshortreadscompressionoptimizerviamemorymodelingandredundantclustering AT xiaoguangliu pmffrcalargescalegenomicshortreadscompressionoptimizerviamemorymodelingandredundantclustering AT gangwang pmffrcalargescalegenomicshortreadscompressionoptimizerviamemorymodelingandredundantclustering |