Local Processing of Massive Databases with R: A National Analysis of a Brazilian Social Programme
The analysis of massive databases is a key issue for most applications today and the use of parallel computing techniques is one of the suitable approaches for that. Apache Spark is a widely employed tool within this context, aiming at processing large amounts of data in a distributed way. For the S...
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-10-01
|
| Series: | Stats |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2571-905X/3/4/28 |
| _version_ | 1827704061216948224 |
|---|---|
| author | Hellen Paz Mateus Maia Fernando Moraes Ricardo Lustosa Lilia Costa Samuel Macêdo Marcos E. Barreto Anderson Ara |
| author_facet | Hellen Paz Mateus Maia Fernando Moraes Ricardo Lustosa Lilia Costa Samuel Macêdo Marcos E. Barreto Anderson Ara |
| author_sort | Hellen Paz |
| collection | DOAJ |
| description | The analysis of massive databases is a key issue for most applications today and the use of parallel computing techniques is one of the suitable approaches for that. Apache Spark is a widely employed tool within this context, aiming at processing large amounts of data in a distributed way. For the Statistics community, R is one of the preferred tools. Despite its growth in the last years, it still has limitations for processing large volumes of data in single local machines. In general, the data analysis community has difficulty to handle a massive amount of data on local machines, often requiring high-performance computing servers. One way to perform statistical analyzes over massive databases is combining both tools (Spark and R) via the sparklyr package, which allows for an R application to use Spark. This paper presents an analysis of Brazilian public data from the Bolsa Família Programme (BFP—conditional cash transfer), comprising a large data set with 1.26 billion observations. Our goal was to understand how this social program acts in different cities, as well as to identify potentially important variables reflecting its utilization rate. Statistical modeling was performed using random forest to predict the utilization rated of BFP. Variable selection was performed through a recent method based on the importance and interpretation of variables in the random forest model. Among the 89 variables initially considered, the final model presented a high predictive performance capacity with 17 selected variables, as well as indicated high importance of some variables for the observed utilization rate in income, education, job informality, and inactive youth, namely: family income, education, occupation and density of people in the homes. In this work, using a local machine, we highlighted the potential of aggregating Spark and R for analysis of a large database of 111.6 GB. This can serve as proof of concept or reference for other similar works within the Statistics community, as well as our case study can provide important evidence for further analysis of this important social support programme. |
| first_indexed | 2024-03-10T15:31:30Z |
| format | Article |
| id | doaj.art-5b94d1c539714cb18c120b2189f3589d |
| institution | Directory Open Access Journal |
| issn | 2571-905X |
| language | English |
| last_indexed | 2024-03-10T15:31:30Z |
| publishDate | 2020-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Stats |
| spelling | doaj.art-5b94d1c539714cb18c120b2189f3589d2023-11-20T17:36:04ZengMDPI AGStats2571-905X2020-10-013444446410.3390/stats3040028Local Processing of Massive Databases with R: A National Analysis of a Brazilian Social ProgrammeHellen Paz0Mateus Maia1Fernando Moraes2Ricardo Lustosa3Lilia Costa4Samuel Macêdo5Marcos E. Barreto6Anderson Ara7Statistics Department, Federal University of Bahia, 40.170-110 Salvador-BA, BrazilStatistics Department, Federal University of Bahia, 40.170-110 Salvador-BA, BrazilStatistics Department, Federal University of Bahia, 40.170-110 Salvador-BA, BrazilInstitute of Collective Health, Federal University of Bahia, 40.110-040 Salvador-BA, BrazilStatistics Department, Federal University of Bahia, 40.170-110 Salvador-BA, BrazilDepartment of Natural Sciences and Mathematics, Federal Institute of Pernambuco, 50.740-545 Recife-PE, BrazilComputer Science Department, Federal University of Bahia, 40.170-110 Salvador-BA, BrazilStatistics Department, Federal University of Bahia, 40.170-110 Salvador-BA, BrazilThe analysis of massive databases is a key issue for most applications today and the use of parallel computing techniques is one of the suitable approaches for that. Apache Spark is a widely employed tool within this context, aiming at processing large amounts of data in a distributed way. For the Statistics community, R is one of the preferred tools. Despite its growth in the last years, it still has limitations for processing large volumes of data in single local machines. In general, the data analysis community has difficulty to handle a massive amount of data on local machines, often requiring high-performance computing servers. One way to perform statistical analyzes over massive databases is combining both tools (Spark and R) via the sparklyr package, which allows for an R application to use Spark. This paper presents an analysis of Brazilian public data from the Bolsa Família Programme (BFP—conditional cash transfer), comprising a large data set with 1.26 billion observations. Our goal was to understand how this social program acts in different cities, as well as to identify potentially important variables reflecting its utilization rate. Statistical modeling was performed using random forest to predict the utilization rated of BFP. Variable selection was performed through a recent method based on the importance and interpretation of variables in the random forest model. Among the 89 variables initially considered, the final model presented a high predictive performance capacity with 17 selected variables, as well as indicated high importance of some variables for the observed utilization rate in income, education, job informality, and inactive youth, namely: family income, education, occupation and density of people in the homes. In this work, using a local machine, we highlighted the potential of aggregating Spark and R for analysis of a large database of 111.6 GB. This can serve as proof of concept or reference for other similar works within the Statistics community, as well as our case study can provide important evidence for further analysis of this important social support programme.https://www.mdpi.com/2571-905X/3/4/28big datamassive databasesimpact evaluationsparklyrBolsa Família |
| spellingShingle | Hellen Paz Mateus Maia Fernando Moraes Ricardo Lustosa Lilia Costa Samuel Macêdo Marcos E. Barreto Anderson Ara Local Processing of Massive Databases with R: A National Analysis of a Brazilian Social Programme Stats big data massive databases impact evaluation sparklyr Bolsa Família |
| title | Local Processing of Massive Databases with R: A National Analysis of a Brazilian Social Programme |
| title_full | Local Processing of Massive Databases with R: A National Analysis of a Brazilian Social Programme |
| title_fullStr | Local Processing of Massive Databases with R: A National Analysis of a Brazilian Social Programme |
| title_full_unstemmed | Local Processing of Massive Databases with R: A National Analysis of a Brazilian Social Programme |
| title_short | Local Processing of Massive Databases with R: A National Analysis of a Brazilian Social Programme |
| title_sort | local processing of massive databases with r a national analysis of a brazilian social programme |
| topic | big data massive databases impact evaluation sparklyr Bolsa Família |
| url | https://www.mdpi.com/2571-905X/3/4/28 |
| work_keys_str_mv | AT hellenpaz localprocessingofmassivedatabaseswithranationalanalysisofabraziliansocialprogramme AT mateusmaia localprocessingofmassivedatabaseswithranationalanalysisofabraziliansocialprogramme AT fernandomoraes localprocessingofmassivedatabaseswithranationalanalysisofabraziliansocialprogramme AT ricardolustosa localprocessingofmassivedatabaseswithranationalanalysisofabraziliansocialprogramme AT liliacosta localprocessingofmassivedatabaseswithranationalanalysisofabraziliansocialprogramme AT samuelmacedo localprocessingofmassivedatabaseswithranationalanalysisofabraziliansocialprogramme AT marcosebarreto localprocessingofmassivedatabaseswithranationalanalysisofabraziliansocialprogramme AT andersonara localprocessingofmassivedatabaseswithranationalanalysisofabraziliansocialprogramme |