Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus
As researchers use computational methods to study complex social behaviors at scale, the validity of this computational social science depends on the integrity of the data. On July 2, 2015, Jason Baumgartner published a dataset advertised to include “every publicly available Reddit comment” which wa...
| Main Authors: | , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
PLOS One
2020
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/1721.1/123458 |
| _version_ | 1826215500277874688 |
|---|---|
| author | Gaffney, Devin Matias, J. Nathan |
| author2 | Massachusetts Institute of Technology. Center for Civic Media |
| author_facet | Massachusetts Institute of Technology. Center for Civic Media Gaffney, Devin Matias, J. Nathan |
| author_sort | Gaffney, Devin |
| collection | MIT |
| description | As researchers use computational methods to study complex social behaviors at scale, the validity of this computational social science depends on the integrity of the data. On July 2, 2015, Jason Baumgartner published a dataset advertised to include “every publicly available Reddit comment” which was quickly shared on Bittorrent and the Internet Archive. This data quickly became the basis of many academic papers on topics including machine learning, social behavior, politics, breaking news, and hate speech. We have discovered substantial gaps and limitations in this dataset which may contribute to bias in the findings of that research. In this paper, we document the dataset, substantial missing observations in the dataset, and the risks to research validity from those gaps. In summary, we identify strong risks to research that considers user histories or network analysis, moderate risks to research that compares counts of participation, and lesser risk to machine learning research that avoids making representative claims about behavior and participation on Reddit. |
| first_indexed | 2024-09-23T16:32:07Z |
| format | Article |
| id | mit-1721.1/123458 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2025-02-19T04:25:34Z |
| publishDate | 2020 |
| publisher | PLOS One |
| record_format | dspace |
| spelling | mit-1721.1/1234582025-02-06T18:44:51Z Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus Gaffney, Devin Matias, J. Nathan Massachusetts Institute of Technology. Center for Civic Media audit, data, reddit, social media, data analysis As researchers use computational methods to study complex social behaviors at scale, the validity of this computational social science depends on the integrity of the data. On July 2, 2015, Jason Baumgartner published a dataset advertised to include “every publicly available Reddit comment” which was quickly shared on Bittorrent and the Internet Archive. This data quickly became the basis of many academic papers on topics including machine learning, social behavior, politics, breaking news, and hate speech. We have discovered substantial gaps and limitations in this dataset which may contribute to bias in the findings of that research. In this paper, we document the dataset, substantial missing observations in the dataset, and the risks to research validity from those gaps. In summary, we identify strong risks to research that considers user histories or network analysis, moderate risks to research that compares counts of participation, and lesser risk to machine learning research that avoids making representative claims about behavior and participation on Reddit. 2020-01-16T18:39:58Z 2020-01-16T18:39:58Z 2018-07-06 Article https://hdl.handle.net/1721.1/123458 en application/pdf PLOS One |
| spellingShingle | audit, data, reddit, social media, data analysis Gaffney, Devin Matias, J. Nathan Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus |
| title | Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus |
| title_full | Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus |
| title_fullStr | Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus |
| title_full_unstemmed | Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus |
| title_short | Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus |
| title_sort | caveat emptor computational social science large scale missing data in a widely published reddit corpus |
| topic | audit, data, reddit, social media, data analysis |
| url | https://hdl.handle.net/1721.1/123458 |
| work_keys_str_mv | AT gaffneydevin caveatemptorcomputationalsocialsciencelargescalemissingdatainawidelypublishedredditcorpus AT matiasjnathan caveatemptorcomputationalsocialsciencelargescalemissingdatainawidelypublishedredditcorpus |