Cite-seeing and reviewing: A study on citation bias in peer review.
Citations play an important role in researchers' careers as a key factor in evaluation of scientific impact. Many anecdotes advice authors to exploit this fact and cite prospective reviewers to try obtaining a more positive evaluation for their submission. In this work, we investigate if such a...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2023-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0283980&type=printable |
| _version_ | 1797394486216622080 |
|---|---|
| author | Ivan Stelmakh Charvi Rastogi Ryan Liu Shuchi Chawla Federico Echenique Nihar B Shah |
| author_facet | Ivan Stelmakh Charvi Rastogi Ryan Liu Shuchi Chawla Federico Echenique Nihar B Shah |
| author_sort | Ivan Stelmakh |
| collection | DOAJ |
| description | Citations play an important role in researchers' careers as a key factor in evaluation of scientific impact. Many anecdotes advice authors to exploit this fact and cite prospective reviewers to try obtaining a more positive evaluation for their submission. In this work, we investigate if such a citation bias actually exists: Does the citation of a reviewer's own work in a submission cause them to be positively biased towards the submission? In conjunction with the review process of two flagship conferences in machine learning and algorithmic economics, we execute an observational study to test for citation bias in peer review. In our analysis, we carefully account for various confounding factors such as paper quality and reviewer expertise, and apply different modeling techniques to alleviate concerns regarding the model mismatch. Overall, our analysis involves 1,314 papers and 1,717 reviewers and detects citation bias in both venues we consider. In terms of the effect size, by citing a reviewer's work, a submission has a non-trivial chance of getting a higher score from the reviewer: an expected increase in the score is approximately 0.23 on a 5-point Likert item. For reference, a one-point increase of a score by a single reviewer improves the position of a submission by 11% on average. |
| first_indexed | 2024-03-09T00:20:29Z |
| format | Article |
| id | doaj.art-7f5673ebb6b8421a892acb555869b948 |
| institution | Directory Open Access Journal |
| issn | 1932-6203 |
| language | English |
| last_indexed | 2024-03-09T00:20:29Z |
| publishDate | 2023-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj.art-7f5673ebb6b8421a892acb555869b9482023-12-12T05:35:14ZengPublic Library of Science (PLoS)PLoS ONE1932-62032023-01-01187e028398010.1371/journal.pone.0283980Cite-seeing and reviewing: A study on citation bias in peer review.Ivan StelmakhCharvi RastogiRyan LiuShuchi ChawlaFederico EcheniqueNihar B ShahCitations play an important role in researchers' careers as a key factor in evaluation of scientific impact. Many anecdotes advice authors to exploit this fact and cite prospective reviewers to try obtaining a more positive evaluation for their submission. In this work, we investigate if such a citation bias actually exists: Does the citation of a reviewer's own work in a submission cause them to be positively biased towards the submission? In conjunction with the review process of two flagship conferences in machine learning and algorithmic economics, we execute an observational study to test for citation bias in peer review. In our analysis, we carefully account for various confounding factors such as paper quality and reviewer expertise, and apply different modeling techniques to alleviate concerns regarding the model mismatch. Overall, our analysis involves 1,314 papers and 1,717 reviewers and detects citation bias in both venues we consider. In terms of the effect size, by citing a reviewer's work, a submission has a non-trivial chance of getting a higher score from the reviewer: an expected increase in the score is approximately 0.23 on a 5-point Likert item. For reference, a one-point increase of a score by a single reviewer improves the position of a submission by 11% on average.https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0283980&type=printable |
| spellingShingle | Ivan Stelmakh Charvi Rastogi Ryan Liu Shuchi Chawla Federico Echenique Nihar B Shah Cite-seeing and reviewing: A study on citation bias in peer review. PLoS ONE |
| title | Cite-seeing and reviewing: A study on citation bias in peer review. |
| title_full | Cite-seeing and reviewing: A study on citation bias in peer review. |
| title_fullStr | Cite-seeing and reviewing: A study on citation bias in peer review. |
| title_full_unstemmed | Cite-seeing and reviewing: A study on citation bias in peer review. |
| title_short | Cite-seeing and reviewing: A study on citation bias in peer review. |
| title_sort | cite seeing and reviewing a study on citation bias in peer review |
| url | https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0283980&type=printable |
| work_keys_str_mv | AT ivanstelmakh citeseeingandreviewingastudyoncitationbiasinpeerreview AT charvirastogi citeseeingandreviewingastudyoncitationbiasinpeerreview AT ryanliu citeseeingandreviewingastudyoncitationbiasinpeerreview AT shuchichawla citeseeingandreviewingastudyoncitationbiasinpeerreview AT federicoechenique citeseeingandreviewingastudyoncitationbiasinpeerreview AT niharbshah citeseeingandreviewingastudyoncitationbiasinpeerreview |