Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid
How trustworthy are peer reviews by applicant-nominated assessors (ANAs)? For Australian Research Council (ARC) proposals (N = 2,330) with at least one ANA and one assessor nominated by the funding panel (PNAs), ANAs gave substantially higher ratings in all nine discipline panels (covering sciences,...
| Main Authors: | , , |
|---|---|
| Format: | Journal article |
| Language: | English |
| Published: |
2007
|
| _version_ | 1826292881342595072 |
|---|---|
| author | Marsh, H Bonds, N Jayasinghe, U |
| author_facet | Marsh, H Bonds, N Jayasinghe, U |
| author_sort | Marsh, H |
| collection | OXFORD |
| description | How trustworthy are peer reviews by applicant-nominated assessors (ANAs)? For Australian Research Council (ARC) proposals (N = 2,330) with at least one ANA and one assessor nominated by the funding panel (PNAs), ANAs gave substantially higher ratings in all nine discipline panels (covering sciences, social sciences, and humanities). Compared to reviews by PNAs, ANA ratings were less related to ratings by other assessors, less related to the ARC final assessment, and contributed to the unreliability of peer reviews. Furthermore, when the same assessor was both an ANA and PNA for different proposals, ratings in the role of ANA were biased whereas those by the same person in the role of PNA were not. ANA ratings of ARC grant proposals are biased, inflated, unreliable, and invalid, leading the ARC to abandon use of ANAs. Particularly if replicated in other situations, the results have important implications for other evaluations based on ANAs. © The Australian Psychological Society Ltd. |
| first_indexed | 2024-03-07T03:21:32Z |
| format | Journal article |
| id | oxford-uuid:b799c7da-43a3-4bc4-9d0e-860ab9301d86 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T03:21:32Z |
| publishDate | 2007 |
| record_format | dspace |
| spelling | oxford-uuid:b799c7da-43a3-4bc4-9d0e-860ab9301d862022-03-27T04:49:47ZPeer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalidJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:b799c7da-43a3-4bc4-9d0e-860ab9301d86EnglishSymplectic Elements at Oxford2007Marsh, HBonds, NJayasinghe, UHow trustworthy are peer reviews by applicant-nominated assessors (ANAs)? For Australian Research Council (ARC) proposals (N = 2,330) with at least one ANA and one assessor nominated by the funding panel (PNAs), ANAs gave substantially higher ratings in all nine discipline panels (covering sciences, social sciences, and humanities). Compared to reviews by PNAs, ANA ratings were less related to ratings by other assessors, less related to the ARC final assessment, and contributed to the unreliability of peer reviews. Furthermore, when the same assessor was both an ANA and PNA for different proposals, ratings in the role of ANA were biased whereas those by the same person in the role of PNA were not. ANA ratings of ARC grant proposals are biased, inflated, unreliable, and invalid, leading the ARC to abandon use of ANAs. Particularly if replicated in other situations, the results have important implications for other evaluations based on ANAs. © The Australian Psychological Society Ltd. |
| spellingShingle | Marsh, H Bonds, N Jayasinghe, U Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid |
| title | Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid |
| title_full | Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid |
| title_fullStr | Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid |
| title_full_unstemmed | Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid |
| title_short | Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid |
| title_sort | peer review process assessments by applicant nominated referees are biased inflated unreliable and invalid |
| work_keys_str_mv | AT marshh peerreviewprocessassessmentsbyapplicantnominatedrefereesarebiasedinflatedunreliableandinvalid AT bondsn peerreviewprocessassessmentsbyapplicantnominatedrefereesarebiasedinflatedunreliableandinvalid AT jayasingheu peerreviewprocessassessmentsbyapplicantnominatedrefereesarebiasedinflatedunreliableandinvalid |