Is big team research fair in national research assessments? The case of the UK Research Excellence Framework 2021
Collaborative research causes problems for research assessments because of the difficulty in fairly crediting its authors. Whilst splitting the rewards for an article amongst its authors has the greatest surface-level fairness, many important evaluations assign full credit to each author, irrespecti...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Sciendo
2023-03-01
|
| Series: | Journal of Data and Information Science |
| Subjects: | |
| Online Access: | https://doi.org/10.2478/jdis-2023-0004 |
| _version_ | 1797848519751499776 |
|---|---|
| author | Thelwall Mike Kousha Kayvan Makita Meiko Abdoli Mahshid Stuart Emma Wilson Paul Levitt Jonathan |
| author_facet | Thelwall Mike Kousha Kayvan Makita Meiko Abdoli Mahshid Stuart Emma Wilson Paul Levitt Jonathan |
| author_sort | Thelwall Mike |
| collection | DOAJ |
| description | Collaborative research causes problems for research assessments because of the difficulty in fairly crediting its authors. Whilst splitting the rewards for an article amongst its authors has the greatest surface-level fairness, many important evaluations assign full credit to each author, irrespective of team size. The underlying rationales for this are labour reduction and the need to incentivise collaborative work because it is necessary to solve many important societal problems. This article assesses whether full counting changes results compared to fractional counting in the case of the UK's Research Excellence Framework (REF) 2021. For this assessment, fractional counting reduces the number of journal articles to as little as 10% of the full counting value, depending on the Unit of Assessment (UoA). Despite this large difference, allocating an overall grade point average (GPA) based on full counting or fractional counting gives results with a median Pearson correlation within UoAs of 0.98. The largest changes are for Archaeology (r=0.84) and Physics (r=0.88). There is a weak tendency for higher scoring institutions to lose from fractional counting, with the loss being statistically significant in 5 of the 34 UoAs. Thus, whilst the apparent over-weighting of contributions to collaboratively authored outputs does not seem too problematic from a fairness perspective overall, it may be worth examining in the few UoAs in which it makes the most difference. |
| first_indexed | 2024-04-09T18:28:55Z |
| format | Article |
| id | doaj.art-9c296828fcd840159bebd9d28389f1ee |
| institution | Directory Open Access Journal |
| issn | 2543-683X |
| language | English |
| last_indexed | 2024-04-09T18:28:55Z |
| publishDate | 2023-03-01 |
| publisher | Sciendo |
| record_format | Article |
| series | Journal of Data and Information Science |
| spelling | doaj.art-9c296828fcd840159bebd9d28389f1ee2023-04-11T17:27:17ZengSciendoJournal of Data and Information Science2543-683X2023-03-018192010.2478/jdis-2023-0004Is big team research fair in national research assessments? The case of the UK Research Excellence Framework 2021Thelwall Mike0Kousha Kayvan1Makita Meiko2Abdoli Mahshid3Stuart Emma4Wilson Paul5Levitt Jonathan6Statistical Cybermetrics and Research Evaluation Group, University of Wolverhampton, UK.Statistical Cybermetrics and Research Evaluation Group, University of Wolverhampton, UK.Statistical Cybermetrics and Research Evaluation Group, University of Wolverhampton, UK.Statistical Cybermetrics and Research Evaluation Group, University of Wolverhampton, UK.Statistical Cybermetrics and Research Evaluation Group, University of Wolverhampton, UK.Statistical Cybermetrics and Research Evaluation Group, University of Wolverhampton, UK.Statistical Cybermetrics and Research Evaluation Group, University of Wolverhampton, UK.Collaborative research causes problems for research assessments because of the difficulty in fairly crediting its authors. Whilst splitting the rewards for an article amongst its authors has the greatest surface-level fairness, many important evaluations assign full credit to each author, irrespective of team size. The underlying rationales for this are labour reduction and the need to incentivise collaborative work because it is necessary to solve many important societal problems. This article assesses whether full counting changes results compared to fractional counting in the case of the UK's Research Excellence Framework (REF) 2021. For this assessment, fractional counting reduces the number of journal articles to as little as 10% of the full counting value, depending on the Unit of Assessment (UoA). Despite this large difference, allocating an overall grade point average (GPA) based on full counting or fractional counting gives results with a median Pearson correlation within UoAs of 0.98. The largest changes are for Archaeology (r=0.84) and Physics (r=0.88). There is a weak tendency for higher scoring institutions to lose from fractional counting, with the loss being statistically significant in 5 of the 34 UoAs. Thus, whilst the apparent over-weighting of contributions to collaboratively authored outputs does not seem too problematic from a fairness perspective overall, it may be worth examining in the few UoAs in which it makes the most difference.https://doi.org/10.2478/jdis-2023-0004collaborationresearch assessmentrefref2021research qualityscientometrics |
| spellingShingle | Thelwall Mike Kousha Kayvan Makita Meiko Abdoli Mahshid Stuart Emma Wilson Paul Levitt Jonathan Is big team research fair in national research assessments? The case of the UK Research Excellence Framework 2021 Journal of Data and Information Science collaboration research assessment ref ref2021 research quality scientometrics |
| title | Is big team research fair in national research assessments? The case of the UK Research Excellence Framework 2021 |
| title_full | Is big team research fair in national research assessments? The case of the UK Research Excellence Framework 2021 |
| title_fullStr | Is big team research fair in national research assessments? The case of the UK Research Excellence Framework 2021 |
| title_full_unstemmed | Is big team research fair in national research assessments? The case of the UK Research Excellence Framework 2021 |
| title_short | Is big team research fair in national research assessments? The case of the UK Research Excellence Framework 2021 |
| title_sort | is big team research fair in national research assessments the case of the uk research excellence framework 2021 |
| topic | collaboration research assessment ref ref2021 research quality scientometrics |
| url | https://doi.org/10.2478/jdis-2023-0004 |
| work_keys_str_mv | AT thelwallmike isbigteamresearchfairinnationalresearchassessmentsthecaseoftheukresearchexcellenceframework2021 AT koushakayvan isbigteamresearchfairinnationalresearchassessmentsthecaseoftheukresearchexcellenceframework2021 AT makitameiko isbigteamresearchfairinnationalresearchassessmentsthecaseoftheukresearchexcellenceframework2021 AT abdolimahshid isbigteamresearchfairinnationalresearchassessmentsthecaseoftheukresearchexcellenceframework2021 AT stuartemma isbigteamresearchfairinnationalresearchassessmentsthecaseoftheukresearchexcellenceframework2021 AT wilsonpaul isbigteamresearchfairinnationalresearchassessmentsthecaseoftheukresearchexcellenceframework2021 AT levittjonathan isbigteamresearchfairinnationalresearchassessmentsthecaseoftheukresearchexcellenceframework2021 |