Comparison of bibliographic data sources: Implications for the robustness of university rankings
Universities are increasingly evaluated on the basis of their outputs. These are often converted to simple and contested rankings with substantial implications for recruitment, income, and perceived prestige. Such evaluation usually relies on a single data source to define the set of outputs for a u...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The MIT Press
2020-03-01
|
| Series: | Quantitative Science Studies |
| Online Access: | https://www.mitpressjournals.org/doi/abs/10.1162/qss_a_00031 |
| _version_ | 1811276272465084416 |
|---|---|
| author | Huang, Chun-Kai (Karl) Neylon, Cameron Brookes-Kenworthy, Chloe Hosking, Richard Montgomery, Lucy Wilson, Katie Ozaygen, Alkim |
| author_facet | Huang, Chun-Kai (Karl) Neylon, Cameron Brookes-Kenworthy, Chloe Hosking, Richard Montgomery, Lucy Wilson, Katie Ozaygen, Alkim |
| author_sort | Huang, Chun-Kai (Karl) |
| collection | DOAJ |
| description | Universities are increasingly evaluated on the basis of their outputs. These are often converted to simple and contested rankings with substantial implications for recruitment, income, and perceived prestige. Such evaluation usually relies on a single data source to define the set of outputs for a university. However, few studies have explored differences across data sources and their implications for metrics and rankings at the institutional scale. We address this gap by performing detailed bibliographic comparisons between Web of Science (WoS), Scopus, and Microsoft Academic (MSA) at the institutional level and supplement this with a manual analysis of 15 universities. We further construct two simple rankings based on citation count and open access status. Our results show that there are significant differences across databases. These differences contribute to drastic changes in rank positions of universities, which are most prevalent for
non-English-speaking universities and those outside the top positions in international university rankings. Overall, MSA has greater coverage than Scopus and WoS, but with less complete affiliation metadata. We suggest that robust evaluation measures need to consider the effect of choice of data sources and recommend an approach where data from multiple sources is integrated to provide a more robust data set. |
| first_indexed | 2024-04-12T23:53:26Z |
| format | Article |
| id | doaj.art-67408dec182148c0ab1db1bcd748c69f |
| institution | Directory Open Access Journal |
| issn | 2641-3337 |
| language | English |
| last_indexed | 2024-04-12T23:53:26Z |
| publishDate | 2020-03-01 |
| publisher | The MIT Press |
| record_format | Article |
| series | Quantitative Science Studies |
| spelling | doaj.art-67408dec182148c0ab1db1bcd748c69f2022-12-22T03:11:35ZengThe MIT PressQuantitative Science Studies2641-33372020-03-0113410.1162/qss_a_00031Comparison of bibliographic data sources: Implications for the robustness of university rankingsHuang, Chun-Kai (Karl)Neylon, CameronBrookes-Kenworthy, ChloeHosking, RichardMontgomery, LucyWilson, KatieOzaygen, AlkimUniversities are increasingly evaluated on the basis of their outputs. These are often converted to simple and contested rankings with substantial implications for recruitment, income, and perceived prestige. Such evaluation usually relies on a single data source to define the set of outputs for a university. However, few studies have explored differences across data sources and their implications for metrics and rankings at the institutional scale. We address this gap by performing detailed bibliographic comparisons between Web of Science (WoS), Scopus, and Microsoft Academic (MSA) at the institutional level and supplement this with a manual analysis of 15 universities. We further construct two simple rankings based on citation count and open access status. Our results show that there are significant differences across databases. These differences contribute to drastic changes in rank positions of universities, which are most prevalent for non-English-speaking universities and those outside the top positions in international university rankings. Overall, MSA has greater coverage than Scopus and WoS, but with less complete affiliation metadata. We suggest that robust evaluation measures need to consider the effect of choice of data sources and recommend an approach where data from multiple sources is integrated to provide a more robust data set.https://www.mitpressjournals.org/doi/abs/10.1162/qss_a_00031 |
| spellingShingle | Huang, Chun-Kai (Karl) Neylon, Cameron Brookes-Kenworthy, Chloe Hosking, Richard Montgomery, Lucy Wilson, Katie Ozaygen, Alkim Comparison of bibliographic data sources: Implications for the robustness of university rankings Quantitative Science Studies |
| title | Comparison of bibliographic data sources: Implications for the robustness of university rankings |
| title_full | Comparison of bibliographic data sources: Implications for the robustness of university rankings |
| title_fullStr | Comparison of bibliographic data sources: Implications for the robustness of university rankings |
| title_full_unstemmed | Comparison of bibliographic data sources: Implications for the robustness of university rankings |
| title_short | Comparison of bibliographic data sources: Implications for the robustness of university rankings |
| title_sort | comparison of bibliographic data sources implications for the robustness of university rankings |
| url | https://www.mitpressjournals.org/doi/abs/10.1162/qss_a_00031 |
| work_keys_str_mv | AT huangchunkaikarl comparisonofbibliographicdatasourcesimplicationsfortherobustnessofuniversityrankings AT neyloncameron comparisonofbibliographicdatasourcesimplicationsfortherobustnessofuniversityrankings AT brookeskenworthychloe comparisonofbibliographicdatasourcesimplicationsfortherobustnessofuniversityrankings AT hoskingrichard comparisonofbibliographicdatasourcesimplicationsfortherobustnessofuniversityrankings AT montgomerylucy comparisonofbibliographicdatasourcesimplicationsfortherobustnessofuniversityrankings AT wilsonkatie comparisonofbibliographicdatasourcesimplicationsfortherobustnessofuniversityrankings AT ozaygenalkim comparisonofbibliographicdatasourcesimplicationsfortherobustnessofuniversityrankings |