Counting Research Publications, Citations, and Topics: A Critical Assessment of the Empirical Basis of Scientometrics and Research Evaluation
Scientometrics and research evaluation describe and analyze research publications when conducting publication, citation, and topic analyses. However, what exactly is a (scientific, academic, scholarly or research) publication? This article demonstrates that there are many problems when it comes to l...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Korea Institute of Science and Technology Information
2023-06-01
|
| Series: | Journal of Information Science Theory and Practice |
| Subjects: | |
| Online Access: | https://data.doi.or.kr/10.1633/JISTaP.2023.11.2.4 |
| _version_ | 1797793902161297408 |
|---|---|
| author | Wolfgang G. Stock Gerhard Reichmann Isabelle Dorsch Christian Schlögl |
| author_facet | Wolfgang G. Stock Gerhard Reichmann Isabelle Dorsch Christian Schlögl |
| author_sort | Wolfgang G. Stock |
| collection | DOAJ |
| description | Scientometrics and research evaluation describe and analyze research publications when conducting publication, citation, and topic analyses. However, what exactly is a (scientific, academic, scholarly or research) publication? This article demonstrates that there are many problems when it comes to looking in detail at quantitative publication analyses, citation analyses, altmetric analyses, and topic analyses. When is a document a publication and when is it not? We discuss authorship and contribution, formally and informally published documents, as well as documents in between (preprints, research data) and the characteristics of references, citations, and topics. What is a research publication? Is there a commonly accepted criterion for distinguishing between research and non-research? How complete and unbiased are data sources for research publications and sources for altmetrics? What is one research publication? What is the unit of a publication that causes us to count it as “1?” In this regard, we report problems related to multi-author publications and their counting, weighted document types, the unit and weighting of citations and references, the unit of topics, and counting problems—not only at the article and individual researcher level (micro-level), but also at the meso-level (e.g., institutions) and macro-level (e.g., countries). Our results suggest that scientometric counting units are not reliable and clear. Many scientometric and research evaluation studies must therefore be used with the utmost caution. |
| first_indexed | 2024-03-13T02:54:59Z |
| format | Article |
| id | doaj.art-dbd038fa82484145ac472234cba9042d |
| institution | Directory Open Access Journal |
| issn | 2287-9099 2287-4577 |
| language | English |
| last_indexed | 2024-03-13T02:54:59Z |
| publishDate | 2023-06-01 |
| publisher | Korea Institute of Science and Technology Information |
| record_format | Article |
| series | Journal of Information Science Theory and Practice |
| spelling | doaj.art-dbd038fa82484145ac472234cba9042d2023-06-28T07:23:07ZengKorea Institute of Science and Technology InformationJournal of Information Science Theory and Practice2287-90992287-45772023-06-0111110.1633/JISTaP.2023.11.2.4JISTaP.2023.11.2.4Counting Research Publications, Citations, and Topics: A Critical Assessment of the Empirical Basis of Scientometrics and Research EvaluationWolfgang G. Stock0Gerhard Reichmann1Isabelle Dorsch2Christian Schlögl3Depearment of Information Science, Heinrich Heine University Düsseldorf, Düsseldorf, Germany E-mail: wolfgang.stock@hhu.deDepartment of Operations and Information Systems, University of Graz, Graz, Austria E-mail: gerhard.reichmann@uni-graz.atZBW – Leibniz Information Centre for Economics, Kiel, Germany E-mail: i.dorsch@zbw-online.euDepartment of Operations and Information Systems, University of Graz, Graz, Austria E-mail: christian.schloegl@uni-graz.atScientometrics and research evaluation describe and analyze research publications when conducting publication, citation, and topic analyses. However, what exactly is a (scientific, academic, scholarly or research) publication? This article demonstrates that there are many problems when it comes to looking in detail at quantitative publication analyses, citation analyses, altmetric analyses, and topic analyses. When is a document a publication and when is it not? We discuss authorship and contribution, formally and informally published documents, as well as documents in between (preprints, research data) and the characteristics of references, citations, and topics. What is a research publication? Is there a commonly accepted criterion for distinguishing between research and non-research? How complete and unbiased are data sources for research publications and sources for altmetrics? What is one research publication? What is the unit of a publication that causes us to count it as “1?” In this regard, we report problems related to multi-author publications and their counting, weighted document types, the unit and weighting of citations and references, the unit of topics, and counting problems—not only at the article and individual researcher level (micro-level), but also at the meso-level (e.g., institutions) and macro-level (e.g., countries). Our results suggest that scientometric counting units are not reliable and clear. Many scientometric and research evaluation studies must therefore be used with the utmost caution.https://data.doi.or.kr/10.1633/JISTaP.2023.11.2.4scientometricsresearch publicationcounting unitreferencecitationtopic |
| spellingShingle | Wolfgang G. Stock Gerhard Reichmann Isabelle Dorsch Christian Schlögl Counting Research Publications, Citations, and Topics: A Critical Assessment of the Empirical Basis of Scientometrics and Research Evaluation Journal of Information Science Theory and Practice scientometrics research publication counting unit reference citation topic |
| title | Counting Research Publications, Citations, and Topics: A Critical Assessment of the Empirical Basis of Scientometrics and Research Evaluation |
| title_full | Counting Research Publications, Citations, and Topics: A Critical Assessment of the Empirical Basis of Scientometrics and Research Evaluation |
| title_fullStr | Counting Research Publications, Citations, and Topics: A Critical Assessment of the Empirical Basis of Scientometrics and Research Evaluation |
| title_full_unstemmed | Counting Research Publications, Citations, and Topics: A Critical Assessment of the Empirical Basis of Scientometrics and Research Evaluation |
| title_short | Counting Research Publications, Citations, and Topics: A Critical Assessment of the Empirical Basis of Scientometrics and Research Evaluation |
| title_sort | counting research publications citations and topics a critical assessment of the empirical basis of scientometrics and research evaluation |
| topic | scientometrics research publication counting unit reference citation topic |
| url | https://data.doi.or.kr/10.1633/JISTaP.2023.11.2.4 |
| work_keys_str_mv | AT wolfganggstock countingresearchpublicationscitationsandtopicsacriticalassessmentoftheempiricalbasisofscientometricsandresearchevaluation AT gerhardreichmann countingresearchpublicationscitationsandtopicsacriticalassessmentoftheempiricalbasisofscientometricsandresearchevaluation AT isabelledorsch countingresearchpublicationscitationsandtopicsacriticalassessmentoftheempiricalbasisofscientometricsandresearchevaluation AT christianschlogl countingresearchpublicationscitationsandtopicsacriticalassessmentoftheempiricalbasisofscientometricsandresearchevaluation |