Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project
The low reproducibility rate in social sciences has produced hesitation among researchers in accepting published findings at their face value. Despite the advent of initiatives to increase transparency in research reporting, the field is still lacking tools to verify the credibility of research repo...
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
The Royal Society
2023-02-01
|
| Series: | Royal Society Open Science |
| Subjects: | |
| Online Access: | https://royalsocietypublishing.org/doi/10.1098/rsos.191375 |
| _version_ | 1797937805027966976 |
|---|---|
| author | Zoltan Kekecs Bence Palfi Barnabas Szaszi Peter Szecsi Mark Zrubka Marton Kovacs Bence E. Bakos Denis Cousineau Patrizio Tressoldi Kathleen Schmidt Massimo Grassi Thomas Rhys Evans Yuki Yamada Jeremy K. Miller Huanxu Liu Fumiya Yonemitsu Dmitrii Dubrov Jan Philipp Röer Marvin Becker Roxane Schnepper Atsunori Ariga Patrícia Arriaga Raquel Oliveira Nele Põldver Kairi Kreegipuu Braeden Hall Sera Wiechert Bruno Verschuere Kyra Girán Balazs Aczel |
| author_facet | Zoltan Kekecs Bence Palfi Barnabas Szaszi Peter Szecsi Mark Zrubka Marton Kovacs Bence E. Bakos Denis Cousineau Patrizio Tressoldi Kathleen Schmidt Massimo Grassi Thomas Rhys Evans Yuki Yamada Jeremy K. Miller Huanxu Liu Fumiya Yonemitsu Dmitrii Dubrov Jan Philipp Röer Marvin Becker Roxane Schnepper Atsunori Ariga Patrícia Arriaga Raquel Oliveira Nele Põldver Kairi Kreegipuu Braeden Hall Sera Wiechert Bruno Verschuere Kyra Girán Balazs Aczel |
| author_sort | Zoltan Kekecs |
| collection | DOAJ |
| description | The low reproducibility rate in social sciences has produced hesitation among researchers in accepting published findings at their face value. Despite the advent of initiatives to increase transparency in research reporting, the field is still lacking tools to verify the credibility of research reports. In the present paper, we describe methodologies that let researchers craft highly credible research and allow their peers to verify this credibility. We demonstrate the application of these methods in a multi-laboratory replication of Bem's Experiment 1 (Bem 2011 J. Pers. Soc. Psychol. 100, 407–425. (doi:10.1037/a0021524)) on extrasensory perception (ESP), which was co-designed by a consensus panel including both proponents and opponents of Bem's original hypothesis. In the study we applied direct data deposition in combination with born-open data and real-time research reports to extend transparency to protocol delivery and data collection. We also used piloting, checklists, laboratory logs and video-documented trial sessions to ascertain as-intended protocol delivery, and external research auditors to monitor research integrity. We found 49.89% successful guesses, while Bem reported 53.07% success rate, with the chance level being 50%. Thus, Bem's findings were not replicated in our study. In the paper, we discuss the implementation, feasibility and perceived usefulness of the credibility-enhancing methodologies used throughout the project. |
| first_indexed | 2024-04-10T18:49:55Z |
| format | Article |
| id | doaj.art-03421869af02442eb64c56cb49061e54 |
| institution | Directory Open Access Journal |
| issn | 2054-5703 |
| language | English |
| last_indexed | 2024-04-10T18:49:55Z |
| publishDate | 2023-02-01 |
| publisher | The Royal Society |
| record_format | Article |
| series | Royal Society Open Science |
| spelling | doaj.art-03421869af02442eb64c56cb49061e542023-02-01T08:05:52ZengThe Royal SocietyRoyal Society Open Science2054-57032023-02-0110210.1098/rsos.191375Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi projectZoltan Kekecs0Bence Palfi1Barnabas Szaszi2Peter Szecsi3Mark Zrubka4Marton Kovacs5Bence E. Bakos6Denis Cousineau7Patrizio Tressoldi8Kathleen Schmidt9Massimo Grassi10Thomas Rhys Evans11Yuki Yamada12Jeremy K. Miller13Huanxu Liu14Fumiya Yonemitsu15Dmitrii Dubrov16Jan Philipp Röer17Marvin Becker18Roxane Schnepper19Atsunori Ariga20Patrícia Arriaga21Raquel Oliveira22Nele Põldver23Kairi Kreegipuu24Braeden Hall25Sera Wiechert26Bruno Verschuere27Kyra Girán28Balazs Aczel29Institute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, HungaryDepartment of Surgery and Cancer, Imperial College London, London, UKInstitute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, HungaryInstitute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, HungaryDepartment of Psychology, University of Amsterdam, P.O. Box 19268, 1000 GG Amsterdam, The NetherlandsInstitute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, HungaryInstitute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, HungaryÉcole de psychologie, University of Ottawa, 136, Jean-Jacques Lussier, Ontario, Canada, K1N 6N5Studium Patavinum, Università di Padova via Venezia 8, 35131 Padova, ItalyDepartment of Psychology, Ashland University, Ashland, OH 44805, USADipartimento di Psicologia Generale, Università di Padova via Venezia 8, 35131 Padova, ItalySchool of Human Sciences, University of Greenwich, UKFaculty of Arts and Science, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka 819-0395, JapanDepartment of Psychology, Willamette University, 900 State Street, Salem, OR 97301, USAGraduate School of Human-Environment Studies, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka 819-0395, JapanFaculty of Letters, Chuo University, Hachioji, Tokyo 192-0393, JapanNational Research University Higher School of Economics, Russian FederationDepartment of Psychology and Psychotherapy, Witten/Herdecke University, Witten, GermanyDepartment of Psychology and Psychotherapy, Witten/Herdecke University, Witten, GermanyDepartment of Psychology and Psychotherapy, Witten/Herdecke University, Witten, GermanyFaculty of Letters, Chuo University, Hachioji, Tokyo 192-0393, JapanIscte-University Institute of Lisbon, CIS_Iscte, Av. das Forças Armadas, 1649-026, Lisbon, PortugalIscte-University Institute of Lisbon, CIS_Iscte, Av. das Forças Armadas, 1649-026, Lisbon, PortugalUniversity of Tartu Institute of Psychology, EstoniaUniversity of Tartu Institute of Psychology, EstoniaSchool of Psychological & Behavioral Sciences, Southern Illinois University, Carbondale, USAUniversity of Amsterdam, Amsterdam, The NetherlandsUniversity of Amsterdam, Amsterdam, The NetherlandsInstitute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, HungaryInstitute of Psychology, ELTE, Eotvos Lorand University, Izabella u 46. 1064, Budapest, HungaryThe low reproducibility rate in social sciences has produced hesitation among researchers in accepting published findings at their face value. Despite the advent of initiatives to increase transparency in research reporting, the field is still lacking tools to verify the credibility of research reports. In the present paper, we describe methodologies that let researchers craft highly credible research and allow their peers to verify this credibility. We demonstrate the application of these methods in a multi-laboratory replication of Bem's Experiment 1 (Bem 2011 J. Pers. Soc. Psychol. 100, 407–425. (doi:10.1037/a0021524)) on extrasensory perception (ESP), which was co-designed by a consensus panel including both proponents and opponents of Bem's original hypothesis. In the study we applied direct data deposition in combination with born-open data and real-time research reports to extend transparency to protocol delivery and data collection. We also used piloting, checklists, laboratory logs and video-documented trial sessions to ascertain as-intended protocol delivery, and external research auditors to monitor research integrity. We found 49.89% successful guesses, while Bem reported 53.07% success rate, with the chance level being 50%. Thus, Bem's findings were not replicated in our study. In the paper, we discuss the implementation, feasibility and perceived usefulness of the credibility-enhancing methodologies used throughout the project.https://royalsocietypublishing.org/doi/10.1098/rsos.191375metascienceresearch methodscredibilitytransparencyreplicationreal-time procedures |
| spellingShingle | Zoltan Kekecs Bence Palfi Barnabas Szaszi Peter Szecsi Mark Zrubka Marton Kovacs Bence E. Bakos Denis Cousineau Patrizio Tressoldi Kathleen Schmidt Massimo Grassi Thomas Rhys Evans Yuki Yamada Jeremy K. Miller Huanxu Liu Fumiya Yonemitsu Dmitrii Dubrov Jan Philipp Röer Marvin Becker Roxane Schnepper Atsunori Ariga Patrícia Arriaga Raquel Oliveira Nele Põldver Kairi Kreegipuu Braeden Hall Sera Wiechert Bruno Verschuere Kyra Girán Balazs Aczel Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project Royal Society Open Science metascience research methods credibility transparency replication real-time procedures |
| title | Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project |
| title_full | Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project |
| title_fullStr | Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project |
| title_full_unstemmed | Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project |
| title_short | Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project |
| title_sort | raising the value of research studies in psychological science by increasing the credibility of research reports the transparent psi project |
| topic | metascience research methods credibility transparency replication real-time procedures |
| url | https://royalsocietypublishing.org/doi/10.1098/rsos.191375 |
| work_keys_str_mv | AT zoltankekecs raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT bencepalfi raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT barnabasszaszi raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT peterszecsi raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT markzrubka raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT martonkovacs raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT benceebakos raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT deniscousineau raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT patriziotressoldi raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT kathleenschmidt raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT massimograssi raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT thomasrhysevans raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT yukiyamada raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT jeremykmiller raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT huanxuliu raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT fumiyayonemitsu raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT dmitriidubrov raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT janphilipproer raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT marvinbecker raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT roxaneschnepper raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT atsunoriariga raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT patriciaarriaga raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT raqueloliveira raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT nelepoldver raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT kairikreegipuu raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT braedenhall raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT serawiechert raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT brunoverschuere raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT kyragiran raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject AT balazsaczel raisingthevalueofresearchstudiesinpsychologicalsciencebyincreasingthecredibilityofresearchreportsthetransparentpsiproject |