In models we trust: preregistration, large samples, and replication may not suffice
Despite discussions about the replicability of findings in psychological research, two issues have been largely ignored: selection mechanisms and model assumptions. Both topics address the same fundamental question: Does the chosen statistical analysis tool adequately model the data generation proce...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-09-01
|
| Series: | Frontiers in Psychology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1266447/full |
| _version_ | 1827808971456512000 |
|---|---|
| author | Martin Spiess Pascal Jordan |
| author_facet | Martin Spiess Pascal Jordan |
| author_sort | Martin Spiess |
| collection | DOAJ |
| description | Despite discussions about the replicability of findings in psychological research, two issues have been largely ignored: selection mechanisms and model assumptions. Both topics address the same fundamental question: Does the chosen statistical analysis tool adequately model the data generation process? In this article, we address both issues and show, in a first step, that in the face of selective samples and contrary to common practice, the validity of inferences, even when based on experimental designs, can be claimed without further justification and adaptation of standard methods only in very specific situations. We then broaden our perspective to discuss consequences of violated assumptions in linear models in the context of psychological research in general and in generalized linear mixed models as used in item response theory. These types of misspecification are oftentimes ignored in the psychological research literature. It is emphasized that the above problems cannot be overcome by strategies such as preregistration, large samples, replications, or a ban on testing null hypotheses. To avoid biased conclusions, we briefly discuss tools such as model diagnostics, statistical methods to compensate for selectivity and semi- or non-parametric estimation. At a more fundamental level, however, a twofold strategy seems indispensable: (1) iterative, cumulative theory development based on statistical methods with theoretically justified assumptions, and (2) empirical research on variables that affect (self-) selection into the observed part of the sample and the use of this information to compensate for selectivity. |
| first_indexed | 2024-03-11T22:35:17Z |
| format | Article |
| id | doaj.art-d77787fed5b94d7690b81687ac6af081 |
| institution | Directory Open Access Journal |
| issn | 1664-1078 |
| language | English |
| last_indexed | 2024-03-11T22:35:17Z |
| publishDate | 2023-09-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Psychology |
| spelling | doaj.art-d77787fed5b94d7690b81687ac6af0812023-09-22T15:32:53ZengFrontiers Media S.A.Frontiers in Psychology1664-10782023-09-011410.3389/fpsyg.2023.12664471266447In models we trust: preregistration, large samples, and replication may not sufficeMartin SpiessPascal JordanDespite discussions about the replicability of findings in psychological research, two issues have been largely ignored: selection mechanisms and model assumptions. Both topics address the same fundamental question: Does the chosen statistical analysis tool adequately model the data generation process? In this article, we address both issues and show, in a first step, that in the face of selective samples and contrary to common practice, the validity of inferences, even when based on experimental designs, can be claimed without further justification and adaptation of standard methods only in very specific situations. We then broaden our perspective to discuss consequences of violated assumptions in linear models in the context of psychological research in general and in generalized linear mixed models as used in item response theory. These types of misspecification are oftentimes ignored in the psychological research literature. It is emphasized that the above problems cannot be overcome by strategies such as preregistration, large samples, replications, or a ban on testing null hypotheses. To avoid biased conclusions, we briefly discuss tools such as model diagnostics, statistical methods to compensate for selectivity and semi- or non-parametric estimation. At a more fundamental level, however, a twofold strategy seems indispensable: (1) iterative, cumulative theory development based on statistical methods with theoretically justified assumptions, and (2) empirical research on variables that affect (self-) selection into the observed part of the sample and the use of this information to compensate for selectivity.https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1266447/fullpopulationsampling designnon-responseselectivitymisspecificationbiased inference |
| spellingShingle | Martin Spiess Pascal Jordan In models we trust: preregistration, large samples, and replication may not suffice Frontiers in Psychology population sampling design non-response selectivity misspecification biased inference |
| title | In models we trust: preregistration, large samples, and replication may not suffice |
| title_full | In models we trust: preregistration, large samples, and replication may not suffice |
| title_fullStr | In models we trust: preregistration, large samples, and replication may not suffice |
| title_full_unstemmed | In models we trust: preregistration, large samples, and replication may not suffice |
| title_short | In models we trust: preregistration, large samples, and replication may not suffice |
| title_sort | in models we trust preregistration large samples and replication may not suffice |
| topic | population sampling design non-response selectivity misspecification biased inference |
| url | https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1266447/full |
| work_keys_str_mv | AT martinspiess inmodelswetrustpreregistrationlargesamplesandreplicationmaynotsuffice AT pascaljordan inmodelswetrustpreregistrationlargesamplesandreplicationmaynotsuffice |