Flexibility in reaction time analysis: many roads to a false positive?
In the present article, we explore the influence of undisclosed flexibility in the analysis of reaction times (RTs). RTs entail some degrees of freedom of their own, due to their skewed distribution, the potential presence of outliers and the availability of different methods to deal with these issu...
Main Authors: | , |
---|---|
Format: | Article |
Language: | English |
Published: |
The Royal Society
2020-02-01
|
Series: | Royal Society Open Science |
Subjects: | |
Online Access: | https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.190831 |
_version_ | 1818251237759385600 |
---|---|
author | Luis Morís Fernández Miguel A. Vadillo |
author_facet | Luis Morís Fernández Miguel A. Vadillo |
author_sort | Luis Morís Fernández |
collection | DOAJ |
description | In the present article, we explore the influence of undisclosed flexibility in the analysis of reaction times (RTs). RTs entail some degrees of freedom of their own, due to their skewed distribution, the potential presence of outliers and the availability of different methods to deal with these issues. Moreover, these degrees of freedom are usually not considered part of the analysis itself, but preprocessing steps that are contingent on data. We analysed the impact of these degrees of freedom on the false-positive rate using simulations over real and simulated data. When several preprocessing methods are used in combination, the false-positive rate can easily rise to 17%. This figure becomes more concerning if we consider that more degrees of freedom are awaiting down the analysis pipeline, potentially making the final false-positive rate much higher. |
first_indexed | 2024-12-12T16:05:06Z |
format | Article |
id | doaj.art-8c7956c21ffd4097b0936c93b9a64343 |
institution | Directory Open Access Journal |
issn | 2054-5703 |
language | English |
last_indexed | 2024-12-12T16:05:06Z |
publishDate | 2020-02-01 |
publisher | The Royal Society |
record_format | Article |
series | Royal Society Open Science |
spelling | doaj.art-8c7956c21ffd4097b0936c93b9a643432022-12-22T00:19:20ZengThe Royal SocietyRoyal Society Open Science2054-57032020-02-017210.1098/rsos.190831190831Flexibility in reaction time analysis: many roads to a false positive?Luis Morís FernándezMiguel A. VadilloIn the present article, we explore the influence of undisclosed flexibility in the analysis of reaction times (RTs). RTs entail some degrees of freedom of their own, due to their skewed distribution, the potential presence of outliers and the availability of different methods to deal with these issues. Moreover, these degrees of freedom are usually not considered part of the analysis itself, but preprocessing steps that are contingent on data. We analysed the impact of these degrees of freedom on the false-positive rate using simulations over real and simulated data. When several preprocessing methods are used in combination, the false-positive rate can easily rise to 17%. This figure becomes more concerning if we consider that more degrees of freedom are awaiting down the analysis pipeline, potentially making the final false-positive rate much higher.https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.190831reaction timequestionable research practicesp-hackingfalse positives |
spellingShingle | Luis Morís Fernández Miguel A. Vadillo Flexibility in reaction time analysis: many roads to a false positive? Royal Society Open Science reaction time questionable research practices p-hacking false positives |
title | Flexibility in reaction time analysis: many roads to a false positive? |
title_full | Flexibility in reaction time analysis: many roads to a false positive? |
title_fullStr | Flexibility in reaction time analysis: many roads to a false positive? |
title_full_unstemmed | Flexibility in reaction time analysis: many roads to a false positive? |
title_short | Flexibility in reaction time analysis: many roads to a false positive? |
title_sort | flexibility in reaction time analysis many roads to a false positive |
topic | reaction time questionable research practices p-hacking false positives |
url | https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.190831 |
work_keys_str_mv | AT luismorisfernandez flexibilityinreactiontimeanalysismanyroadstoafalsepositive AT miguelavadillo flexibilityinreactiontimeanalysismanyroadstoafalsepositive |