Flexibility in reaction time analysis: many roads to a false positive?

In the present article, we explore the influence of undisclosed flexibility in the analysis of reaction times (RTs). RTs entail some degrees of freedom of their own, due to their skewed distribution, the potential presence of outliers and the availability of different methods to deal with these issu...

Full description

Bibliographic Details
Main Authors: Luis Morís Fernández, Miguel A. Vadillo
Format: Article
Language:English
Published: The Royal Society 2020-02-01
Series:Royal Society Open Science
Subjects:
Online Access:https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.190831
_version_ 1818251237759385600
author Luis Morís Fernández
Miguel A. Vadillo
author_facet Luis Morís Fernández
Miguel A. Vadillo
author_sort Luis Morís Fernández
collection DOAJ
description In the present article, we explore the influence of undisclosed flexibility in the analysis of reaction times (RTs). RTs entail some degrees of freedom of their own, due to their skewed distribution, the potential presence of outliers and the availability of different methods to deal with these issues. Moreover, these degrees of freedom are usually not considered part of the analysis itself, but preprocessing steps that are contingent on data. We analysed the impact of these degrees of freedom on the false-positive rate using simulations over real and simulated data. When several preprocessing methods are used in combination, the false-positive rate can easily rise to 17%. This figure becomes more concerning if we consider that more degrees of freedom are awaiting down the analysis pipeline, potentially making the final false-positive rate much higher.
first_indexed 2024-12-12T16:05:06Z
format Article
id doaj.art-8c7956c21ffd4097b0936c93b9a64343
institution Directory Open Access Journal
issn 2054-5703
language English
last_indexed 2024-12-12T16:05:06Z
publishDate 2020-02-01
publisher The Royal Society
record_format Article
series Royal Society Open Science
spelling doaj.art-8c7956c21ffd4097b0936c93b9a643432022-12-22T00:19:20ZengThe Royal SocietyRoyal Society Open Science2054-57032020-02-017210.1098/rsos.190831190831Flexibility in reaction time analysis: many roads to a false positive?Luis Morís FernándezMiguel A. VadilloIn the present article, we explore the influence of undisclosed flexibility in the analysis of reaction times (RTs). RTs entail some degrees of freedom of their own, due to their skewed distribution, the potential presence of outliers and the availability of different methods to deal with these issues. Moreover, these degrees of freedom are usually not considered part of the analysis itself, but preprocessing steps that are contingent on data. We analysed the impact of these degrees of freedom on the false-positive rate using simulations over real and simulated data. When several preprocessing methods are used in combination, the false-positive rate can easily rise to 17%. This figure becomes more concerning if we consider that more degrees of freedom are awaiting down the analysis pipeline, potentially making the final false-positive rate much higher.https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.190831reaction timequestionable research practicesp-hackingfalse positives
spellingShingle Luis Morís Fernández
Miguel A. Vadillo
Flexibility in reaction time analysis: many roads to a false positive?
Royal Society Open Science
reaction time
questionable research practices
p-hacking
false positives
title Flexibility in reaction time analysis: many roads to a false positive?
title_full Flexibility in reaction time analysis: many roads to a false positive?
title_fullStr Flexibility in reaction time analysis: many roads to a false positive?
title_full_unstemmed Flexibility in reaction time analysis: many roads to a false positive?
title_short Flexibility in reaction time analysis: many roads to a false positive?
title_sort flexibility in reaction time analysis many roads to a false positive
topic reaction time
questionable research practices
p-hacking
false positives
url https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.190831
work_keys_str_mv AT luismorisfernandez flexibilityinreactiontimeanalysismanyroadstoafalsepositive
AT miguelavadillo flexibilityinreactiontimeanalysismanyroadstoafalsepositive