Standard experimental paradigm designs and data exclusion practices in cognitive psychology can inadvertently introduce systematic “shadow” biases in participant samples
Abstract Standard cognitive psychology research practices can introduce inadvertent sampling biases that reduce the reliability and generalizability of the findings. Researchers commonly acknowledge and understand that any given study sample is not perfectly generalizable, especially when implementi...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SpringerOpen
2023-10-01
|
| Series: | Cognitive Research |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s41235-023-00520-y |
| _version_ | 1797454168321949696 |
|---|---|
| author | Emma M. Siritzky Patrick H. Cox Sydni M. Nadler Justin N. Grady Dwight J. Kravitz Stephen R. Mitroff |
| author_facet | Emma M. Siritzky Patrick H. Cox Sydni M. Nadler Justin N. Grady Dwight J. Kravitz Stephen R. Mitroff |
| author_sort | Emma M. Siritzky |
| collection | DOAJ |
| description | Abstract Standard cognitive psychology research practices can introduce inadvertent sampling biases that reduce the reliability and generalizability of the findings. Researchers commonly acknowledge and understand that any given study sample is not perfectly generalizable, especially when implementing typical experimental constraints (e.g., limiting recruitment to specific age ranges or to individuals with normal color vision). However, less obvious systematic sampling constraints, referred to here as “shadow” biases, can be unintentionally introduced and can easily go unnoticed. For example, many standard cognitive psychology study designs involve lengthy and tedious experiments with simple, repetitive stimuli. Such testing environments may 1) be aversive to some would-be participants (e.g., those high in certain neurodivergent symptoms) who may self-select not to enroll in such studies, or 2) contribute to participant attrition, both of which reduce the sample’s representativeness. Likewise, standard performance-based data exclusion efforts (e.g., minimum accuracy or response time) or attention checks can systematically remove data from participants from subsets of the population (e.g., those low in conscientiousness). This commentary focuses on the theoretical and practical issues behind these non-obvious and often unacknowledged “shadow” biases, offers a simple illustration with real data as a proof of concept of how applying attention checks can systematically skew latent/hidden variables in the included population, and then discusses the broader implications with suggestions for how to manage and reduce, or at a minimum acknowledge, the problem. |
| first_indexed | 2024-03-09T15:33:24Z |
| format | Article |
| id | doaj.art-a6adc07e9e864cafb1b68407958642b4 |
| institution | Directory Open Access Journal |
| issn | 2365-7464 |
| language | English |
| last_indexed | 2024-03-09T15:33:24Z |
| publishDate | 2023-10-01 |
| publisher | SpringerOpen |
| record_format | Article |
| series | Cognitive Research |
| spelling | doaj.art-a6adc07e9e864cafb1b68407958642b42023-11-26T12:08:24ZengSpringerOpenCognitive Research2365-74642023-10-018111010.1186/s41235-023-00520-yStandard experimental paradigm designs and data exclusion practices in cognitive psychology can inadvertently introduce systematic “shadow” biases in participant samplesEmma M. Siritzky0Patrick H. Cox1Sydni M. Nadler2Justin N. Grady3Dwight J. Kravitz4Stephen R. Mitroff5Department of Psychological & Brain Sciences, The George Washington UniversityDepartment of Psychological & Brain Sciences, The George Washington UniversityDepartment of Psychological & Brain Sciences, The George Washington UniversityDepartment of Psychological & Brain Sciences, The George Washington UniversityDepartment of Psychological & Brain Sciences, The George Washington UniversityDepartment of Psychological & Brain Sciences, The George Washington UniversityAbstract Standard cognitive psychology research practices can introduce inadvertent sampling biases that reduce the reliability and generalizability of the findings. Researchers commonly acknowledge and understand that any given study sample is not perfectly generalizable, especially when implementing typical experimental constraints (e.g., limiting recruitment to specific age ranges or to individuals with normal color vision). However, less obvious systematic sampling constraints, referred to here as “shadow” biases, can be unintentionally introduced and can easily go unnoticed. For example, many standard cognitive psychology study designs involve lengthy and tedious experiments with simple, repetitive stimuli. Such testing environments may 1) be aversive to some would-be participants (e.g., those high in certain neurodivergent symptoms) who may self-select not to enroll in such studies, or 2) contribute to participant attrition, both of which reduce the sample’s representativeness. Likewise, standard performance-based data exclusion efforts (e.g., minimum accuracy or response time) or attention checks can systematically remove data from participants from subsets of the population (e.g., those low in conscientiousness). This commentary focuses on the theoretical and practical issues behind these non-obvious and often unacknowledged “shadow” biases, offers a simple illustration with real data as a proof of concept of how applying attention checks can systematically skew latent/hidden variables in the included population, and then discusses the broader implications with suggestions for how to manage and reduce, or at a minimum acknowledge, the problem.https://doi.org/10.1186/s41235-023-00520-ySampling biasesCognitive psychologyData exclusionIndividual differences |
| spellingShingle | Emma M. Siritzky Patrick H. Cox Sydni M. Nadler Justin N. Grady Dwight J. Kravitz Stephen R. Mitroff Standard experimental paradigm designs and data exclusion practices in cognitive psychology can inadvertently introduce systematic “shadow” biases in participant samples Cognitive Research Sampling biases Cognitive psychology Data exclusion Individual differences |
| title | Standard experimental paradigm designs and data exclusion practices in cognitive psychology can inadvertently introduce systematic “shadow” biases in participant samples |
| title_full | Standard experimental paradigm designs and data exclusion practices in cognitive psychology can inadvertently introduce systematic “shadow” biases in participant samples |
| title_fullStr | Standard experimental paradigm designs and data exclusion practices in cognitive psychology can inadvertently introduce systematic “shadow” biases in participant samples |
| title_full_unstemmed | Standard experimental paradigm designs and data exclusion practices in cognitive psychology can inadvertently introduce systematic “shadow” biases in participant samples |
| title_short | Standard experimental paradigm designs and data exclusion practices in cognitive psychology can inadvertently introduce systematic “shadow” biases in participant samples |
| title_sort | standard experimental paradigm designs and data exclusion practices in cognitive psychology can inadvertently introduce systematic shadow biases in participant samples |
| topic | Sampling biases Cognitive psychology Data exclusion Individual differences |
| url | https://doi.org/10.1186/s41235-023-00520-y |
| work_keys_str_mv | AT emmamsiritzky standardexperimentalparadigmdesignsanddataexclusionpracticesincognitivepsychologycaninadvertentlyintroducesystematicshadowbiasesinparticipantsamples AT patrickhcox standardexperimentalparadigmdesignsanddataexclusionpracticesincognitivepsychologycaninadvertentlyintroducesystematicshadowbiasesinparticipantsamples AT sydnimnadler standardexperimentalparadigmdesignsanddataexclusionpracticesincognitivepsychologycaninadvertentlyintroducesystematicshadowbiasesinparticipantsamples AT justinngrady standardexperimentalparadigmdesignsanddataexclusionpracticesincognitivepsychologycaninadvertentlyintroducesystematicshadowbiasesinparticipantsamples AT dwightjkravitz standardexperimentalparadigmdesignsanddataexclusionpracticesincognitivepsychologycaninadvertentlyintroducesystematicshadowbiasesinparticipantsamples AT stephenrmitroff standardexperimentalparadigmdesignsanddataexclusionpracticesincognitivepsychologycaninadvertentlyintroducesystematicshadowbiasesinparticipantsamples |