Logical and methodological issues affecting genetic studies of humans reported in top neuroscience journals
Genetics and neuroscience are two areas of science that pose particular methodological problems because they involve detecting weak signals (i.e., small effects) in noisy data. In recent years, increasing numbers of studies have attempted to bridge these disciplines by looking for genetic factors as...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Journal article |
| Published: |
Massachusetts Institute of Technology
2017
|
| _version_ | 1826257393138270208 |
|---|---|
| author | Grabitz, CR Button, KS Munafò, MR Newbury, DF Pernet, CR Thompson, PA Bishop, DVM |
| author_facet | Grabitz, CR Button, KS Munafò, MR Newbury, DF Pernet, CR Thompson, PA Bishop, DVM |
| author_sort | Grabitz, CR |
| collection | OXFORD |
| description | Genetics and neuroscience are two areas of science that pose particular methodological problems because they involve detecting weak signals (i.e., small effects) in noisy data. In recent years, increasing numbers of studies have attempted to bridge these disciplines by looking for genetic factors associated with individual differences in behaviour, cognition and brain structure or function. However, different methodological approaches to guarding against false positives have evolved in the two disciplines. To explore methodological issues affecting neurogenetic studies, we conducted an in-depth analysis of 30 consecutive articles in 12 top neuroscience journals that reported on genetic associations in non-clinical human samples. It was often difficult to estimate effect sizes in neuroimaging paradigms. Where effect sizes could be calculated, the studies reporting the largest effect sizes tended to have two features: (i) they had the smallest samples, and were generally underpowered to detect genetic effects; and (ii) they did not fully correct for multiple comparisons. Furthermore, only a minority of studies used statistical methods for multiple comparisons that took into account correlations between phenotypes or genotypes, and only nine studies included a replication sample, or explicitly set out to replicate a prior finding. Finally, presentation of methodological information was not standardized and was often distributed across Methods sections and Supplementary Material, making it challenging to assemble basic information from many studies. Space limits imposed by journals could mean that highly complex statistical methods were described in only a superficial fashion. In sum, methods which have become standard in the genetics literature – stringent statistical standards, use of large samples and replication of findings – are not always adopted when behavioural, cognitive or neuroimaging phenotypes are used, leading to an increased risk of false positive findings. Studies need to correct not just for the number of phenotypes collected, but also for number of genotypes examined, genetic models tested and subsamples investigated. The field would benefit from more widespread use of methods that take into account correlations between the factors corrected for, such as spectral decomposition, or permutation approaches. Replication should become standard practice; this, together with the need for larger sample sizes, will entail greater emphasis on collaboration between research groups. We conclude with some specific suggestions for standardized reporting in this area. |
| first_indexed | 2024-03-06T18:17:29Z |
| format | Journal article |
| id | oxford-uuid:051d3521-9238-4adb-92bf-208f4cc4ed13 |
| institution | University of Oxford |
| last_indexed | 2024-03-06T18:17:29Z |
| publishDate | 2017 |
| publisher | Massachusetts Institute of Technology |
| record_format | dspace |
| spelling | oxford-uuid:051d3521-9238-4adb-92bf-208f4cc4ed132022-03-26T08:55:23ZLogical and methodological issues affecting genetic studies of humans reported in top neuroscience journalsJournal articlehttp://purl.org/coar/resource_type/c_dcae04bcuuid:051d3521-9238-4adb-92bf-208f4cc4ed13Symplectic Elements at OxfordMassachusetts Institute of Technology2017Grabitz, CRButton, KSMunafò, MRNewbury, DFPernet, CRThompson, PABishop, DVMGenetics and neuroscience are two areas of science that pose particular methodological problems because they involve detecting weak signals (i.e., small effects) in noisy data. In recent years, increasing numbers of studies have attempted to bridge these disciplines by looking for genetic factors associated with individual differences in behaviour, cognition and brain structure or function. However, different methodological approaches to guarding against false positives have evolved in the two disciplines. To explore methodological issues affecting neurogenetic studies, we conducted an in-depth analysis of 30 consecutive articles in 12 top neuroscience journals that reported on genetic associations in non-clinical human samples. It was often difficult to estimate effect sizes in neuroimaging paradigms. Where effect sizes could be calculated, the studies reporting the largest effect sizes tended to have two features: (i) they had the smallest samples, and were generally underpowered to detect genetic effects; and (ii) they did not fully correct for multiple comparisons. Furthermore, only a minority of studies used statistical methods for multiple comparisons that took into account correlations between phenotypes or genotypes, and only nine studies included a replication sample, or explicitly set out to replicate a prior finding. Finally, presentation of methodological information was not standardized and was often distributed across Methods sections and Supplementary Material, making it challenging to assemble basic information from many studies. Space limits imposed by journals could mean that highly complex statistical methods were described in only a superficial fashion. In sum, methods which have become standard in the genetics literature – stringent statistical standards, use of large samples and replication of findings – are not always adopted when behavioural, cognitive or neuroimaging phenotypes are used, leading to an increased risk of false positive findings. Studies need to correct not just for the number of phenotypes collected, but also for number of genotypes examined, genetic models tested and subsamples investigated. The field would benefit from more widespread use of methods that take into account correlations between the factors corrected for, such as spectral decomposition, or permutation approaches. Replication should become standard practice; this, together with the need for larger sample sizes, will entail greater emphasis on collaboration between research groups. We conclude with some specific suggestions for standardized reporting in this area. |
| spellingShingle | Grabitz, CR Button, KS Munafò, MR Newbury, DF Pernet, CR Thompson, PA Bishop, DVM Logical and methodological issues affecting genetic studies of humans reported in top neuroscience journals |
| title | Logical and methodological issues affecting genetic studies of humans reported in top neuroscience journals |
| title_full | Logical and methodological issues affecting genetic studies of humans reported in top neuroscience journals |
| title_fullStr | Logical and methodological issues affecting genetic studies of humans reported in top neuroscience journals |
| title_full_unstemmed | Logical and methodological issues affecting genetic studies of humans reported in top neuroscience journals |
| title_short | Logical and methodological issues affecting genetic studies of humans reported in top neuroscience journals |
| title_sort | logical and methodological issues affecting genetic studies of humans reported in top neuroscience journals |
| work_keys_str_mv | AT grabitzcr logicalandmethodologicalissuesaffectinggeneticstudiesofhumansreportedintopneurosciencejournals AT buttonks logicalandmethodologicalissuesaffectinggeneticstudiesofhumansreportedintopneurosciencejournals AT munafomr logicalandmethodologicalissuesaffectinggeneticstudiesofhumansreportedintopneurosciencejournals AT newburydf logicalandmethodologicalissuesaffectinggeneticstudiesofhumansreportedintopneurosciencejournals AT pernetcr logicalandmethodologicalissuesaffectinggeneticstudiesofhumansreportedintopneurosciencejournals AT thompsonpa logicalandmethodologicalissuesaffectinggeneticstudiesofhumansreportedintopneurosciencejournals AT bishopdvm logicalandmethodologicalissuesaffectinggeneticstudiesofhumansreportedintopneurosciencejournals |