How to Make Nothing Out of Something: Analyses of the Impact of Study Sampling and Statistical Interpretation in Misleading Meta-Analytic Conclusions
The limited resource model states that self-control is governed by a relatively finite set of inner resources on which people draw when exerting willpower. Once self-control resources have been used up or depleted, they are less available for other self-control tasks, leading to a decrement in subse...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2016-10-01
|
| Series: | Frontiers in Psychology |
| Subjects: | |
| Online Access: | http://journal.frontiersin.org/Journal/10.3389/fpsyg.2016.01639/full |
| _version_ | 1811237188869816320 |
|---|---|
| author | Michael Robert Cunningham Roy F. Baumeister |
| author_facet | Michael Robert Cunningham Roy F. Baumeister |
| author_sort | Michael Robert Cunningham |
| collection | DOAJ |
| description | The limited resource model states that self-control is governed by a relatively finite set of inner resources on which people draw when exerting willpower. Once self-control resources have been used up or depleted, they are less available for other self-control tasks, leading to a decrement in subsequent self-control success. The depletion effect has been studied for over 20 years, tested or extended in more than 600 studies, and supported in an independent meta-analysis (Hagger, Wood, Stiff, and Chatzisarantis, 2010). Meta-analyses are supposed to reduce bias in literature reviews. Carter, Kofler, Forster, and McCullough’s (2015) meta-analysis, by contrast, included a series of questionable decisions involving sampling, methods, and data analysis. We provide quantitative analyses of key sampling issues: exclusion of many of the best depletion studies based on idiosyncratic criteria and the emphasis on mini meta-analyses with low statistical power as opposed to the overall depletion effect. We discuss two key methodological issues: failure to code for research quality, and the quantitative impact of weak studies by novice researchers. We discuss two key data analysis issues: questionable interpretation of the results of trim and fill and funnel plot asymmetry test procedures, and the use and misinterpretation of the untested Precision Effect Test [PET] and Precision Effect Estimate with Standard Error (PEESE) procedures. Despite these serious problems, the Carter et al. meta-analysis results actually indicate that there is a real depletion effect – contrary to their title. |
| first_indexed | 2024-04-12T12:22:28Z |
| format | Article |
| id | doaj.art-240a3fc0f92c49c1bb007d7b19549316 |
| institution | Directory Open Access Journal |
| issn | 1664-1078 |
| language | English |
| last_indexed | 2024-04-12T12:22:28Z |
| publishDate | 2016-10-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Psychology |
| spelling | doaj.art-240a3fc0f92c49c1bb007d7b195493162022-12-22T03:33:15ZengFrontiers Media S.A.Frontiers in Psychology1664-10782016-10-01710.3389/fpsyg.2016.01639197757How to Make Nothing Out of Something: Analyses of the Impact of Study Sampling and Statistical Interpretation in Misleading Meta-Analytic ConclusionsMichael Robert Cunningham0Roy F. Baumeister1University of LouisvilleFlorida State UniversityThe limited resource model states that self-control is governed by a relatively finite set of inner resources on which people draw when exerting willpower. Once self-control resources have been used up or depleted, they are less available for other self-control tasks, leading to a decrement in subsequent self-control success. The depletion effect has been studied for over 20 years, tested or extended in more than 600 studies, and supported in an independent meta-analysis (Hagger, Wood, Stiff, and Chatzisarantis, 2010). Meta-analyses are supposed to reduce bias in literature reviews. Carter, Kofler, Forster, and McCullough’s (2015) meta-analysis, by contrast, included a series of questionable decisions involving sampling, methods, and data analysis. We provide quantitative analyses of key sampling issues: exclusion of many of the best depletion studies based on idiosyncratic criteria and the emphasis on mini meta-analyses with low statistical power as opposed to the overall depletion effect. We discuss two key methodological issues: failure to code for research quality, and the quantitative impact of weak studies by novice researchers. We discuss two key data analysis issues: questionable interpretation of the results of trim and fill and funnel plot asymmetry test procedures, and the use and misinterpretation of the untested Precision Effect Test [PET] and Precision Effect Estimate with Standard Error (PEESE) procedures. Despite these serious problems, the Carter et al. meta-analysis results actually indicate that there is a real depletion effect – contrary to their title.http://journal.frontiersin.org/Journal/10.3389/fpsyg.2016.01639/fullSelf-ControlEgo DepletionMeta-Analysis as Topictrim and fillstrength depletionPrecision Effects Test |
| spellingShingle | Michael Robert Cunningham Roy F. Baumeister How to Make Nothing Out of Something: Analyses of the Impact of Study Sampling and Statistical Interpretation in Misleading Meta-Analytic Conclusions Frontiers in Psychology Self-Control Ego Depletion Meta-Analysis as Topic trim and fill strength depletion Precision Effects Test |
| title | How to Make Nothing Out of Something: Analyses of the Impact of Study Sampling and Statistical Interpretation in Misleading Meta-Analytic Conclusions |
| title_full | How to Make Nothing Out of Something: Analyses of the Impact of Study Sampling and Statistical Interpretation in Misleading Meta-Analytic Conclusions |
| title_fullStr | How to Make Nothing Out of Something: Analyses of the Impact of Study Sampling and Statistical Interpretation in Misleading Meta-Analytic Conclusions |
| title_full_unstemmed | How to Make Nothing Out of Something: Analyses of the Impact of Study Sampling and Statistical Interpretation in Misleading Meta-Analytic Conclusions |
| title_short | How to Make Nothing Out of Something: Analyses of the Impact of Study Sampling and Statistical Interpretation in Misleading Meta-Analytic Conclusions |
| title_sort | how to make nothing out of something analyses of the impact of study sampling and statistical interpretation in misleading meta analytic conclusions |
| topic | Self-Control Ego Depletion Meta-Analysis as Topic trim and fill strength depletion Precision Effects Test |
| url | http://journal.frontiersin.org/Journal/10.3389/fpsyg.2016.01639/full |
| work_keys_str_mv | AT michaelrobertcunningham howtomakenothingoutofsomethinganalysesoftheimpactofstudysamplingandstatisticalinterpretationinmisleadingmetaanalyticconclusions AT royfbaumeister howtomakenothingoutofsomethinganalysesoftheimpactofstudysamplingandstatisticalinterpretationinmisleadingmetaanalyticconclusions |