Natural experiments for the evaluation of place-based public health interventions: a methodology scoping review
IntroductionPlace-based public health evaluations are increasingly making use of natural experiments. This scoping review aimed to provide an overview of the design and use of natural experiment evaluations (NEEs), and an assessment of the plausibility of the as-if randomization assumption.MethodsA...
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-06-01
|
| Series: | Frontiers in Public Health |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fpubh.2023.1192055/full |
| _version_ | 1797797797537251328 |
|---|---|
| author | Patricia N. Albers Chiara Rinaldi Heather Brown Kate E. Mason Kate E. Mason Katrina d'Apice Elizabeth McGill Cheryl McQuire Peter Craig Anthony A. Laverty Morgan Beeson Mhairi Campbell Matt Egan Marcia Gibson Maxwell Fuller Amy Dillon David Taylor-Robinson Russell Jago Russell Jago Russell Jago Kate Tilling Kate Tilling Benjamin Barr Falko F. Sniehotta Falko F. Sniehotta Matthew Hickman Christopher J. Millett Frank de Vocht Frank de Vocht |
| author_facet | Patricia N. Albers Chiara Rinaldi Heather Brown Kate E. Mason Kate E. Mason Katrina d'Apice Elizabeth McGill Cheryl McQuire Peter Craig Anthony A. Laverty Morgan Beeson Mhairi Campbell Matt Egan Marcia Gibson Maxwell Fuller Amy Dillon David Taylor-Robinson Russell Jago Russell Jago Russell Jago Kate Tilling Kate Tilling Benjamin Barr Falko F. Sniehotta Falko F. Sniehotta Matthew Hickman Christopher J. Millett Frank de Vocht Frank de Vocht |
| author_sort | Patricia N. Albers |
| collection | DOAJ |
| description | IntroductionPlace-based public health evaluations are increasingly making use of natural experiments. This scoping review aimed to provide an overview of the design and use of natural experiment evaluations (NEEs), and an assessment of the plausibility of the as-if randomization assumption.MethodsA systematic search of three bibliographic databases (Pubmed, Web of Science and Ovid-Medline) was conducted in January 2020 to capture publications that reported a natural experiment of a place-based public health intervention or outcome. For each, study design elements were extracted. An additional evaluation of as-if randomization was conducted by 12 of this paper's authors who evaluated the same set of 20 randomly selected studies and assessed ‘as-if ' randomization for each.Results366 NEE studies of place-based public health interventions were identified. The most commonly used NEE approach was a Difference-in-Differences study design (25%), followed by before-after studies (23%) and regression analysis studies. 42% of NEEs had likely or probable as-if randomization of exposure (the intervention), while for 25% this was implausible. An inter-rater agreement exercise indicated poor reliability of as-if randomization assignment. Only about half of NEEs reported some form of sensitivity or falsification analysis to support inferences.ConclusionNEEs are conducted using many different designs and statistical methods and encompass various definitions of a natural experiment, while it is questionable whether all evaluations reported as natural experiments should be considered as such. The likelihood of as-if randomization should be specifically reported, and primary analyses should be supported by sensitivity analyses and/or falsification tests. Transparent reporting of NEE designs and evaluation methods will contribute to the optimum use of place-based NEEs. |
| first_indexed | 2024-03-13T03:54:49Z |
| format | Article |
| id | doaj.art-5f65b05348c247ac9b814521c24c3021 |
| institution | Directory Open Access Journal |
| issn | 2296-2565 |
| language | English |
| last_indexed | 2024-03-13T03:54:49Z |
| publishDate | 2023-06-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Public Health |
| spelling | doaj.art-5f65b05348c247ac9b814521c24c30212023-06-22T07:52:50ZengFrontiers Media S.A.Frontiers in Public Health2296-25652023-06-011110.3389/fpubh.2023.11920551192055Natural experiments for the evaluation of place-based public health interventions: a methodology scoping reviewPatricia N. Albers0Chiara Rinaldi1Heather Brown2Kate E. Mason3Kate E. Mason4Katrina d'Apice5Elizabeth McGill6Cheryl McQuire7Peter Craig8Anthony A. Laverty9Morgan Beeson10Mhairi Campbell11Matt Egan12Marcia Gibson13Maxwell Fuller14Amy Dillon15David Taylor-Robinson16Russell Jago17Russell Jago18Russell Jago19Kate Tilling20Kate Tilling21Benjamin Barr22Falko F. Sniehotta23Falko F. Sniehotta24Matthew Hickman25Christopher J. Millett26Frank de Vocht27Frank de Vocht28Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, United KingdomDepartment of Public Health, Environments and Society, London School of Hygiene and Tropical Medicine, London, United KingdomHealth Research, Lancaster University, Lancaster, United KingdomDepartment of Public Health Policy and Systems, University of Liverpool, Liverpool, United KingdomCentre for Health Policy, University of Melbourne, Parkville, VIC, AustraliaPopulation Health Sciences, Bristol Medical School, University of Bristol, Bristol, United KingdomDepartment of Public Health, Environments and Society, London School of Hygiene and Tropical Medicine, London, United KingdomPopulation Health Sciences, Bristol Medical School, University of Bristol, Bristol, United KingdomMRC/CSO Social and Public Health Sciences Unit, School of Health and Wellbeing, University of Glasgow, Glasgow, United KingdomSchool of Public Health, Imperial College London, London, United KingdomNewcastle University Business School, Newcastle University, Newcastle upon Tyne, United KingdomMRC/CSO Social and Public Health Sciences Unit, School of Health and Wellbeing, University of Glasgow, Glasgow, United KingdomDepartment of Public Health, Environments and Society, London School of Hygiene and Tropical Medicine, London, United KingdomMRC/CSO Social and Public Health Sciences Unit, School of Health and Wellbeing, University of Glasgow, Glasgow, United KingdomPopulation Health Sciences, Bristol Medical School, University of Bristol, Bristol, United KingdomPopulation Health Sciences, Bristol Medical School, University of Bristol, Bristol, United KingdomDepartment of Public Health, Policy and Systems. University of Liverpool, Liverpool, United KingdomPopulation Health Sciences, Bristol Medical School, University of Bristol, Bristol, United Kingdom0The National Institute for Health Research, Applied Research Collaboration West (NIHR ARC West), University Hospitals Bristol and Weston NHS Foundation Trust, Bristol, United Kingdom1Centre for Exercise, Nutrition and Health Sciences, School for Policy Studies, University of Bristol, Bristol, United KingdomPopulation Health Sciences, Bristol Medical School, University of Bristol, Bristol, United Kingdom2MRC Integrative Epidemiology Unit, University of Bristol, Bristol, United Kingdom3Institute of Population Health, University of Liverpool, Liverpool, United Kingdom4NIHR Policy Research Unit Behavioural Science, Newcastle University, Newcastle upon Tyne, United Kingdom5Department of Public Health, Social and Preventive Medicine, Medical Faculty Mannheim, Heidelberg University, Heidelberg, GermanyPopulation Health Sciences, Bristol Medical School, University of Bristol, Bristol, United KingdomSchool of Public Health, Imperial College London, London, United KingdomPopulation Health Sciences, Bristol Medical School, University of Bristol, Bristol, United Kingdom0The National Institute for Health Research, Applied Research Collaboration West (NIHR ARC West), University Hospitals Bristol and Weston NHS Foundation Trust, Bristol, United KingdomIntroductionPlace-based public health evaluations are increasingly making use of natural experiments. This scoping review aimed to provide an overview of the design and use of natural experiment evaluations (NEEs), and an assessment of the plausibility of the as-if randomization assumption.MethodsA systematic search of three bibliographic databases (Pubmed, Web of Science and Ovid-Medline) was conducted in January 2020 to capture publications that reported a natural experiment of a place-based public health intervention or outcome. For each, study design elements were extracted. An additional evaluation of as-if randomization was conducted by 12 of this paper's authors who evaluated the same set of 20 randomly selected studies and assessed ‘as-if ' randomization for each.Results366 NEE studies of place-based public health interventions were identified. The most commonly used NEE approach was a Difference-in-Differences study design (25%), followed by before-after studies (23%) and regression analysis studies. 42% of NEEs had likely or probable as-if randomization of exposure (the intervention), while for 25% this was implausible. An inter-rater agreement exercise indicated poor reliability of as-if randomization assignment. Only about half of NEEs reported some form of sensitivity or falsification analysis to support inferences.ConclusionNEEs are conducted using many different designs and statistical methods and encompass various definitions of a natural experiment, while it is questionable whether all evaluations reported as natural experiments should be considered as such. The likelihood of as-if randomization should be specifically reported, and primary analyses should be supported by sensitivity analyses and/or falsification tests. Transparent reporting of NEE designs and evaluation methods will contribute to the optimum use of place-based NEEs.https://www.frontiersin.org/articles/10.3389/fpubh.2023.1192055/fullpublic healthpublic health policynatural experimentsquasi experimentsevaluationsplace-based |
| spellingShingle | Patricia N. Albers Chiara Rinaldi Heather Brown Kate E. Mason Kate E. Mason Katrina d'Apice Elizabeth McGill Cheryl McQuire Peter Craig Anthony A. Laverty Morgan Beeson Mhairi Campbell Matt Egan Marcia Gibson Maxwell Fuller Amy Dillon David Taylor-Robinson Russell Jago Russell Jago Russell Jago Kate Tilling Kate Tilling Benjamin Barr Falko F. Sniehotta Falko F. Sniehotta Matthew Hickman Christopher J. Millett Frank de Vocht Frank de Vocht Natural experiments for the evaluation of place-based public health interventions: a methodology scoping review Frontiers in Public Health public health public health policy natural experiments quasi experiments evaluations place-based |
| title | Natural experiments for the evaluation of place-based public health interventions: a methodology scoping review |
| title_full | Natural experiments for the evaluation of place-based public health interventions: a methodology scoping review |
| title_fullStr | Natural experiments for the evaluation of place-based public health interventions: a methodology scoping review |
| title_full_unstemmed | Natural experiments for the evaluation of place-based public health interventions: a methodology scoping review |
| title_short | Natural experiments for the evaluation of place-based public health interventions: a methodology scoping review |
| title_sort | natural experiments for the evaluation of place based public health interventions a methodology scoping review |
| topic | public health public health policy natural experiments quasi experiments evaluations place-based |
| url | https://www.frontiersin.org/articles/10.3389/fpubh.2023.1192055/full |
| work_keys_str_mv | AT patricianalbers naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT chiararinaldi naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT heatherbrown naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT kateemason naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT kateemason naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT katrinadapice naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT elizabethmcgill naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT cherylmcquire naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT petercraig naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT anthonyalaverty naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT morganbeeson naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT mhairicampbell naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT mattegan naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT marciagibson naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT maxwellfuller naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT amydillon naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT davidtaylorrobinson naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT russelljago naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT russelljago naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT russelljago naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT katetilling naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT katetilling naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT benjaminbarr naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT falkofsniehotta naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT falkofsniehotta naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT matthewhickman naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT christopherjmillett naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT frankdevocht naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview AT frankdevocht naturalexperimentsfortheevaluationofplacebasedpublichealthinterventionsamethodologyscopingreview |