| Summary: | The objectives of this Campbell systematic review were to respond to the following questions: Main Question: What are the effects of interventions implemented in developing countries on measures of students' enrollment, attendance, graduation, and progression? Supplemental Question: Within those studies that report the effects of an intervention on measures of students' enrollment, attendance, graduation or progression, what are the ancillary effects on learning outcomes as measured by students' test scores, grades, and other achievement measures? The sample includes 73 experiments and quasi‐experiments. Across all interventions, the average effect size was positive in direction for all outcomes, and was largest for enrollment (d=.18; 95% CI[.13‐.24]), attendance (d=.15, 95% CI [.10‐.20]), progression (d=.13, 95% CI [.08‐.18]), math (d=.16, 95% CI [.10‐.23]) and language (d=.18, 95% CI [.12‐.25]) outcomes. However, the results were not uniform across every study; given the large variation in programs, participants, settings and designs, there was no surprise that there was significant heterogeneity in effect sizes in these main analyses. Based on the evidence presented in this report, interventions that address getting children into school and keeping them there have, on average, positive effects. This is also true of learning outcomes reported within those same studies. Although effects could be considered small, they represent 3‐9% increases in positive outcomes compared to the control/comparison group in the studies. Policymakers would have to assess whether such outcomes are worth investments, given costs of implementation and how widespread the problem is that the intervention will address. Executive Summary BACKGROUND Education is considered critical to economic development and social welfare in developing nations. In light of compelling evidence that links expanded education systems and socioeconomic development while highlighting the importance of policies to offset inequality in access, governments and donor agencies have invested considerable funding to promote educational initiatives. Considerable funding for such initiatives has brought with it a concomitant increase in accountability and decision‐makers want to know if the funds they have put toward such programs are having positive impact. Concurrently, there has also been a rise in impact evaluations in the developing world, particularly in education. Given the importance of education, particularly to outcomes in the most economically challenged nations, the amount of interventions that have been implemented to address education in developing nations, and the increase in relevant controlled impact evaluations, the need for a systematic review seems clear. No systematic review of randomized controlled trials and quasi‐experiments of strategies in developing nations to get children into school (enrollment) and keep them there (attendance, persistence, continuation) has yet been reported, nor has any looked at supplemental outcomes focused on learning. By systematically gathering and analyzing rigorous research about the program effects of primary and secondary school enrollment and completion policies, our review will hopefully provide evidence to inform the next wave of funding, intervention and evaluation efforts in this area. OBJECTIVES For this project, our objectives were to respond to the following questions: • Main Question: What are the effects of interventions implemented in developing countries on measures of students' enrollment, attendance, graduation, and progression? • Supplemental Question: Within those studies that report the effects of an intervention on measures of students' enrollment, attendance, graduation or progression, what are the ancillary effects on learning outcomes as measured by students' test scores, grades, and other achievement measures? SEARCH STRATEGY Five main strategies were used to identify eligible reports: (1) electronic searches of bibliographic databases; (2) hand searches of relevant journals; (3) examining the citations of every retrieved report; (4) contacting the “informal college” of researchers working in the area; and (5) searches of the internet and specialized holdings. SELECTION CRITERIA To be eligible, studies had to: (1) assess the impact of an intervention that included primary or secondary school outcomes (Kindergarten‐12th grade in the U.S. context) relevant to the main research question; (2) use a randomized controlled trial (with or without baseline control), or a quasi‐experimental approach in which baseline controls on main outcomes were included; (3) be conducted in a country classified as a “low or middle income nation” by the World Bank at the time the intervention being studied was implemented; (4) include at least one quantifiable main outcome measure (enrollment, attendance, dropout, or progression); (5) be published or made available before December 2009, without regard to language or publication type; and (6) include data on participants from 1990 or beyond. If a study satisfied those criteria, we then examined it for quantifiable measures relevant to the supplemental question on learning outcomes. These studies are not representative, of course, of all evaluations that included learning outcomes, but only of those studies that included learning outcomes along with at least one quantifiable outcome of enrollment, attendance, dropout and progression. DATA COLLECTION AND ANALYSIS A preliminary instrument was designed to extract data on substantive and methodological characteristics from each of the reports. Standardized mean difference (Cohen's d) effect sizes were computed for the first effect reported in each study. Given the presumed heterogeneity of true effects in the population, analyses of effect sizes were estimated using random effects models. Main effects were analyzed for each main outcome reported: enrollment, attendance, dropout, and progression. Supplemental learning outcomes reported within the same studies were also coded; these included math, language, standardized assessment scores, and other achievement measures. Seven moderator analyses were also conducted. RESULTS The sample includes 73 experiments and quasi‐experiments. Across all interventions, the average effect size was positive in direction for all outcomes, and was largest for enrollment (d=.18; 95% CI[.13‐.24]), attendance (d=.15, 95% CI [.10‐.20]), progression (d=.13, 95% CI [.08‐.18]), math (d=.16, 95% CI [.10‐.23]) and language (d=.18, 95% CI [.12‐.25]) outcomes. However, the results were not uniform across every study; given the large variation in programs, participants, settings and designs, there was no surprise that there was significant heterogeneity in effect sizes in these main analyses. Examining only outcomes of enrollment and attendance (n=59), studies that focused on new schools and other infrastructure interventions (d=.44, 95% CI [.40‐.47]) reported the largest average effects. Studies that were conducted either in Europe or Central Asia (d=.58, 95% CI [.23‐.93]), or East Asia and the Pacific (d=.36, 95% CI [.25‐.48]), were also associated with larger average effects. AUTHORS' CONCLUSIONS Based on the evidence presented in this report, interventions that address getting children into school and keeping them there have, on average, positive effects. This is also true of learning outcomes reported within those same studies. Although effects could be considered small, they represent 3‐9% increases in positive outcomes compared to the control/comparison group in the studies. Policymakers would have to assess whether such outcomes are worth investments, given costs of implementation and how widespread the problem is that the intervention will address.
|
|---|