Learn more about the basics of evaluation, as well as how to find an evaluator for your program.
View Afterschool Alliance resources, including a glossary of terms used in the database, evaluation-related blogs, webinars and more.
Our list of evaluation resources from other organizations, including how to collect and work with data.
Want to find what we know about afterschool programs more broadly, not just individual programs? Head to our Afterschool Research page!
Year Published: 2016
A randomized controlled study following 5,000 low-income, predominantly African-American and Hispanic students from third to seventh grade in five urban school districts located in Boston, MA; Dallas, TX; Duval County, FL; Pittsburgh, PA and Rochester, NY, assessing the impacts of no-cost, voluntary summer learning programs on academic performance and social and emotional skills. Students who had high attendance in the summer programming saw significant near term benefits (gains in the fall after the summer program) and long-term benefits (gains seen through the following spring after the summer program) in math after summer programming in 2013 and 2014, near and long-term benefits in language arts after summer 2014, and positive benefits to their social and emotional skills after summer 2014.
Scope of the Evaluation: Multi-city
Program Type: Summer
Location: Boston, MA; Dallas, TX; Duval County, FL; Pittsburgh, PA and Rochester, NY.
Community Type: Urban
Grade level: Elementary School, Middle School
Program Demographics: Eighty-six percent of students were eligible for free or reduced price lunch. Forty-nine percent were African American, 38 percent were Hispanic, 13 percent were other racial/ethnic groups, and 29 percent were English language learners.
Evaluator: Augustine, C. H., McCombs, J. S., Pane, J.F., Schwartz, H. L., Schweig, J., McEachin, A., & Siler-Evans, K. The RAND Corporation.
Evaluation Methods: At the start of this longitudinal randomized controlled trial during the spring of 2013, more than 5,000 third grade students were randomly selected to participate in the study, 3,194 of whom were assigned to the treatment group (participating in a summer learning program) and 2,445 of whom were assigned to the control group (students who did not participate in summer programming). In order to select students for the study, school districts advertised and recruited students to apply for a spot in the voluntary summer learning programming, and through a lottery program, students were randomly selected to participate in two summers of programming, with the remaining students serving as the control group. The study included direct program observation, teacher surveys, district data, as well as academic and social emotional assessments. This report continued to track students during the summer of 2014 and spring of 2015. Student academic performance, social and emotional skills and program attendance were tracked through seventh grade. Outcomes for the two groups were analyzed and compared as well as outcomes for high-attenders in comparison with all other students.
Evaluation Type: Experimental
Summary of Outcomes: The study found that students with high attendance in the summer programming saw significant near term benefits (gains in the fall after the summer program) and long-term benefits (gains seen through the following spring after the summer program) in math after summer programming in 2013 and 2014, near and long-term benefits in language arts after summer 2014, and positive benefits to their social and emotional skills after summer 2014.
When programs focused specifically on math or language arts, students saw lasting positive gains in these subjects. Students who received at least 25 hours of mathematics instruction or 34 hours of language arts outperformed students with less instruction time on subject-specific fall assessments. The estimated gains were between 23 percent and 31 percent of typical annual gains in mathematics, and 14 percent and 23 percent of typical annual gains in language arts.
The study also found that consecutive attendance in summer programs has a positive impact on students’ academic performance. Students who had a high level of attendance (20 days or more) during both the 2013 and 2014 summers saw greater gains in the math and language arts assessments compared to their non-participating peers.
The only significant results from the causal analysis between the control and treatment groups were short term impacts on math performance, and there was no clear evidence that offering two summers of programming had greater impacts on math performance than one summer. However, this is not surprising, given the fact that only 50 percent of treatment students participated in programming during summer of 2014. This decrease in participation is partially because 11 percent of students left the district before the second summer.
Associated Evaluation: http://www.rand.org/pubs/research_reports/RR1557.html
Date Added: December 19, 2016