Afterschool Impacts Database
Evaluation 101
Learn more about the basics of evaluation, as well as how to find an evaluator for your program.
Afterschool Evaluation
View Afterschool Alliance resources, including a glossary of terms used in the database, evaluation-related blogs, webinars and more.
Additional Resources
Our list of evaluation resources from other organizations, including how to collect and work with data.
Afterschool Research
Want to find what we know about afterschool programs more broadly, not just individual programs? Head to our Afterschool Research page!
Examining the Long-Term Effects of the Horizons National Student Enrichment Program on Student Academic Outcomes
Year Published: 2018
This evaluation of the Horizons National Student Enrichment Program—a multi-year intensive summer learning program with year-round supports—found long-term, positive impacts of participation. When compared to a control group, Horizons students who participated in programming for at least four years had higher school-day attendance rates and lower rates of disciplinary referrals, outperformed the comparison group on math and science elementary assessments, had higher GPAs, and were less likely to repeat a grade during middle and high school.
Program Name: Horizons Student Enrichment Program
Program Description:
Horizons offers multi-year, intensive summer programming for low-income youth to help combat learning loss over the summer. Students typically enroll in the program starting in kindergarten, and are invited back each year through at least the 8th grade, with some programs going through high school. The program intentionally takes place at private school, college, or university campuses to provide an environment that is different than students’ schools. During the summer, Horizons students take part in literacy and math activities, swimming, art, and other various recreation and enrichment opportunities. Horizons serves nearly 6,000 students in 59 programs across 19 states.
Scope of the Evaluation: National
Program Type: Summer
Grade level: Elementary School, Middle School, High School
Program Demographics:
This study has three separate analysis groups for elementary, middle, and high school students. For the elementary school sample, 46 percent identified as Latino, 44 percent identified as African American, and 10 percent identified as White. 30 percent were English language learners, and just under 7 percent received special education services. Among the middle school sample, 48 percent identified as Latino, 44 percent identified as African American, and 6 percent as White. 25 percent were English language learners, and 8 percent received special education services. Demographics were similar for the high school sample, with 47 percent identifying as Latino, 46 percent identifying as African American, and 3 percent identifying as White. 16 percent were English language learners, and 9 percent received special education services.
Program Website: https://www.horizonsnational.org/
Evaluator: Concentric Research & Evaluation
Evaluation Methods:
This evaluation used a quasi-experimental design to compare long-term Horizons students (those who participated for at least four consecutive years) to a similar group of comparison students in the same school districts who did not attend programming. School administrative data from school years 2010-11 through 2015-16 was used in the analysis to compare the two groups of students. Data included student demographic information, attendance records, disciplinary referrals, standardized test scores, credits earned, and GPAs. Horizons students were matched with comparison students using propensity score matching, and multiple regression techniques were used to measure program effects.
Evaluation Type: Quasi-experimental
Summary of Outcomes:
This evaluation of the Horizons program found positive academic and behavioral impacts among long-term Horizons participants who participated in programming for at least four consecutive summers. Regarding the academic impacts, the evaluation found that Horizons students outperformed the control group of students on math (10.5 percentage point difference) and science (10.4 percentage point difference) standardized assessments during elementary school. While the difference in proficiency rates for reading was not significant, the rate was still higher among Horizons students (6.7 percentage point difference). Middle school students also scored better than nonparticipants, however, the differences were not significant.
Horizons students in high school earned better grades and more credits toward graduation compared to nonparticipants. Ninth grade Horizons students had higher GPAs, with the average GPA for Horizons students being a 2.9 (B average) while the comparison group was a 2.3 (C+ average). At the end of 9th grade, Horizons students had earned, on average, one additional credit towards graduation compared to the control group, and by the end of 10th grade, had earned almost two additional credits. Finally, during middle and early high school, Horizons students were significantly less likely to repeat a grade. Horizons students in elementary school were also less likely to repeat a grade, however, the differences were not statistically significant.
Regarding the behavioral impacts of participation, Horizons students had higher school-day attendance rates and were less likely to be chronically absent compared to the control group. Both elementary and middle school participants had attendance rates that were 1.1 percentage points higher than the comparison group, equivalent to two additional days of school, a statistically significant difference. Differences in attendance rates were even more pronounced among high school students, with the rate for Horizons students being 3.6 percentage points higher than for nonparticipants, equivalent to 6.5 more days of school. Rates of chronic absenteeism were also significantly lower than those of comparison students in elementary school (3.8 percent vs 9.3 percent), middle school (8.6 percent vs. 13.9 percent), and high school (13.1 percent vs. 27.6 percent).
In addition, Horizons students had significantly lower rates of disciplinary referrals during middle and early high school. In middle school, 12.1 percent of Horizons participants received one disciplinary referral by the 2015-16 school year, compared to 17 percent of comparison students. In high school, 16.2 percent of Horizons students earned one disciplinary referral compared to 26.7 percent of comparison students.
Date Added: June 9, 2022
