Evaluator: Bloomberg Philanthropies; MGT Consulting; Borman, G., Arizona State University; Kane, T., Harvard University; McEachin. A, ETS Research Institute

Evaluation Methods:

The evaluation used a quasi-experimental Value Added Model (VAM) to assess Summer Boost’s impact on student growth by comparing predicted student growth without the program (based on past test scores, student characteristics, etc.) to how students actually grew, attributing unexpected growth to the program. Building on 2023 data analysis, researchers were able to analyze the impact on participating students for two years in a row, as well as the lasting impact for students who participated in 2023 but not in 2024. Researchers also conducted a Difference-in-Differences analysis to compare with the 2023 evaluation, providing a comparison in achievement gains for participants and non-participants. Researchers analyzed multiple datasets, including two years of MAP Growth and i-Ready assessment data, student characteristic information, attendance data, pre- and post-program assessments, and program information. The dataset included a total of around 160,000 students across 206 grantees, including 35,000 Summer Boost participants and 125,000 comparison students. Limitations included missing assessment data, smaller subgroup samples, overlapping student characteristics, and limits to generalizability.

Evaluation Type: Quasi-experimental;Non-experimental

Summary of Outcomes:

The evaluation found that Summer Boost 2024 had positive, statistically significant academic impacts. Overall, participating students gained approximately five additional weeks of math learning and two additional weeks of English Language Arts (ELA) learning compared with students in the same schools and grades who did not participate in the program. These impacts were measured through Fall 2024 i-Ready and MAP Growth assessments, taken weeks or months after the summer program ended. 

The evaluation also examined multi-year participation. Students who attended Summer Boost in both summers 2023 and 2024 gained about two additional weeks of math learning compared with students who attended only one year. Additionally, students who participated in 2023 but not in 2024 still showed signs of positive math growth more than a year later, retaining about two weeks of additional math learning compared with students who did not participate in either year. Outcomes for math were directional and not statistically significant, suggesting that additional years of data are needed to increase confidence in these effects. Lasting effects were not observed in ELA learning. 

The evaluation also highlighted that Summer Boost worked across student characteristics, cities, achievement levels, and grade levels. Students eligible for Free or Reduced Price Lunch, students in special education, and English Language Learners all showed growth, with the strongest and most consistent subgroup evidence in math learning. For instance, compared to their non-participating, similarly situated peers, FRPL participants gained 4.2 additional weeks of math learning, special education participants gained an extra 7.6 weeks of math learning, and English Language Learner participants gained 5.5 additional weeks of math learning. Regarding ELA, FRPL-eligible participants gained an additional week of reading, while special education participants gained three weeks compared to their matched non-participating peers. English Language Learner results for ELA were not statistically significant. 

The evaluation further showed that students with lower baseline achievement levels demonstrated statistically significant math growth. The majority of students participating in Summer Boost 2024 started off in lower average baseline achievement levels (60%), including lowest and below average baseline achievement levels, and through the program, participating students in these achievement levels received, respectively, 4.2 and 8.3 additional weeks of math learning compared to non-participating peers. These results suggest that the program supports students who could benefit the most from summer learning. Students across grades also grew in math learning, with upper grades showing signs of accelerated academic growth. In ELA, only 2nd graders demonstrated statistically significant reading growth. 

The evaluation linked stronger outcomes to higher program attendance and smaller class sizes. The average 2024 attendance rate was 79%, up from 75% in 2023. Regarding math scores, students who had an attendance rate of 70% or more experienced an approximate 0.10-point gain on a standardized scale, compared to approximately 0.02 for students attending only 30% of program days. In ELA, students attending 70% of days experienced an approximate 0.05-point gain, compared to a 0.01-point gain for students attending only 30% of programming. Additionally, students attending programs with a smaller student-teacher ratio of 15:1 or lower also saw greater gains in weeks of math learning than those attending programs with ratios of 16:1 or above. 

Pre- and post-assessment results provide further evidence of program impact, with the percentage of students scoring “proficient” on Lavinia math assessments increasing by 19.75 percentage points and on ELA assessments by 17.83 percentage points.