Evaluator: Modestino, A. & Paulsen, R.

Evaluation Methods:

This randomized control trial assesses the impact of SYEP participation on high school graduation and dropout rates. The study relies on the Action for Boston Community Development (ABCD) lottery system, which assigns a SYEP job offer to applicants at random. The treatment group is students selected to participate, and the control group is youth who applied but were not chosen to participate. Of the 4,235 youth who applied, 1,186 were offered a job. Evaluators used a matching procedure to match study participants to state-level administrative school records from the Massachusetts Department of Elementary and Secondary Education (DESE). These state-level profiles included information on students' attendance, course grades, standardized test scores, and high school graduation time frames. In order to track the outcomes of participation in SYEP, evaluators analyzed participants' DESE profiles over a 4 year-observation period following participation in SYEP, starting in the 2015-2016 school year and ending in the 2018-2019 school year. Researchers performed a mediation analysis to understand the mechanisms driving improved graduation and dropout rates.

Evaluation Type: Experimental

Summary of Outcomes:

This study found that students selected to participate in SYEP were significantly more likely to graduate on time during the four-year post-program observation period than non-participating students, largely attributable to improved attendance and academic performance.

Compared to the control group, those selected to participate in SYEP were 4.4 percentage points more likely to graduate from high school on time during the four-year post-program observation period – a 7% improvement over the control group 's mean graduation rate of 63.4%. In addition, students in the treatment group experienced a 2.5 percentage-point reduction in the likelihood of dropping out of high school during the observation window, with dropout rates lowest one year after selection to participate in SYEP.

Evaluators conducted a mediation analysis to understand the factors driving improved on-time high school graduation and reduced dropout rates among the treatment group. The analysis finds that improvement in attendance is positively associated with a greater likelihood of high school graduation. Students selected to participate in SYEP experienced a 2.4 percentage-point increase in attendance rates, translating to an additional 3.4 days of school. The program 's impact on attendance also significantly affected chronic absenteeism rates, revealing that SYEP participants were 5.9 percentage points less likely to experience chronic absenteeism one year into the observation period. Researchers posit that SYEP may be a preventative measure in reducing chronic absenteeism and improving graduation rates among older youth.

Regarding academic performance, the program has a small but statistically significant positive impact on GPA one year after participation. After controlling for individual and school characteristics, the results show that the overall GPA of students in the treatment group was 0.13 points higher than that of the control group, corresponding to a 6.8% improvement.

When testing whether improved attendance and academic performance persisted beyond one year after participation, researchers found that impacts on attendance, including increased overall days attended and reduced unexcused absences, persisted in year two; however, the program 's effect on chronic absenteeism and GPA disappeared the second year after participation. However, students who applied and were accepted for a second summer continued to see positive impacts on their GPA into the second observation year.

Researchers also explored behavioral shifts associated with SYEP participation as mechanisms that explain improved on-time graduation and reduced dropout rates. Using self-reported survey data from the 663 treatment and 664 control group participants, evaluators found that treatment group participants were 15.2 percentage points more likely to report gaining a mentor, 4.3 percentage points more likely to indicate they were saving for tuition, and also were more likely to report having foundational skills like punctuality, organization, emotional regulation, and asking for help, compared to their non-participating peers. Although cautioning these findings as exploratory in nature, researchers note that gaining a mentor, punctuality, and emotional regulation were strongly correlated with on-time high school graduation.