Learn more about the basics of evaluation, as well as how to find an evaluator for your program.
View Afterschool Alliance resources, including a glossary of terms used in the database, evaluation-related blogs, webinars and more.
Our list of evaluation resources from other organizations, including how to collect and work with data.
Want to find what we know about afterschool programs more broadly, not just individual programs? Head to our Afterschool Research page!
Year Published: 2022
This quasi-experimental evaluation of the Please Call Me Mister (PCMM) program—a 4-year afterschool program in Franklin County, Kentucky, for middle and high school African American and Hispanic males focused on violence prevention and positive youth development—found several statistically significant positive outcomes for PCMM participants, including a decrease in carrying weapons and lower levels of alcohol consumption. PCMM participants also saw increases in resiliency and a decrease in levels of depression.
Program Name: Please Call Me Mister
Please Call Me Mister (PCMM) is a 4-year program for African American and Hispanic male youth run by the Promising Youth Center for Excellence at Kentucky State University in Franklin County, Kentucky. PCMM is for African American and Hispanic male youth between the ages of 10-17 who have experienced domestic violence, abuse, mental health issues, involvement in the justice system, and could improve their academic performance. The program aims to decrease anti-social behavior and truancy, improve academic performance, and boost parent/family engagement through mentorship and small group enrichment.
Scope of the Evaluation: Local
Program Type: Afterschool
Location: Franklin County, Kentucky
Community Type: Rural, Urban, Suburban
Grade level: Middle School, High School
African American and Hispanic individuals who identify as male
Program Website: https://www.kysu.edu/calendar/index.php?eID=139
This evaluation was a quasi-experimental cross-sectional study where PCMM participants took a baseline survey at their program site in November 2017, and electronically from their homes in November 2020 (due to the COVID-19 pandemic). The survey included questions from the Kentucky Youth Risk Behavior Surveillance Survey (YRBSS), Kutcher Adolescent Depression Scale (KADS), and The Child and Youth Resilience Measure, and all data from 2017 and 2020 were analyzed by IBM SPSS Statistics (Version 27.0). Survey focus areas were: violence and victimization, substance use/misuse, health and nutrition, and sexual risk behaviors.
Evaluation Type: Quasi-experimental
Summary of Outcomes:
Researchers found several statistically significant conclusions among PCMM participants after comparing the baseline and follow-up surveys: 22.1 percent of participants reported a decrease in carrying weapons and 13.7 percent fewer participants reported drinking alcohol. Decreases were also seen in participants reporting nicotine vaping and cigarette use, although statistically insignificant (decreases of 6.5 and 2.9 percentage points, respectively).
Incidents of cyberbullying (also called online victimization) decreased significantly between survey administration: down from 25.74 percent to 21.94 percent. Participants reported a nearly significant difference in resilience on The Child and Youth Resilience Measure questions in the survey, from a 67.49 percent to a 74.31 percent (a 6.82 percent increase) due to the resources and assistance participants received from PCMM. Less positive findings include that the follow-up survey showed lower levels of depression, a decrease from 8.43 down to 6.92 on the Kutcher Adolescent Depression Scale (KADS); however, the KADS threshold for probable depression is 6 and above. Additionally, marijuana use increased from 7.2 percent to 12.5 percent (a 5.3 percent growth) from 2017 to 2020.
The study included context around the COVID-19 pandemic to further explain these outcomes. For instance, the over 20 percent decrease in carrying weapons could be partially attributed to the differences in socialization in the pandemic lockdown, as youth were home more often and less likely to be interacting with their peers in person. The decrease in nicotine vaping is related to a decrease in availability, including vaping shop closures in the pandemic, and requirements to purchase e-cigarettes with age verification, while alcohol and marijuana remained more easily accessible to youth. However, the authors of study note that PCMM participants’ resilience was higher due to PCMM mentors and staff providing them with support and resources to help in this area, which the authors also posit may explain why PCMM participants’ levels of depression decreased. Researchers attributed that depression dropped from baseline to follow-up (8.43 down to 6.92) due to the services and mentorship support offered by PCMM, but stayed above 6 as a result of the social isolation and related stresses of the COVID-19 pandemic