Evaluating afterschool: Answering questions about quality

By Jocelyn Michelsen, Senior Research Associate at Public Profit, an Oakland, California-based evaluation consultancy focused on helping high-performing organizations do their best, data-driven work with children, youth, and families.

The Afterschool Alliance is pleased to present the fifth installment of our "Evaluating afterschool" blog series, which answers some of the common questions asked about program evaluation. Be sure to take a look at the first, second, third, and fourth posts of the series.

Raise your hand if this scenario sounds familiar: you keep up with new research on afterschool by reading articles and newsletters, following thought leaders, and attending conferences—but it is still hard to sort through all the information, let alone implement new strategies. Research often seems out of touch with the realities of programs on the ground, and while many anecdotal examples are offered, how-to guidelines are few and far between.

As an evaluator consulting with diverse afterschool programs across the San Francisco Bay Area and beyond, I frequently hear this frustration from program leaders. There is a real gap between the research and the steps that staff, leadership, and boards can take to build quality in their own programs. Additionally, it can be hard to sift through the research to get to the ‘why’—why implement these recommendations, why invest time and resources, why change?

We’ve taken on some of that sifting to help programs move toward actually using the research. In our brief After School Quality: Moving From Research to Practice, we plumb the literature and distill our firm’s experience to answer some of these challenging questions:

Why does quality matter?

Program quality is the hinge between youths’ afterschool program participation and youths’ positive outcomes. It is not enough for youth to be present at a program – to truly benefit in the short-term and later on in life, youth must have access to afterschool experiences that are of high quality.

As well, program quality increasingly matters to districts and state departments of education (including the California Department of Education). Measuring program quality is increasingly a feature of afterschool funding and as such is growing in importance and visibility within the field.

What does quality look like?

Programs that are high quality weave in positive youth development practices such as relationship building between peers and between youth and staff, platforms for youth choice and voice, chances to develop leadership, opportunities to gain skills and build mastery, and access to well-organized activities that challenge youth in a positive way.

Why should I focus on staff-youth practices?

Program leadership and staff have direct control over the staff-youth interactions within their programs, and as such focusing on how staff interact with and support youth is the best way to understand how programs function and can be refined. The ultimate goal is to scaffold staff to keep helping youth make positive gains by helping them base their practice in well-defined program quality mile markers.

Where do I start?

Depending on your program’s capacity, resources, and goals, there are a number of ways you can start to grow program quality. Creating a culture of feedback and iteration among program staff is a great way to build a strong foundation that will underlie other quality efforts. As well, initiating program observations (also called site visits) using a common observational framework is one of the best ways to assess point-of-service quality over time. Program quality is founded on the tenets of positive youth development, and so any observational tool you are likely to select will be content-neutral – this is great news for programs concerned about matching their program model, needs, content focus, or staff capacity to a specific tool.

How can I learn more?

Public Profit’s After School Quality: Moving From Research to Practice is designed to help programs bring high-quality youth development practices to their own work in a way that is right-sized and feels manageable and sustainable. The brief also links to some free additional resources about selecting an observational tool and how to collect and make sense of your data. Together, these resources will help guide programs through collecting and using data to refine services—and ultimately to work toward achieving even greater positive impacts for youth.

For more information about Public Profit, visit www.publicprofit.net .