Evaluating afterschool: The evaluation basics, part II

The Afterschool Alliance is pleased to present the ninth installment of our "Evaluating afterschool" blog series, which answers some of the common questions asked about program evaluation and highlights program evaluation best practices. Be sure to take a look at earlier posts of the series, including blogs on how to collect data and how to build your own evaluation advisory board.

Regino Chávez has worked for LA’s BEST since 2007, designing and implementing formative research for program improvement and internal evaluations, as well as monitoring external evaluations of the program. LA’s BEST serves approximately 25,000 children ages 5 to 12 in close to 200 Los Angeles Unified School District elementary schools. A recent evaluation of LA’s BEST by the Center for Research and Evaluation of Student Standards and Testing (CRESST) at University of California, Los Angeles, found that LA’s BEST students with high levels of attendance in the program were 5 percent less likely to drop out of school and 6 percent more likely to graduate from high school on time compared to their peers who did not participate in the program.

Find part I of Regino’s blog here.

There are two basic types of evaluations: formative and impact. A formative evaluation focuses on improving strategies, materials, implementation conditions, or other aspects of programming to determine if the objective of the program can be reached. In this type of evaluation, we can begin looking at the internal workings of a program (training, staffing, materials, nature of implementation, activities, etc.) to assess what can be altered both for students and for staff to better understand and deliver the program as intended. This is the heart of a continuous quality improvement (CQI) process, a type of formative assessment that examines the program planning: the assessment that occurs to monitor what is implemented and how, and to develop subsequent action plans for improvements. CQI can support identifying the conditions necessary for successful implementation. Such evaluations may point out, for example, that the program actually works better for upper grades, or that it should be carried out in the spring rather than the fall. Similarly, CQI can help identify if staff need to have background knowledge of selected content or support with goal setting, or perhaps that the program may need only nine rather than 17 activities for the students to master a specific program’s content.

In 2007, LA’s BEST carried out an impact evaluation and framed the questions around the long-term benefits of participation in our program. UCLA’s CRESST program was contracted to carry out the study and used student school records to examine graduation rates, dropout rates, and juvenile crime records. They found positive long-term benefits for our participants – higher graduation rates, lower dropout rates, and less involvement in juvenile crime. Using a more sophisticated evaluation design, the CRESST team replicated the evaluation with new cohorts and found that participants in the LA’s BEST program still demonstrated higher graduation rates and lower dropout rates than non-participants.

Impact evaluations are of a different sort. They examine what about our programs made a long- term difference and for whom. It is always better if these also examine the circumstances that contributed to success. Outcomes here might look at comparisons between your students and how they performed on selected measures (e.g., attendance, self-management, leadership, etc.) when compared to students who did not participate in the program.

What do we want to know?

One of the first things about conducting evaluation is that one needs resources to do it! When we look for resources for evaluation, LA’s BEST staff seek to establish how closely the potential source for revenue aligns to our mission. If it does not, we pass on that source and look for funding where there is synergy with our work and the way we choose to develop youth. It may not be a 1-to-1 match. However, this just means that we will need to develop the relationship with the potential funder and engage in some negotiation and education to arrive at a common understanding of how the expected outcomes will be achieved via a youth development approach.

After that, we begin framing the evaluation questions. LA’s BEST staff play a major role in defining the questions to assure we get information about what we need to know about the program, the implementation process and the conditions needed to meet the goals. When program providers set the questions, you are the driving force for the evaluation.

For many programs—in particular those that receive state or federal funds—funding is dependent on how many and how frequently students attend the program. Questions, then, are often framed around attendance. Nevertheless, it is just as important to assess whether students have improved on any measures embedded in the program’s short-term or long-term goals. For example, in LA’s BEST, we were curious about the following issues:

• Is there a difference between students in our program versus others on measures such as high school dropout rates and graduation rates?
• What are some strengths of our staff that support student success?
• Which social emotional learning (SEL) skills do we develop well? Which ones need further work?

Research designs for an evaluation can be as complex or simple as the research question requires it to be. Research strategies will depend on the type of data that you will be collecting. For example, a question may be whether your program adds value to the day school, and one can use attendance data to test the association. While attendance data is convenient as a focus for an evaluation, it may be difficult to get if your organization does not have an agreement with the school district.

Other research strategies are useful to consider, but each has its own share of advantages and disadvantages. Surveys are quick and less costly to administer, especially if one uses digital means for data collection. Focus groups, like surveys, are quick and provide a deeper understanding of the issues relevant to the evaluation focus. Individual interviews offer opportunities to explore themes in-depth. However, surveys and interviews, including focus groups, have their limitations. For example, analyzing qualitative data from interviews does require special skills and take time to collect, code, and analyze. Surveys capture data that is limited to the response categories offered to respondents, their recall ability, as well as their perception of the degree of safety in responding to a question.

The type of evaluation (formative vs. impact) may at times dictate the evaluation design. Formative evaluations monitor progress toward meeting goals; impact evaluations seek to establish what occurred because of exposure to a program (intervention). Impact evaluation may need comparison groups to demonstrate whether there is an association between program participants and the variable(s) of interest. Formative evaluations can also be complex if you include multiple research strategies. For example, you may want surveys of staff to probe attitudes; an implementation checklist to see what exactly staff implemented in the setting; and focus groups with students to probe for benefits. At LA’s BEST, external contractors have carried out complex studies using sophisticated statistical models to examine long-term benefits of programming; use of these strategies has enhanced the credibility of the results.