Afterschool Impacts Database: Record Details

Evaluation 101

Learn more about the basics of evaluation, as well as how to find an evaluator for your program.

Afterschool Evaluation

View Afterschool Alliance resources, including a glossary of terms used in the database, evaluation-related blogs, webinars and more.

Additional Resources

Our list of evaluation resources from other organizations, including how to collect and work with data.

Afterschool Research

Want to find what we know about afterschool programs more broadly, not just individual programs? Head to our Afterschool Research page!

Afterschool & STEM: System-Building Evaluation 2016

Year Published: 2017

A total of 1,599 students participating in afterschool programs with a STEM focus in grades 4-12 from 11 states completed retrospective self-report surveys measuring STEM-related attitudes and 21^st century skills. Facilitators completed a survey about their experiences leading afterschool STEM, and the programs’ STEM activities were observed and evaluated to establish levels of quality. The evaluation found that students that participated in STEM-focused afterschool programs led to positive changes in students’ attitudes toward science, STEM interest, STEM identity, STEM career interest, career knowledge, 21^st century skills, and critical thinking. Larger effects were seen in students who participated in programs for a minimum of four weeks. Higher quality STEM programs reported more positive gains than lower quality programs.

Keywords: National Multi-city Rural Urban Suburban English Language Learners Populations of Color College & Career Readiness STEM

Scope of the Evaluation: National, Multi-city

Program Type: Afterschool

Community Type: Rural, Urban, Suburban

Grade level: Elementary School, Middle School, High School

Program Demographics:

A total of 1,599 students (733 female, 866 male) in Grades 4–12 completed the Common Instrument Suite (CIS) self-report survey measuring STEM-related attitudes and 21^st century skills. The sample was diverse and included groups that are traditionally underrepresented in STEM. Specifically, students identified as American Indian (1.83%) or Alaska Native (0.21%), African American/Black (25.05%), Asian/Asian American (3.11%), Latino or Hispanic (13.90%), Middle Eastern/North African (0.42%), Native Hawaiian or other Pacific Islander (0.71%), and more than one group (10.44%). About 10.5% of students preferred not to answer. In addition, approximately one third of students speak a language other than English at home (29.9%).

Evaluator: The PEAR Institute: Partnerships in Education and Resilience at Harvard University and McLean Hospital and IMMAP: Institute for Measurement, Methodology, Analysis & Policy at Texas Tech University.

Evaluation Methods:

A total of 11 state afterschool networks were chosen to participate in this evaluation based on a priori criteria agreed upon by funders, PEAR, and IMMAP: (1) the collection of participating states together reflect the demographic diversity of the U.S., including rural, suburban, and urban composition; (2) the networks receive system-building support from the two funders, and (3) the networks demonstrate prior experience and capacity to implement a large-scale and complex evaluation within the designated evaluation time frame (March–June 2016). An expert demographer at Texas Tech University served as consultant to inform on the choice of states to ensure the representativeness of the sample.
Nearly 1,600 students (grades 4–12) enrolled in 160 afterschool STEM programs across 11 states completed a retrospective self-report survey called the Common Instrument Suite (CIS), which measures STEM-related attitudes and 21st-century skills. STEM facilitators completed a survey about their experiences leading afterschool STEM, and the programs’ STEM activities were observed by professionals certified to use the Dimensions of Success (DoS) tool to establish levels of quality.

Evaluation Type: Non-experimental

Summary of Outcomes: Participation in an afterschool program with a STEM-focus led to positive changes in students’ attitudes toward science. More than 70% of students reported positive gains in areas such as STEM interest, STEM identity, STEM career interest and career knowledge, and 21st-century skills, including perseverance and critical thinking. When compared to male counterparts, female students were significantly more likely to report gains in relationships with adults and peers. Larger positive effects were also noted in students who participated in their programs for a minimum of four weeks. Students participating in higher quality STEM programs reported more positive gains than students participating in lower quality STEM programs. There were clear variations in outcomes across states, which the author suggests is a reflection of states’ different phases of system building implementation and different strategies for supporting and training programs.

Associated Evaluation: https://docs.wixstatic.com/ugd/e45463_e14ee6fac98d405e950c66fe28de9bf8.pdf