Figuring Out If It Matters

Can three hours, three times a week for ten weeks, for a total of 180 hours of training outside of school make a difference in a teenager’s life? The answer is a (qualified) yes.

Several pieces of research have shown that Chicago’s After School Matters (ASM) apprenticeship program works. But the latest evaluation of the ASM program, led by Barton J. Hirsch, finds mixed results for ASM and suggests potential improvements.

Maggie Daley began the ASM program in 1991 with a focus on arts-based apprenticeship opportunities for Chicago youth. By 2000, other types of apprenticeships, including science, sports, and technology, were incorporated into ASM programming.

Authors Hirsch, Hedges, Stawicki, and Mekinda conducted a rigorous three-year randomized controlled trial study of the ASM program from 2006 to 2009. The authors identified four primary outcomes of interest: positive youth development, frequency of behavioral problems, marketable job skills, and positive academic performance. The sample consisted of a total of 535 youths in the apprenticeship programs and in the control group. During the study, the authors found that 91 percent of the control youth were involved in an after-school activity or paid work.

The authors provide two types of analyses: an intent-to-treat analysis, in which they include all program participants assigned to the treatment group and all participants assigned to the control group, and a treatment-on-treated analysis, in which they remove the youth who have less than 73 percent attendance and who participated in an alternative after-school activity.

In the intent-to-treat group, the authors find evidence of improvement for positive youth development and reduced behavioral problems, especially with respect to selling drugs, gang activity, and carrying a weapon. However, they find that ASM participants showed no significant difference from other youth in developing marketable job skills and obtaining positive academic outcomes.  The results for the treatment-on-treated group were weaker still, with ASM producing favorable results on measures of behavior and job skills.

So is the program really effective? According to Hirsch:

There’s some evidence of it. It’s not easy to get results that are statistically significant…There were no instances in which the control group did better.

Hirsch believes that the ASM program is “a good start, but they need to make improvements.”  The authors suggest improving the quality of instructors, as well as the development of a stronger ASM curriculum that teaches transferable skills and imparts knowledge that will help youth in the workplace.

Additionally, despite the study’s mixed results, the authors emphasize that the improvements that the study did identify should not be overlooked:

Although it is frequently the case that no significant treatment effects are found in experimental outcome studies, in this research ASM did have a significant impact in areas that are important to adolescent development and to policy. Moreover, it demonstrated these impacts in relation to what was essentially an alternative treatment comparison group.

Despite the modest evidence from the Hirsch study, the ASM program has been recreated throughout the country. “Organizations in New York, Boston, and other cities have replicated our apprenticeship model,” says David Sinski, ASM’s Chief Officer of Strategy and Innovation. Sinski says that the study will help ASM “recognize the key components that make the apprenticeship experience so valuable for young people.”