Investing for Impact: Innovations in Measurement and Evaluation
The philanthropic sector has a compelling opportunity to advance the base of evidence of social change interventions that generate measurable impact. An evolving field of practice, measurement and evaluation (M&E) has the potential to uncover critical information about the efficacy of various social change strategies and to catalyze the expansion and replication of organizations and initiatives with proven social benefit.
The Corporation for National and Community Service's Social Innovation Fund ("SIF") embodies a pioneering national policy agenda focused on expanding the practice of M&E. This unique public-private partnership places the philanthropic sector at the center of the national effort to develop new evidence-based systems to help guide social change investment. The SIF initiative defines four levels of evidence:
Strong Impact. Strong impact is impact with a substantial likelihood of yielding a major change in life outcomes for individuals or improvements in community standards of living. The definition varies with context. For example, a mentoring program that cuts youth crime by 2 percent over a given period would not have "strong" impact, but a program that cuts such crime by 20 percent might. A program that increases an individual's earnings by $50 per week for a month and then fades out would not have strong impact. A program that increases earnings by that amount for a period of years would.
Strong Evidence. Strong evidence is evidence from previous studies whose designs can support causal conclusions (studies with high internal validity), and studies that in total include enough of the range of participants and settings to support scaling up to the state, regional, or national level (studies with high external validity). For example: more than one well-designed and -implemented experimental study or well-designed and -implemented quasi-experimental study that supports the effectiveness of the practice, strategy, or program; or one large, well-designed and -implemented multi-site randomized controlled trial that supports the effectiveness of the practice, strategy, or program.
Moderate Evidence. Moderate evidence is evidence from previous studies whose designs can support causal conclusions (studies with high internal validity) but have limited generalizability (moderate external validity), or studies with high external validity but moderate internal validity. For example: at least one well-designed and -implemented experimental or quasi-experimental study supporting the effectiveness of the practice strategy, or program, with small sample sizes or other conditions of implementation or analysis that limit generalizability; at least one well-designed and -implemented experimental or quasi-experimental study that does not demonstrate equivalence between the intervention and comparison groups at program entry but that has no other major flaws related to internal validity; or correlational research with strong statistical controls for selection bias and for discerning the influence of internal factors.
Preliminary Evidence. Preliminary evidence is evidence that is based on a reasonable hypothesis supported by research findings; in other words, research that has yielded promising results for either the program, or a similar program, constitutes preliminary evidence. For example: outcome studies that track program participants through a service "pipeline" and measure participants' responses at the end of the program; and pre- and post-test research that determines whether participants have improved on an outcome of interest.
Three leading philanthropic organizations — Venture Philanthropy Partners, the Edna McConnell Clark Foundation, and the Robin Hood Foundation — have made M&E a cornerstone of their investment strategies. These organizations provide valuable emerging models for how the philanthropic sector might advance M&E efforts.
Venture Philanthropy Partners. Venture Philanthropy Partner's youthCONNECT initiative, a SIF partnership, supports an integrated approach to addressing the education and employment needs of low-income and vulnerable youth between the ages of 14 and 24 living in the Washington, D.C., region. In collaboration with Child Trends, a nonpartisan research and policy center specializing in child development, youthCONNECT grantees are contributing data as part of a common framework that measures outcomes in education and employment across target populations. Child Trends will analyze data for critical trends and opportunities for learning and improvement, while youthCONNECT will support experimental and quasi-experimental evaluation of all projects at the organizational level. VPP expects that such rigorous evaluations, coupled with cross-organization analysis of trends, will yield lessons with broad implications for the youth development sector.
Edna McConnell Clark Foundation. The Edna McConnell Clark Foundation identifies highly effective nonprofits serving disadvantaged youth from low-income communities and makes substantial long-term investments to strengthen their evidence base and replicate and expand their programs. In 2010, EMCF created the True North Fund in partnership with SIF to further develop its focus on evidence-based investment and expand its support for M&E practices among its grantees. Two core aspects of EMCF's True North strategy are funding for multi-year business and strategic plans; and funding for rigorous evaluation of programmatic outcomes. EMCF is collaborating with MDRC, a pioneer in the use of random assignment research designs, to build the evidence base of the fund's grantees. MDRC conducts evidence audits of grantee programs to assess their readiness and appropriateness for rigorous evaluation studies and conducts, as appropriate, random control trials (RCTs) of grantee programs. To date, EMCF has funded a host of RCTs, including programmatic evaluations for Big Brothers Big Sisters of America, Citizens Schools, Harlem Children's Zone, Youth Villages, and the National Guard Youth ChalleNGe, among others. Such evaluations, along with current and future RCT efforts, provide valuable confirmation that investee programs generate desired impacts, offer valuable means for internal program improvement, and dramatically increase the base of evidence for programs that drive measurable impact.
Robin Hood Foundation. The Robin Hood Foundation targets poverty in New York City by finding and funding the best and most effective anti-poverty programs and partnering with them to maximize results. Robin Hood has developed a rigorous cost/benefit metrics system to help it analyze the best use of funds among its target population. Through a comprehensive analytical process, Robin Hood calculates the boost in lifetime earnings and living standards of poor individuals that are likely to result from the programmatic interventions it funds. In effect, these ratios are a philanthropic-sector analog to commercial rates of return in the private sector and provide a valuable analytical construct for investment decision making as well as an effective diagnostic tool to help the foundation identify common attributes among its most successful grantees.
While the efforts outlined above demonstrate some of the best practices being adopted by individual nonprofits and foundations, the great promise of M&E lies in coordinating these types of efforts across the sector in a way that enables a large number of organizations to collectively measure, coordinate, and improve their performance. Such shared M&E systems have the potential to dramatically expand the value of M&E practice — and to fuel breakthroughs in learning that increase the collective impact of nonprofits and their funders. Indeed, we are beginning to see the potential of shared M&E in several emerging efforts, including Grantmakers for Effective Organization's Scaling What Works project, the Strive Partnership, and the Cultural Data Project.
The M&E field will evolve as the philanthropic sector continues to invests in data collection and analysis efforts. But make no mistake: The train is leaving the station. Leaders in the sector should aggressively support innovation in M&E and make an ongoing commitment to developing effective, collective performance measurement mechanisms that can be widely replicated.
Ashley Allen is a partner in the Endeavor Group, a strategy and legal consultancy based in Washington, D.C.