City Year & Cornell: Evaluating Impact Outcomes in Partnership

City Year and Cornell University, two long-standing Einhorn Collaborative partners, recently embarked on a joint project to understand their alumni's ability to work with others, engage in civic and community issues, and find meaning and satisfaction in career and personal life. This piece represents the work of several individuals over the course of decades of evolving work.

Over the past ten years, nonprofits of many stripes have become increasingly concerned with the desire to measure and demonstrate their impact. This trend has been encouraged by funders, who tend to reward quantifiable results. Yet this push towards quantitative monitoring and evaluation has come at a cost. When research does not prioritize equity and collaboration, external interests can override the needs of the community where the work is conducted. In this situation, valuable and scarce community resources are directed towards a well-intended project that furthers academic knowledge but may offer few tangible benefits to the people and institutions at the heart of the study. In response, we have witnessed a rising commitment to equity-based research. These intentional practices center communities and program participants in both formulating research questions and uncovering their answers.

Evaluating the impact of youth development programs on young people can be challenging. In particular, it can be difficult to balance the desire for more “objective,” quantitative data, such as those obtained through randomized control trials, with the needs and voices of the people being researched. Both Cornell University and City Year have devoted substantial resources to developing equity-based approaches to evaluation that capture the full context of impact.

Evaluating Community-Engaged Learning at Cornell University

Over the past decade, evaluation of Community-Engaged Learning (i.e., CEL including service-learning, community service, internships, study abroad) at Cornell has occurred at a number of different levels. The propitious timing of the initial gift from Einhorn Collaborative in 2011 to support Cornell’s community-engaged learning initiative coincided with the university receiving the Carnegie Foundation's elective Community Engagement Classification which offered an opportunity for the Vice Provost's Office overseeing Land Grant Affairs to undertake a university-wide assessment of community engagement activities at Cornell. As a preliminary evaluation, the data collected to support Cornell’s application to Carnegie is a rare and unique example of a systematic university-wide process that requires substantial relationship-building and involvement from various stakeholders across campus. Along with the data and contributions collected from colleagues across campus, the initial evaluation provided useful baseline information about CEL, community partnerships, and student opportunities and gaps. It also informed an asset-based, stakeholder-driven approach to the evaluation of CEL moving forward.

From there in 2012, Cornell’s Office of Institutional Research and Policy (IRP) embedded specific questions in student surveys to better understand the level and type of student participation in CEL throughout their time at Cornell. In 2016, IRP conducted a study and found that student participation in various forms of CEL had significant correlation with a number of university-wide learning outcomes including critical thinking, teamwork, personal ethics, and environmental responsibility, in addition to well-being and sense of community. Furthermore, in 2015, to assist in part with ongoing evaluation efforts, Cornell steered the adoption of a university-wide CEL course designation based on four quality criteria. With support from IRP, Cornell also developed a CEL dashboard to monitor and track participation in course-based CEL across the university.

The evaluation efforts included extensive outreach and relationship-building with faculty, administrators, and staff in each of the colleges and with multiple other units on campus that support CEL. In 2020, Cornell received reclassification for Carnegie’s elective Community Engagement designation which is a testament to the value of efforts to approach evaluation as a collective endeavor more fully representing the growing participation in and support for CEL. This engagement includes the extensive network of relationships built across campus to create and support a university-wide system and infrastructure for monitoring and evaluating CEL. Collaborative efforts to evaluate CEL from an equity lens at Cornell remain a work in progress.

Developing an Equity-Based Research Approach at City Year

As a learning organization, City Year has prioritized research and evaluation since its founding in 1988, but the transition to focus its service exclusively in schools in 2012 brought an increased focus on understanding the program’s impact on both individual students and whole school communities. City Year recognized that it had to show impact especially in math and reading and that centering an approach that prioritized social, emotional and academic development would yield such results. The organization learned that some of the methods traditionally regarded as the most rigorous do not prioritize community involvement and input in the research design in ways that ensure the findings are actionable, inclusive, and relevant to those most affected by the research. As a result, City Year started to shift how it approaches research. With every new research project, the organization is learning from the communities with which it partners and other organizations committed to equitable research and evaluation practices.

City Year now publishes a set of guidelines based on ongoing learning to structure their approach. These guidelines represent an effort to learn out loud and ensure continued transparency of evaluation processes. The organization also released in Spring 2022 a Research and Learning Agenda that describes the organization’s journey to adopt an equity-based research approach and signals how and what City Year plans to study in the future.

A central aspect of equity-based research is collaboration, including researcher-practitioner collaboration. City Year has intentionally set up project management aspects around research projects that ensure local voice is elevated throughout. During consistent partnership conversations with researchers, City Year applies the lens of equity-based research by elevating site concerns or thoughts, flagging potential challenges with solutions to mitigate them, and sharing as much context as is possible about the programming and communities being researched. These efforts facilitate smooth progress of the research process, increase community buy-in, and strengthen the organization’s research-practice-policy system by ensuring that results are directly applicable for the community and the larger organization.

Embarking on a Joint Research Project to Understand Longer-Term Civic Development

In this period of learning and reflection, Itai Dinour, the Bridging Lead at Einhorn Collaborative, who is both a Cornell alum and a former City Year staff member, connected leaders at City Year and Cornell to discuss their respective learning journeys and to reflect on the challenges and opportunities of impact evaluation. Both organizations have been long-term partners of Einhorn Collaborative and have spent considerable time evaluating the experiences of City Year AmeriCorps members and Cornell students respectively. City Year and Cornell were curious to understand their programs’ impact on alumni and the ways that service experiences before, during, and after college can affect an individual’s ability to work with others, engage in civic and community issues, and find meaning and satisfaction in career and personal life.

This shared curiosity inspired the Cornell/City Year alumni evaluation project. Youth involvement in traditional forms of civic engagement has been on the decline for several decades, despite the fact that young people are deeply invested in social and political issues, as demonstrated by youth leadership in movements like Occupy Wall Street, March for Our Lives, and the Dakota Access Pipeline protests. College-level service learning programs represent one major avenue through which young adults can participate in civic and community service. For many, however, this engagement ends after college. This project will reveal ways that programs like City Year and Cornell can better support young people to stay involved in civic and community life, form relationships with those who are different from them, and work productively with diverse social networks to serve their professional readiness and success.

A group of practitioners and researchers from Cornell and City Year determined the key questions to guide the project. Both organizations seek to understand which aspects of the programming and which specific aspects of an individual’s experience help lead to increased civic and community engagement. Under what circumstances do alumni put into practice what they learned at City Year or Cornell in their own communities? What prevents them from doing so? And how might both programs better prepare future alumni to navigate those obstacles?

Equity-based research is often more time-consuming and resource-intensive than traditional evaluation methods, due to its emphasis on partnership, transparency, inclusion, and consensus. Designing a research process that includes, and is actionable for, community members engaged in the work requires the resources to co-design both the research process and the delivery and integration of insights in the application of results. This process demands time and commitment from numerous stakeholders throughout. Support from Einhorn Collaborative has helped create this capacity across the project and within each organization.

Cornell and City Year could have chosen to hire external evaluators to assess the impact of their respective programs. However, doing so in partnership with another organization with similar values but different experiences creates an opportunity for the learning from the process to transcend the two organizations. The researcher-practitioner partnership also aims to deepen the research and make it as actionable as possible by engaging with those closest to the work. The insights derived from this project will guide the organizations in how they improve their program experiences and how they communicate about their programs’ benefits. At a strategic level, these cross-organizational findings have the potential to extend well beyond both City Year and Cornell, potentially offering systems-level implications for positive, prosocial youth development.

Both organizations remain committed to sharing insights from this multi-year, mixed methods approach to evaluation, and they invite interested parties to reach out with questions, suggestions, or reactions.

For more information on Cornell’s contributions to this project, contact Esa Burson at emb392@cornell.edu, and for more information about City Year’s contributions to this project, contact Jessica Proett at jproett@cityear.org.