Our equitable research learning journey continues

City Year’s equity-based research journey continues

City Year has always been a learning organization, seeking out feedback, developing deep collaborations, learning from others, and modifying our practices as evidence and experience pointed us in new directions. Our evolution in recent years has focused on reaffirming our commitment to diversity, belonging, inclusion and equity—and that has included the way we select and conduct research projects and share findings.

In 2019, we made a deliberate shift in the way we conduct research to ensure it’s more *equity-based.

*An equity-based approach to research means that the creation of research should begin from a place of mutual understanding among community organizations, researchers, and funders. Those involved in the research design must recognize unintended bias to arrive at an authentic trust that does the most good for those being researched. (Source: Chicago Beyond 2019).

We came to understand that City Year’s research and evaluations can not only make meaningful contributions to organizational learning and improvement, but also inform how we define evidence and impact.

As our team shared in 2022: “We arrived at this vision after acknowledging that some previous research projects did not center the voices of the individuals doing and most affected by the work in the research design, implementation, and dissemination. This caused harm, impeded practical relevance of learnings, and highlighted the need for a new approach.”

A catalyst for change

One of the catalysts for our intentional shift to equity-focused research was what we were learning through a study on our Whole School, Whole Child® services funded by the Institute of Educational Sciences (IES), a project that began in 2017 and concluded in 2023. Analysis and data collection were conducted by the American Institutes for Research (AIR) and MDRC. One of the greatest strengths of this project is the mixed methods approach and collaboration between these two research institutions and our practitioner organization, City Year.

This evaluation involved a comparative interrupted time series design to examine whole-school effects of WSWC programming, a randomized controlled trial to examine the effects of targeted supports, and as a quasi-experimental impact study to explore the effects of the whole-school model for all students.

However, we received constructive feedback during the IES study by practitioners who had a difficult time implementing the randomized control trial (RTC). Whereas City Year’s HQ team was working collaboratively with the researchers and site leadership on planning the project from the outset, we overlooked including deeper level insights from the most proximate practitioner voice in the beginning stages—in this case the voices of City Year AmeriCorps members and City Year school-based staff called impact managers and impact directors, partner teacher and principals and the students these adults were serving—which could have helped flag potential challenges we later faced.

In a feedback session with City Year sites in 2018, participants echoed that “people want to know they are a part of a study that is going to recognize [contextual] factors accurately” and “value AmeriCorps member voice and input.” They also shared that “the lift was heavier than anticipated” when it came to implementation of the study and the support from HQ was less than desired.

City Years Headquarters Team learned a lot from feedback like this and have since evolved our approach to involve community and proximate practitioner input throughout an entire project, from research question generation and planning to dissemination of the findings.

Some examples of our shift to a more equity-based approach throughout this project include:

• • Centering practitioner voice and expertise. The way we intentionally brought in those closest to the work (e.g., site staff who consulted and/or included student success coaches, teachers, and community members) from sites to share their implementation experiences beginning in 2018 and more recently to support in meaning making of the findings. We held focus group feedback sessions yearly and meaning making sessions towards the end of the project.
  • Establishing new communication channels to capture insights in real time. Based on feedback in focus groups that requested “more input for voice” and “reorienting at each state of the project”. From 2018 on, we held consistent bi-weekly or monthly calls with the practitioners closest to the work to greater understand implementation successes and challenges. This allowed us to be more nimble and come together across teams and organizations with potential solutions to challenges as they arose. It also increased buy-in from practitioners to the project.
  • Creating communities of practice. We learned that connecting research to practice includes providing spaces for community input throughout, collaborating on or making tools that make implementation easier and the findings most useful, and taking time to make meaning with those closest to the work in the later stages of a project. An example of community input that helped shift implementation practices was feedback that a community of practice would be helpful to sites to hear about how other sites were facing implementation challenges. We formed a group where representatives from five sites joined monthly to get to know one another and discuss common problems of practice, such as how to incentivize returning consent forms or how to support AmeriCorps members in talking about the study with school partners. After we created this space, one site said at the conclusion of the second year that they “appreciated the ‘community of practice’ calls to hear what’s happening at other sites, and we were comforted knowing we are not the only site experiencing a particular challenge.”
  • Refocusing research as an organizational learning tool. Using findings from our research and evaluation work has been part of City Year’s shift throughout this project as well. We have done this through codeveloping a Research and Learning Agenda with staff at many levels of the organization including AmeriCorps members, launching a Research Steering Committee, a landscape analysis tool, and creating a community-based-action-research function, and centering community and site voice.

• Elevating context. Focusing on the importance of context became increasingly crucial during this study, as site voice facilitated implementation and COVID highlighted the importance of being flexible and centering the practitioner’s experience within research. David Osher, Vice President and Institute Fellow at American Institutes for Research (AIR), reflected on the project and said, “Context matters—and the more we know about it in both the schools receiving the intervention and the schools serving as comparison schools, the more we can discern effects and understand the impacts.”

COVID Considerations

The project experienced COVID-related disruptions, which further highlighted the need to work collaboratively and with flexibility with our sites more deeply. The study had planned two cohorts for the RCT, which were reduced to one. The duration of the interventions was also shortened, and not all students were given the opportunity to take the planned SEL survey. Additionally, in the CITS portion of the study, a survey that was planned to send to school leaders was also dropped.

Experiencing the COVID context together as practitioners and researchers further facilitated a practitioner-research relationship that strived to engage those closest to the work in an ongoing way and to elevate context. “Given the powerful disruptions caused by the pandemic, the evaluators had an ethical responsibility to ensure that their activities did not create additional stress on City Year and school staff,” said David Osher, AIR.

Conclusion: Following our North Star

This projects’ learnings went far beyond the scope of what we had intended to learn, and as an organization and research team we are better for it.

We learned the importance of centering equity and proximate practitioner voice within projects and have made it our north star.

We learned that mixed methods approaches are invaluable in understanding the context in which outcomes occur and that we can be much more intentional about valuing the insights and experiences of individuals just as much as the outcome data we collect.

We created a set of equity guidelines throughout the course of this project that we have iterated on since and will be updating and sharing out as a new version again this year.

Finally, we’d love to hear from you if you’ve had similar challenges with implementation of projects both within and outside of the COVID-context, and what has your equity journey been? Please feel free to contact jproett@cityyear.org with comments or questions.

Background on City Year

City Year is an education and human development organization that works with school and community partners to support student success, advance educational equity, and contribute to systems-level changes in policies and practice that address the root causes of inequitable educational outcomes.

Founded in 1988, City Year recruits and trains a diverse group of 2,000+ AmeriCorps members, ages 17-25, to spend a year in service in public schools, where they serve as student success coaches—near-peer tutors, mentors and role models who provide holistic, personalized support to students as they cultivate foundational social, emotional and academic skills that are essential for success in school and in life. City Year operates in 29 U.S. cities (local City Year offices are referred to as “sites”) and partners with 250 schools nationwide to deliver its Whole School, Whole Child® (WSWC) services.

Jessica Proett is Senior Director, External Evaluation, on City Year’s Education Research & Strategy team. She came to City Year after several years as a teacher and is passionate about educational research as a learning tool that connects directly back to practice.