ReliefWeb

Outcome harvesting is an evaluation approach that allows us to retrospectively identify emergent impact by collecting examples of what has changed in "behavior writ large" (actions, relationships, policies, practices) and then work backwards to determine whether, and how, an intervention has contributed to these changes (1). In a previous blog, Outcome Harvesting: More than a Contingency Plan**, **we spoke about the utility of outcome harvesting as a monitoring and evaluation tool in cases in two different cases: i) where it was a planned component of the MEL strategy, and ii) where it served as a contingency plan. In this blog, we hope to surface what we have learnt about the scenarios the approach is best suited to, how to prepare for it, and considerations for the implementation of its different steps.

WHEN TO USE OUTCOME HARVESTING

Outcome harvesting is particularly suited to projects where the outcomes are not predetermined at the time of planning or projects where the nature of the activities and the context evolve over time. In such complex settings, outcome harvesting encourages a different approach to monitoring and reflection by first facilitating discussions regarding the changes that have taken place, and then ascertaining whether, and to what extent, they can be traced back to the intervention. It also helps capture unintended outcomes of interventions, which can be very valuable in environments where multiple factors are at play. Critically, the findings of this process can help inform how the project design and implementation evolve within the project's lifetime and in future programming.

HOW TO PREPARE FOR IT

If program managers wish to have outcome harvesting as a central component of monitoring, evaluation and learning for a project, they should incorporate it into the MEL plan at the proposal stage to help clarify its purpose, either in conjunction with a logical framework or even as a standalone tool. Planning for it in advance can also ensure that adequate resources are allocated for its effective implementation, in terms of time and expertise.

While outcome harvesting is intuitive and often easier to grasp than traditional results-based approaches, it requires capacity building in other related areas (e.g. making sure those involved are comfortable with the retrospective approach employed by the method, are familiar with how to draft strong outcomes, can trace the contribution of the intervention to the outcomes, and use the findings from the process to inform decisions). It is also an intensive exercise to undertake, so having clear expectations regarding the commitment required by facilitators and participants, in terms of time and resources, is helpful.

In setting up the harvest for success, it is crucial to identify a harvester who can ensure an inclusive and participatory process, and maintain high standards for data collection and analysis. Other actors that need to be identified before the process begins include the primary users of the findings, participants or key informants who are knowledgeable about the changes brought about by the intervention, and independent actors to help substantiate the outcomes of the harvest.

The harvest should also be executed at a time that coincides with when the changes stemming from the project are expected to be actualized.

HOW TO IMPLEMENT IT

1, Design

The guiding questions should be based on the intended uses of the harvest, which is why formulating them is a crucial first step. These questions help determine the scope of the outcome harvest exercise. Instead of exploring all the changes that are likely to be observed, a thoroughly developed scope, as outlined by the guiding questions, can allow the harvest to focus on specific outcomes that respond to the most important areas of interest. The scope is best defined through a participatory conversation that involves the harvest users and harvesters.

Additionally, this stage is ideal for designing a tool for documenting outcomes. The main objective of this tool is to facilitate an ongoing process to capture changes as they happen. This should be done in a manner that is accessible and clear to all who are involved. It can take the form of an outcome journal (2) or a structured questionnaire that feeds into a database, among other possibilities.

2, Review documentation and draft outcome descriptions

Outcomes are initially drafted based on a review of project documentation, and then further built upon in consultation with key informants knowledgeable about the project.

When drafting outcomes and engaging with informants, we need to be mindful of the guiding questions, and aim to capture both positive and negative, as well as intended and unintended outcomes. Accounting for positive bias is important here, as capturing negative outcomes can provide lessons of equal or greater value than positive outcomes.

Capturing unintended outcomes can be a source of unique information that pertains to blind spots not addressed by other methods. It is also important to note that acknowledging no change, or maintenance of the status quo in many cases, can also be quite significant and must be acknowledged as an outcome.

3, Engage with informants in formulating outcome descriptions

Engagement with informants, who are often field staff that are closest to the program implementation, can serve as a very conducive space for reflection. It can provide a periodic moment of pause that is dedicated to learning, i.e. for those involved with the project to step back and reflect on the changes they are seeing in the immediate environment and the broader system, which they might otherwise not have a chance to do.

A participatory and inclusive platform can encourage participants to engage in a nuanced discussion about the diversity of outcomes and perspectives they bring to the table. While the data collected is qualitative, the method controls for rigor as participants are encouraged to provide evidence for the changes they are seeing and how they tie back to the intervention.

When finalizing outcomes, this conversation can help pare down the number of outcomes by limiting them to the changes deemed significant by the participants. The quality of outcomes can also be enhanced by ensuring that all participants have a shared understanding regarding what qualifies as a strong outcome.

4, Substantiate

In order to verify the accuracy and credibility of the final outcomes, we need to engage actors who are independent of the intervention and yet knowledgeable about it, but they can be a challenge to identify. For instance, in some cases it is easier to reach out to partner organizations who are not driving the process but are still very invested in the intervention to substantiate outcomes, rather than having to identify truly independent third parties. This should therefore be done at the very outset, when identifying other key actors who are going to be involved in the harvest.

Another important consideration would be the frequency of the substantiation process. The number of final outcomes identified in each round, and the number of rounds of identification carried out, can greatly influence the time required for substantiation. The challenges can be compounded by the coordination needed to involve independent actors for the exercise, who are likely to be external to the organization. Thus while outcomes might be harvested on a monthly, quarterly or annual basis, substantiation and analysis can be undertaken less frequently.

5, Analyze and interpret

Analyzing the finalized and substantiated outcomes entails harvesters, in consultation with informants, classifying the outcomes as they pertain to the guiding questions, as well as the goals and objectives of the different actors, such as implementers or donors. It also involves harvesters delving into the data collected so far and using it to provide evidence-based answers to the questions identified.

Having a solid framework in place for analysis can aid harvesters in interpreting outcomes by ascertaining their quality and significance, the sustainability of changes beyond the project, and how changes contribute to higher-level impact. Importantly, it can also provide insight into how the outcomes reflect setbacks or failures, and discern the implications of the findings for adaptation of projects and how to overcome challenges (3).

This analysis, although relegated to a specific step, can also happen throughout the course of the harvest. In light of emerging outcomes, this ongoing analysis can shift the focus of the conversation and subsequent iterations of the harvest to the outcomes deemed as most significant by informants.

6, Support use of findings

While outcome harvesting can produce very insightful findings that aid our understanding of the project and the complexity of the system it operates in, the crucial link to action based on these findings can often be missing. Thinking through the utility of the guiding questions to the users of the harvest at the outset can elicit answers that inform crucial decisions and actions related to the project.

Using outcome harvesting as a basis for adaptive management can be further facilitated by adopting a strong framework for analyzing the outcomes. This framework can help clarify how the harvest findings can inform decision-making and what concrete steps can be taken in response.

These stages are not linear, they may overlap and lead to multiple iterations based on the emerging outcomes and analysis. The process therefore offers both flexibility and structure to a conversation about change -- be it positive, negative, intended, or unintended -- and helps us trace this change back to our intervention. It can be particularly effective as a monitoring and reflection tool. In addition to studying and learning from emergent processes, it also allows us to be intentional when intervening in project implementation and broader systems.

Read Chhavi Kotwani's previous blog on 'Outcome Harvesting: More than a Contingency Plan'.

Resources:

1) Ricardo Wilson-Grau and Heather Britt, Outcome Harvesting (Cairo: Ford Foundation, 2012), http://www.managingforimpact.org/sites/default/files/resource/outome_harvesting_brief_final_2012-05-2-1.pdf

2) Outcome journals can include a description of observed changes, contributing factors, evidence, assumptions, lessons learned, action items, etc.

3) Outcome-Based Learning Field Guide, (Washington D.C.: World Bank, 2014), http://documents1.worldbank.org/curated/en/457811468167942364/pdf/901760WP0Box380Learning0Field0Guide.pdf