This study was commissioned by Comic Relief, DFID, Big Lottery Fund, NIDOS and Bond to address the lack of evidence available to support NGOs working in international development in deciding what resources to commit to monitoring, evaluation and learning (MEL).
The study focused on understanding the full investment that NGOs are making on MEL, the kinds of MEL systems that NGOs have, and how NGOs use and value their MEL systems.
Implications for NGOs
The findings of this study have a number of implications for NGOs to take account of when considering their MEL systems:
Position in aid chain
A starting point for designing an efficient and effective MEL system should be to consider the implications of each organisation’s role in the aid chain and what this means in terms of what should be measured.
Purpose of MEL
NGOs should be clear when designing MEL systems as to what their main priorities are with regards to the uses of MEL data and at what level, for both themselves and their partners. They then need to ensure that their system works to support these priorities.
Budgeting for MEL in joint proposals
MEL budgets in joint proposals should be clear on which partner will be responsible for what in terms of MEL data collection, storage and analysis and should also consider whether organisations have sufficient capacity to carry out these roles, with any capacity building requirements for either/ any party being budgeted for.
Understand the full cost of MEL
NGOs should develop systems that allow them to assess the full cost of MEL so that they can a) judge whether their MEL systems are an optimum and proportionate use of resources given the quality of the data and analysis they are getting out of them and, b) ensure full cost recovery of the projects that they deliver.
Implications for Funders
Funders can play their role in supporting effective and efficient MEL by:
Being clear on their expectations
Funders should be clear, and give clear guidance on: What they expect to see in applications with regards to MEL systems for different sizes and kinds of grantee organisations; the kind of costs that should be considered for MEL and the level of detail they want; the level of detail required in applications as to the different roles different partners will play in MEL and how this should be budgeted for; and, the data they require to have reported to them and any expectations they have as to the uses of MEL data for accountability, learning and communication at other levels.
Donors should be clear on the cost implications of their expectations for MEL data
Full cost recovery
Donors should be clear on the cost implications of their expectations for MEL data and be prepared to fund the full costs of this for both UK and overseas partners. This includes taking full account of costs such as staff, capacity building and infrastructure. If they consider these costs to be excessive then they may need to readjust their expectations for MEL data.
Areas for further consideration
This preliminary study has highlighted some areas that merit further consideration either for follow up studies or where it would be worthwhile developing guidance for NGOs:
There can be a tendency to consider bespoke MEL systems as better systems. The findings of this study question this assumption. Further work could usefully be carried out to understand under what circumstances bespoke systems are valuable and what is their full cost.
Analysing qualitative data
The study highlighted a large capacity gap in analysing qualitative data despite its wide use for fundraising. Further work could usefully be done to look at simple ways that NGOs can use qualitative data at a more aggregate level.
The factors underlying the effectiveness of MEL systems
This study was only able to take a broad approach to investigating the factors underlying the effectiveness of MEL systems and relied on NGOs self-reporting of how effective and accurate they found their systems to be. A closer look at what factors support MEL systems to be both accurate and useful at different levels of the aid chain could yield some useful insights.
Risk in managing data
The study showed that some NGOs have not fully considered any legal restrictions or data protection issues that there might be on data they collect. Guidance for development NGOs on data management and risk could usefully be developed, drawing on existing guidance from other sectors such as the humanitarian sector.
How MEL systems can effectively support learning
The study of Comic Relief funding cycles found that a common shortcoming was that it was not clear how learning from previous MEL was influencing the design of new projects. How MEL systems can better support learning is a question that is worth investigating further.
This study’s initial attempts to understand how data flows within MEL systems raise questions that wo