Overview of Evaluation
This endline evaluation report is for the United States Department of Agriculture (USDA) McGovern-Dole (MGD) Food for Education Programme (FFEP) for the period of FY2014-2017. The objective of evaluation is to evaluate the key results areas of FFEP- learning outcomes and health and nutrition outcomes. The evaluation results will be compared against the baseline study conducted in 2015 (Kimetrica, 2015) and the midline evaluation (Mokoro, 2016). The evaluation followed The Organisation for Economic Co-operation and Development's (OECD) Development Assistance Committee (DAC) evaluation criteria of relevance, effectiveness and efficiency, impact, and sustainability.
The primary users of the evaluation broadly are USDA, WFP programme team and Government of Nepal, cofacilitating the programme operation as well as the partners implementing various programmatic components contributing towards holistic outcomes of FFEP. The evaluation results would help provide guidance for further programme conceptualization, operation and management. Further the evaluation will be targeted to provide insights to WFP Country Office, Regional Bureau and its Development Partners.
A mixed methods approach was adopted for the evaluation. Corresponding to the baseline and midline evaluation design, the endline evaluation was descriptive cross-sectional. Structured and semi-structured interview schedules, checklists for Key Informant Interviews (KII) and Focus Group Discussions (FGD) was used for data collection. In addition, secondary literature review was also done. Gender Equality and Social Inclusion (GESI) assessment was conducted using participatory techniques- mobility mapping exercise, attitude mapping and field observations during FGDs. The list of respondents engaged in the evaluation and sample distribution is given in annexure 7 and 8.
A multi-cluster random sampling strategy was adopted for the evaluation. In the first stage, Village Development Committees (VDC) were selected within the 6 programme districts, followed by selection of schools in the selected VDC and then selection of required number of students, headteachers, teachers, cooks, storekeepers and parents in the sampled schools. A total of 112 schools were covered for the evaluation. Detailed sampling strategy and distribution is annexed (annex 8). Pilot testing of tools was conducted to ensure the validity of the tools in terms of their appropriateness, specificity and measurability.
The evaluation encountered some difficulties. First, in availing accurate and updated school level data which had implications on sample selection. This led to lesser sample coverage than expected. Second, a conscious decision was made by the evaluation team and WFP to adopt reliability and validity tested tool for EGRA assessment to measure learning outcomes. The National EGRA tool was used instead of the World Education (WE) EGRA tools, which was used during baseline 2015. This highlights a methodological change from baseline to endline.
Third, the evaluation team could not get hold of raw baseline data to make statistical assessment, especially between baseline and endline. Hence, it was difficult to arrive at accurate baseline values in the report due to inconsistency in the figures.
The overall findings of the evaluation suggest that FFEP or School Meal Programme (SMP) is a holistic approach to programme operation. Over the last four decades, the SMP has been a driving force behind increasing school enrolment and continues to be so. However, the last phase of the programme contributed much towards not only ensuring school enrolment but largely on enhancing quality education service delivery. Amalgamation of Early Grade Reading Programme (EGRP), School Infrastructure Development Programme (SIDP), Digital Literacy Programme (DL) into SMP has enabled better learning environment at Schools.
The SMP design is appropriate in terms of the need of the beneficiaries and geographic remoteness. The students and parent characteristics in terms of sex, caste and religion are found proportionate to population reflecting adequate coverage. The characteristics of headteacher, teacher and cooks reflect skewedness indicating structural underpinnings which the programme can focus upon.
The SMP does not directly focus upon gender parity and inclusiveness as they have been at par as a result of several other interventions, however, the programme has aimed at contributing towards these issues indirectly by addressing social taboos.
The programme has been efficiently handling the logistical arrangement to transport food to schools. Leakages in this system have remained limited and WFP together with Government counterparts have ensured that the logistical mechanisms run smoothly. The findings however suggest, for a programme to run successfully engagement of community and sense of ownership towards the programme needs more emphasis. This is critical in ensuring efficiency of the programme, as the community play a vital role in transporting food commodities from the final delivery points (FDP) to school. In addition, the School Management Committee and Food Management Committee, responsible for the monitoring of the programme on a day-to-day basis, comprises of community members. This also will have implications on programme sustainability as WFP graduates handling over the SMP to government and communities.
In terms of cost-efficiency, the DL component is highly cost-intensive and has a limited coverage narrowing the scope for outcomes. A revisit in the programme design and resource allocation can be done. SIDP is also a broad resource-based component but its efficiency in terms of cost effectiveness is optimum. It has a direct linkage with the meals programme, helps in ensuring safe and secure storage of cooking commodities, hygienic preparation of food through well-ventilated, separate cooking rooms. Investment in SIDP have an overall broader impact and in many ways, it holds the SMP together
The programme has been effective is improving the learning outcomes of school aged children during the programme period. Around 22.7 percent of the students correctly read 45 words or more per minute and were considered as proficient in reading comprehension- a key result outcome. Significant differences (p<0.05) were observed across Listening Comprehension, Letter Sound Knowledge, Matra Reading, Non-word reading and Oral Reading Comprehension across schools with different programme component composition. Of which, students in the schools with SMP, WASH and EGR scored higher than the rest. Comparison of endline results with baseline 2015, show substantial change in the literacy outcome with more than 20 percentage point difference.
The subtask assessment within the EGRA suggest that the students are performing better in letter sound and matra reading, but knowledge is not resulting into comprehension. The students are able to recognize alphabets and vowels but are not able to read words, paragraph and also provide answers by comprehending the text read to them or read by themselves. The lessons taught in the early grade suggest that teachers (more than 60%) are teaching sounds of letter, differences and similarities in sounds and vocabulary, hence, resulting is the related aspect of learning.
On the Health and Dietary Practices, another key results area of SMP, the findings suggest that overall 66% of the children at par with the dietary diversity status, that is, having at least four food groups in a day of the ten food groups (Feed the Future Minimum Acceptable Diet). Dietary diversity score for students continues to be on the lower side as no significant difference is found from the baseline to endline. However, an upward movement is seen as more students (62%) fall into the medium dietary diversity category (DDS 4-6) from the low category (DDS ≤3) which is higher than the baseline figures (47%). The practices of students suggest that 95% of the children have mid-day meals at school. No disparity in terms of caste, religion and gender was seen in food distribution. However, the measure of food given to student of different student group is not age-appropriate.
Knowledge of students on health and hygiene has significantly increased over the project period. The baseline figure suggests that 66% of students could mention at least three good health and hygiene behaviours, the proportion increased to 87% during the endline. Girls and boys were similarly aware of WASH behaviours however, girls did not know much about menstrual hygiene. Most students, boys and girls practiced at least one health and hygiene behaviours, notable being washing hands (73%).