Evaluation in hard to reach areas

OVERVIEW

What is it?

It is an ESS-INTPA initiative to reflect on the possibilities offered by 'innovative' evaluation methods to respond to the difficulties that traditional evaluation missions inevitably face in "hard to reach areas". The initiative started with a Cycle of 7 Conferences on Evaluation in hard-to-reach areas. The main lessons learned and the methods explained during the conferences have been grouped into a document called Call-to-Action paper on Evaluation in Hard-to-Reach Areas. The entire initiative is now collected in an online space on C4D. This space is part of an even more ambitious initiative called Evaluation in Crisis.

What can it be used for?

This initiative provides a call to INTPA Operational and Project Managers to evaluate their interventions even in Hard-to-Reach Areas. It is a fundamental element to take into account when launching an evaluation in Hard-to-Reach Areas and as supporting information for the evaluation contractors. A hard-to-reach area (HRA) is an area that is difficult to access due to conflict, man-made or natural disasters, or other physical, logistical, security or health-related obstacles; this definition also includes FCAS.

When can it be used?

This initiative provides the initial push when adapting the evaluation methods to fragile contexts and hard-to-reach areas is needed.
A traditional approach to evaluation in fragile or conflict-affected states and, more generally, in hard-to-reach areas is destined to fail: the number of professional evaluators available to travel to these countries is limited and the security risks during in-country travel make conventional field missions unrealistic, particularly in the most remote areas. In response to this, various development partners have started encouraging the use of methods and techniques that are innovative in an evaluation context. These include use of geo-spatial data, surveys administered by non-specialised local enumerators, phone/tablet voice or data surveys, location tracking, communication through online platforms, etc. a call to INTPA Operational and Project Managers to evaluate their interventions even in Hard-to-Reach Areas, and to evaluation contractors.

Who can use it?

This initiative is addressed to evaluation managers in the EU Delegations and in INTPA headquarters, as well as partners and professional evaluators in the development field. It may also provide useful insights to colleagues from other Commission DGs and services.

What are its strengths?

• Very helpful to open the reflection on the possibilities offered by 'innovative' evaluation methods to respond to the difficulties that traditional evaluation missions inevitably face in "hard to reach areas"
• Gaining insight on fruitful entry points for an evaluation of an intervention in difficult contexts.

What are its limitations?

• It is not a guide.
• The innovative methods proposed here, as well as many other existing ones, are not the definitive solution to carry out an evaluation in HRA, just one more method to keep in mind, being aware of their limitations and in always in conjunction with other methods of analysis.

PRACTICAL APPLICATION

Key elements

The HRA initiative is gather in the C4D space Conferences on Evaluation in hard-to-reach areas. Conferences on Evaluation in hard-to-reach areas. In the space, you find all the relevant information on the cycle of conferences, including:

1. The Call-to-Action paper on Evaluation in Hard-to-Reach Areas, published in January 2020, laying out the key take-aways and techniques presented in the conferences. 
2. In the Wiki section, you will find the cycle of conferences' presentation paper, including: the description of each conference, the speakers' profiles, their conference slides and material, as well as the video recordings.
3. The referenced literature and useful documentation can be found in the Documents section.
4. Finally, the conferences' pictures are available in the Media section.

Requirements

Data/information.

An evaluation, and more especially a HRA evaluation, starts with a good and solid preparation. You must always adapt the prepared methodology to changing circumstances. Stay creative and flexible and adapt your tools to the environment. When in doubt about the feasibility of an evaluation, carry out an evaluability assessment. Remember to:

• Determine the requirements of team members.
• Involve national staff with appropriate skills.
• Design evaluation approaches and methods.
• Consider ethical issues.
• Assess need for using specific technical tools.

Time.

Evaluations can be conducted at different times of the project cycle and as a result, serve different purposes:

• Mid-term evaluations (performed mid-way during implementation) should focus on progress to date and explain why progress is happening or is not happening as planned; they provide recommendations on how to improve the Intervention.
• Final evaluations take place shortly before the operational closure of an Intervention and should contribute to accountability by providing an assessment of the results achieved and contribute to learning by understanding what have been the factors that made possible or created obstacles to the achievement of results.
• Ex-post evaluations take place one to two years after the operational closure of an Intervention. They should focus on the impacts (expected and unexpected) and sustainability of a given Intervention as to draw conclusions that may inform further Interventions.

Skills.

Evaluations should emphasize the use of highly qualified evaluators, additionally, evaluations in HRA that use some of the methods proposed during this initiative, will need the evaluation team to have the necessary skills to be able to put the identified methodology into operation and to be able to analyse the results obtained and draw conclusions.

Facilities and materials. N/A

Financial costs and sources.

The use of the proposed methods as well as the necessary skills in the evaluation team have to be taken into account from the writing of the terms of reference so that the additional costs of the use of these methods are considered by the participating consortia in the RfS.

Tips and tricks
Evaluation in HRA are always a challenge and as such the evaluation should be transparent about the evaluation's validity and limits, and the methodologies chosen. This includes limits that evaluators encountered when accessing beneficiaries in HRAs and in obtaining comprehensive data on results. Evaluators and those commissioning evaluations should be ready to accept hypotheses based on uncertain findings. This includes identifying areas requiring further investigation whenever the evaluation cannot be conclusive in all its aspects.

RESOURCES

Where to find it

Capacity4dev: Evaluation in Hard-to-Reach Areas.

Complementary guides, methodologies and tools

The European Commission (EC), DG INTPA. INTPA Evaluation Methodology

Capacity4Dev: Evaluation in Crisis