This is under maintenance, please use the Acceptance environment


OVERVIEW

What is it?

It is an initiative led by DG INTPA to reflect on the possibilities offered by 'innovative' evaluation methods to respond to the difficulties that traditional evaluation missions inevitably face in "hard to reach areas". A hard-to-reach area (HRA) is an area that is difficult to access due to conflict, man-made or natural disasters, or other physical, logistical, security or health-related obstacles; this definition also includes Fragile and Conflict-Affected states (FCAS).

The HRA initiative started with a Cycle of 7 Conferences on Evaluation in hard-to-reach areas. The main lessons learned and the methods explained during the conferences were grouped into a Call-to-Action paper on Evaluation in Hard-to-Reach Areas. The entire initiative is now collected in an online space on Capacity4Dev. This space is part of a larger initiative called Evaluation in Crisis.


What can it be used for?

This initiative provides a call to Operational and Project Managers to keep evaluating, even when the areas of intervention are hard to reach, or when local circumstances make it difficult to gather information or conduct activities in the field (e.g., conflict-affected areas, physical inaccessibility, health-related hindrances, etc.), given that it is safe to do so. These considerations must be shared and discussed with evaluation contractors, to allow them to adapt their approach and evaluation methods to better address this hard-to-reach local environment.    


When can it be used?

This initiative provides the initial push when there is a need to adapt evaluation methods to fragile contexts and hard-to-reach areas. A traditional approach to evaluation in fragile or conflict-affected states and, more generally, in hard-to-reach areas is destined to fail: the number of professional evaluators available to travel to these countries is limited and the security risks during in-country travel make conventional field missions unrealistic, particularly in the most remote areas. In response to this, various development partners have started encouraging the use of methods and techniques, tried and tested in other fields, but that are innovative in an evaluation context. These include the use of geo-spatial data, surveys administered by local enumerators, voice or data surveys by phone or tablet, location tracking, communication through online platforms, etc.


Who can use it?

This initiative is addressed to evaluation managers in the EU Delegations and in DG INTPA headquarters, as well as partners and professional evaluators in the development field. It may also provide useful insights to colleagues from other Commission DGs and services.


What are its strengths?

  • Very helpful to open the reflection on the possibilities offered by 'innovative' evaluation methods to respond to the difficulties that traditional evaluation missions inevitably face in HRAs and the challenges that exist in applying them.
  • Gaining insight on fruitful entry points for an evaluation of an intervention in difficult contexts, as well as key takeaways from the experience gathered by different donors and evaluators when carrying out evaluations in HRAs.


What are its limitations?

  • It is not a methodological or how-to guide.
  • The innovative methods proposed here, as well as many other existing ones, are not the single or definitive solutions to carry out evaluations in HRA, they are a few methods to keep in mind, being aware of their limitations, and always in conjunction with other methods of analysis.
  • Innovative methods appear every year, with new tools, technologies, or being adapted to newer contexts; the HRA initiative aims at opening the discussion, but different or new approaches can also be applied.


PRACTICAL APPLICATION

Key elements

The HRA initiative is gathered in the Capacity4dev space on Evaluation in Hard-to-Reach-Areas, including:

  1. The cycle of seven conferences, including: the description of each conference, the speakers' profiles, their conference slides and material, as well as the video recordings.
  2. The Call-to-Action paper on Evaluation in Hard-to-Reach Areas, published in January 2020, laying out the key take-aways and techniques presented in the conferences. 
  3. The referenced literature and useful documentation.


Requirements

Data/information.

An evaluation, and more especially those conducted in highly complex or chaotic contexts, such as HRA, starts with a good and solid preparation. You must always adapt the prepared methodology to changing circumstances. Stay creative and flexible, and adapt your tools to the environment. When in doubt about the feasibility and usefulness of an evaluation, carry out an evaluability assessment.

Remember to:

  • Determine the requirements and extra skills of team members.
  • Involve national staff with appropriate skills where appropriate.
  • Design evaluation approaches and methods grounded on the knowledge of the context.
  • Consider ethical issues including Gender Responsiveness, Do No Harm and Conflict Sensitivity.
  • Assess need for using specific technical tools and their associated risks.

Timeframe.

Evaluations can be conducted at different times of the intervention cycle and as a result, serve different purposes:

  • Mid-term evaluations (performed mid-way during implementation) should focus on progress to date. By explaining why progress is happening, or is not happening as planned, they should provide recommendations on how to improve the intervention during its residual duration in order to achieve the expected objectives, considering problems and opportunities. They should also serve to prepare new interventions and encompass both forward- and backward-looking perspectives.
  • Final evaluations take place shortly before the operational closure of an Intervention and should contribute to accountability by providing an assessment of the results achieved and contribute to learning by understanding what have been the factors that made possible or hindered the achievement of results and by identifying any key lessons that would lead to improved future interventions in the country/region/sector of operation and/or elsewhere.
  • Ex-post evaluations take place one to two years after the operational closure of an Intervention. They should focus on the impacts (expected and unexpected) and sustainability of a given intervention so as to draw conclusions that may inform further Interventions.










Skills.

Evaluations in HRA that use some of the methods proposed during this initiative, will need the evaluation team to have the necessary skills to be able to apply the identified methodology and be able to analyse the results obtained and draw conclusions. These skills will need to be addressed and requested in the evaluation’s terms of reference (ToR), or related commissioning document.

Facilities and materials. 

Some of the techniques presented require specific materials (e.g. mobile phones, tablets), facilities (e.g. field locations to gather enumerators and load their data), or even registration to online resources (e.g. professional satellite data) or analysis tools (e.g. Artificial Intelligence tools). These should be foreseen in the evaluation contractors’ approach and budgeting.

Financial costs and sources.

The use of the proposed methods, as well as the necessary skills, in the evaluation team have to be taken into account in the terms of reference, so that the additional costs of the use of these methods can be planned for by the responding consortia in the RfS.

Tips and tricks

Evaluations in HRA are not business as usual, which is why the evaluation team should be transparent, in their reports, about the evaluation's validity and limitations, and about the reasons why specific methodologies were chosen. This includes limitations encountered by the evaluators when accessing beneficiaries in HRAs, or in obtaining comprehensive data on results, or doubts they may have on the validity of their findings.


EU RESOURCES


  For INTPA Staff: For methodological support throughout the evaluation cycle, from planning and designing, to conducting your evaluations, including its dissemination and follow-up, please contact our helpdesk for advice: INTPA-EVALUATION-SUPPORT@ec.europa.eu




For further information, any revision or comment, please contact INTPA-ICM-GUIDE@ec.europa.eu

Published by INTPA.D.4 - Quality and results, evaluation, knowledge management. Last update May 2025