This is under maintenance, please use the Acceptance environment


OVERVIEW

DG INTPA has released a comprehensive “Evaluation Handbook” aimed at guiding the preparation, launch, and management of evaluations.


What is it?

The 2024 edition of the Evaluation Handbook is a revision of the Methodological Guidance published in 2006 and reflects the significant progress made in the field of evaluation over the past two decades. It incorporates contemporary evaluation literature, international best practices and standards, and leverages the wealth of experience and good practices.

The Handbook represents a comprehensive source of information on how evaluation is performed at DG INTPA and the Service for Foreign Policy Instruments (FPI). To ensure maximum user-friendliness, the information presented in the Handbook is of a ‘hands-on’ nature, placing the emphasis on what has to be done, when it has to be done and how to go about doing it. The Handbook is built around the processes guiding evaluation and hyperlinks are provided to connect to:

  • other parts of the Handbook where more detailed explanations on evaluation approaches, methods and tools as well as related information can be found;
  • online guidance, forms and templates to be used; and
  • the OPSYS web portal and its evaluation section.

The Handbook can thus be read sequentially or as different stand-alone parts.

By navigating and understanding the different phases of the evaluation processes, the user will have the possibility to access different elements of the evaluation theory as well as support and guidance responding to their specific needs. The methodology aims at treating complex concepts and processes with clarity and brevity while offering the reader the possibility to deepen her/his knowledge with additional readings.


What can it be used for?

The primary objective of the Evaluation Handbook is to serve as a reference document to accompany evaluation managers and external evaluation teams through the different phases of an evaluation, from the planning, all the way through their evaluations until dissemination of results, by offering a wide range of different methods, approaches and tools. It provides hands-on practical guidance for evaluation management across the entire process of an evaluation – its preparation, implementation and follow-up activities - detailing the tasks to be accomplished (and by whom), the steps to be followed and the (mandatory or recommended) templates to be used.

The Handbook also helps to understand the evaluation function and its role within the intervention cycle. By supporting the planning and implementation of rigorous and transparent evaluations, the Handbook contributes to the EU’s broader goals of accountability, learning, and continuous improvement in international partnerships.


When can it be used?

It can – and should be used – at each and every step of the evaluation cycle, from the planning and preparation of an evaluation to its implementation and then dissemination and follow-up activities.

In addition, the Handbook can be used at any time during the cycle of an intervention as a valid source of references for key evaluation concepts and theories, thus helping planning and implementing interventions with evaluation in mind.


Who can use it?

This Handbook is primarily aimed at:

  • evaluation managers in EU delegations and at headquarters;
  • monitoring and evaluation focal points in EU delegations;
  • external evaluation teams and contractors implementing EU-managed evaluations;
  • experts and stakeholders who have been asked to serve on evaluation reference groups; and
  • partners and other stakeholders who wish to understand how evaluation is conducted in external action at the European Commission.

The Handbook can also serve as an important reference document for other evaluation stakeholders, including but not limited to other donors, partner governments, the private sector and civil society and the professional evaluation community.


What are its strengths?

The Handbook brings together in a cohesive and comprehensive way, evaluation theory and practice. Its digital format facilitates its use throughout the evaluation processes, while the use of hyperlinks provides quick access to online guidance, forms and templates to be used; to the OPSYS web portal and its evaluation section, as well as to other parts of the Handbook where more detailed explanations on evaluation approaches, methods and tools can be found.

The Handbook is a living document that will be adapted over time to reflect evolving practice and needs.


PRACTICAL APPLICATION

Key elements

The Evaluation Handbook consists of four chapters, one annex (on budget support) and a comprehensive glossary:

  • Chapter 1: Role of evaluation in DG INTPA and FPI introduces the main concepts; describes the various uses and types of evaluations and their primary users; and situates these within the intervention cycle.
  • Chapter 2: Managing an evaluation provides hands-on practical guidance for managing evaluations through the six-phase evaluation process (preparatory phase, inception phase, interim phase, synthesis phase, dissemination phase and follow-up phase). It is divided into eight sections, each correlating to an evaluation phase along with (i) an introductory section describing these phases and the main evaluation stakeholders and (ii) a section on the cross-cutting issue of quality assurance.
  • Chapter 3: Approaches, methods and tools supplements the how-to guidance of Chapter 2 with detailed explanations, examples, rationales and techniques about topics mentioned elsewhere in the Handbook. In particular, it provides a ‘deep dive’ into evaluation approaches, evaluation criteria and questions, evaluation methodologies, data collection tools and management, and data analysis.
  • Chapter 4: Ethics in evaluation underlies all of the material presented in the preceding chapters. Conducting evaluations in an ethical manner is imperative. This chapter discusses fundamental ethical principles and considerations in evaluation and presents proactive, hands-on guidance for specific evaluation aspects and contexts.


Requirements

Data/information

A good evaluation design should consider three sets of factors: (i) the nature of what is being evaluated, (ii) the type of evaluation, and (iii) the resources and constraints. The latter includes the availability, relevance and quality of existing data; the availability of funding; and the time requirements on operational staff and stakeholders such as intended beneficiaries or partner organisations. For these reasons, the specific requirements of each evaluation (what to evaluate, when and how) will need to be defined on a case-by-case basis. Likewise, templates and tools need to be tailored to the specific needs and uses of each evaluation, building on the information and tips provided in the Handbook.









Time

Evaluations can be conducted at different times of the intervention cycle and as a result, serve different purposes:

  • Mid-term evaluations (performed mid-way during implementation) should focus on progress to date. By explaining why progress is happening, or is not happening as planned, they should provide recommendations on how to improve the intervention during its residual duration in order to achieve the expected objectives, considering problems and opportunities. They should also serve to prepare new interventions and encompass both forward- and backward-looking perspectives.
  • Final evaluations take place shortly before the operational closure of an Intervention and should contribute to accountability by providing an assessment of the results achieved and contribute to learning by understanding what have been the factors that made possible or hindered the achievement of results and by identifying any key lessons that would lead to improved future interventions in the country/region/sector of operation and/or elsewhere.
  • Ex-post evaluations take place one to two years after the operational closure of an Intervention. They should focus on the impacts (expected and unexpected) and sustainability of a given intervention so as to draw conclusions that may inform further Interventions.

Skills

EU evaluation managers should possess the necessary knowledge to successfully plan and manage evaluations. To strengthen the skills of evaluation managers, DG INTPA provides a comprehensive training package on “Managing an Evaluation Process”.

As for the evaluation team, the evaluation Terms of Reference (ToR) should emphasize the use of highly qualified evaluators. Beyond this, the expertise requirements should be tailored to the specific needs, sector/themes and context of the evaluation. A dedicated How-to-Guide on ‘Evaluation team profile’ has been developed to assist evaluation managers in developing the ‘Team Profile’ section of an evaluation ToR.

Facilities and materials

The required methodological approach and tools to be used for data collection and analysis will depend on the specific requirements laid down in the evaluation ToR.

Financial costs and sources

The financial means to carry out an evaluation should be included in the intervention budget and the evaluation budget should be commensurate to the scope and requirements of the evaluation. Important to note that in addition to the availability of funding, the undertaking of evaluations implies time requirements for the evaluation managers and other key evaluation stakeholders (reference group members, key informants, etc).

Tips and tricks

N/A


RESOURCES

Where to find it

Evaluation Handbook


Complementary guides, methodologies and tools

new Evaluation Wiki

For INTPA Staff: For methodological support throughout the evaluation cycle, from planning and designing, to conducting your evaluations, including its dissemination and follow-up, please contact our helpdesk for advice: INTPA-EVALUATION-SUPPORT@ec.europa.eu

OPSYS Evaluations: The OPSYS EVAL tool is now integrated into OPSYS, so as of September 2023 and following the issuing of the related instruction note, all new Evaluations will be created and managed in OPSYS EVAL.

My INTPA: Project & Programme Evaluations (europa.eu)

INTPA Academy: training on Monitoring, Reporting and Evaluation for Team Europe Initiatives

EU Learn:  Training on Managing an Evaluation Process  - Module 6 of the Fundamental Training Package on Intervention Cycle Management (6 Modules)

The European Commission (EC), DG NEAR: Programming, M&E WIKI  

The European Commission (EC), DG INTPA D4: Evaluation in hard to reach areas   

Capacity4Dev: Monitoring & Evaluation focal points 

The Organisation for Economic Development and Cooperation, Development Assistance Committee (OECD DAC): Evaluation Criteria (revised in 2019)