View Source

1. Evaluations (EVAL) in OPSYS

1.1 Introduction to OPSYS EVAL

The Evaluation tool and library (EVAL) is used for the management of project and programme, geographic, thematic or strategic evaluations contracted by DG INTPA, DG ENEST, DG MENA and FPI (both in Headquarters and in Delegations), as well as evaluations done by external partners.

The decision to develop EVAL was taken by the European Commission based on its own needs assessment, but the tool also responds to the recommendations of the European Court of Auditors (ECA) on EU External Action's evaluation and results-oriented monitoring systems.

EU External Action has developed a robust evaluation approach where the quality of an evaluation depends on the application of appropriate methodology and professional standards. You can refer to this methodology published on Capacity4Dev, for all evaluation phases.

Evaluations are complementary assessments involving data collection, performance assessment, reporting and learning. EVAL's primary purpose is to provide a systematic, comprehensive, and effective tool to record and report evaluations, as accurate evaluations provide an improved decision-making process for better outcomes.

As of September 2023, the EVAL tool is integrated in to OPSYS (OPSYS EVAL), and following the issuing of the related instruction note, all new evaluations will be created and managed in OPSYS. All evaluations created in the legacy system EVAL module (that have reached the "Inception" phase) will continue to be managed in that system until completion.

EVAL OPSYS is accessible from:

The Funding and Tender Opportunities Portal for external EVAL Contractors, EVAL Experts, and EVAL Reference Group Members; or
OPSYS – MyWorkplace for internal EC EVAL users (Evaluation Managers, M&E Focal Points, Heads of Cooperation/Heads of Delegations/Heads of Sector/Heads of Unit, internal Reference Group Members, Administrators, and EC users).

The OPSYS EVAL tool for externals consists of one main section, Evaluations, where all created Evaluations are listed, can be accessed, and managed.

EVAL Contractors have access to their Evaluations (once the corresponding Evaluation Contract is signed) in statuses/steps "Inception", "Interim", and "Synthesis";
EVAL Experts have access to Evaluations (once the corresponding Evaluation Contract is signed) for which they have been selected and in Evaluation statuses/steps "Inception", "Interim", and "Synthesis";
EVAL Reference Group Members (external) have access to Evaluations for which they have been added as evaluation stakeholders, and in all Evaluation statuses/steps.

This content provides a description of all the steps involved in the Evaluations Workflow of OPSYS, but focuses on the step-by-step instructions and information specific to EVAL Contractors, EVAL Experts, and EVAL Reference Group Members.

The contracting procedure for the Evaluation Contract, and information concerning the signing of the Contract, the management of Contract Deliverables and Contract Amendments (if applicable), will be linked herein to the relevant sections.

1.2 Overview of the EVAL Workflow

Creating evaluations are an integral part of Intervention Cycle and its Management. Evaluations can be conducted at different times of the project cycle and as a result, serve different purposes:

EXACT External Wiki - EN > EVAL in OPSYS > image-2023-10-24_15-2-38.png (EVAL)

Ex ante – Performed before the start of an Intervention.
Mid-term Evaluations – Performed mid-way during implementation of an Intervention or even earlier on in case of need.
Final Evaluations – Take place shortly before the operational closure of an Intervention.
Ex post Evaluations – Take place one to two years after the operational closure of an Intervention.
Mixed – Evaluations that do not fall under any of the previous categories are considered "mixed" evaluations in terms of timing; since the scope of the evaluation is in different phases of progress. Evaluations can be conducted before an intervention(s) starts (ex ante evaluation), and/ or at the midpoint (midterm evaluation) or conclusion (final evaluation) of the intervention (i.e. six months before/after the intervention’s completion date), or following implementation (ex post evaluation) (i.e. at least one year after the intervention’s completion date). Evaluations that do not fall under any of these three categories are considered "mixed" evaluations in terms of timing; since the scope of the evaluation is in different phases of progress.

Overall, the creation and management of the these different types of evaluations take place in OPSYS EVAL, and are part of the Evaluation Workflow with various phases, steps and actions that need to be performed in the system.

EXACT External Wiki - EN > EVAL in OPSYS > image-2023-10-24_15-4-29.png (EVAL)

1.3 Detailed Phases of the EVAL Workflow

Step/Status

Actors involved

Actions

Notifications in OPSYS EVAL

Planning Phase

Planned (1)

Evaluation Manager (internal), EVAL Reference Group Members

In OPSYS EVAL (internal):

The Evaluation Manager will create the Evaluation in OPSYS EVAL, and assign it to an Intervention.
The Evaluation Manager enters the minimum required information to the Evaluation, and adds the evaluation stakeholders (Evaluation backup, M&E Focal Point, Reference Group Members, etc).

Please note that Reference Group Members must have an active EU Login account prior to being added as an evaluation stakeholder to the Evaluation. For more information on how to create and/or manage your EU Login account, please view the EU Login Guide.

The Evaluation Manager adds the Evaluation to the Operational Evaluation Plan (OEP).
The Evaluation Contract is planned.

Once the creation of the Evaluation is completed and the Evaluation Contract is planned, the Evaluation Manager will submit the Evaluation in OPSYS EVAL to the next status/step "TOR".

The Evaluation has been created

TOR (2)

Evaluation Manager (internal), EVAL Contractor, EVAL Expert, EVAL Reference Group Members

The Evaluation is now launched.

In OPSYS (internal):

The contracting procedure is initiated, and the Request for Specific Contract for the Evaluation is launched and published to Framework Contractors on the Funding and Tenders Portal.

In the Funding and Tenders Portal (external):

Framework Contractors can respond to an invitation to participate by acknowledging the invitation and expressing willingness to participate.
Framework Contractors complete their submissions and submit their offers.

In OPSYS (internal):

Once the Specific Contract is awarded to the EVAL Contractor, the Contract is prepared using the Terms of Reference (TOR) templates of the Framework Contract, and with the support of the methodological guidance.
The approval of the TORs is requested from the external Reference Group Members of the Evaluation.

In the Funding and Tenders Portal (external):

The EVAL Contractor confirms that the EVAL Expert(s) are available, and that they have their EU Login details.
Once the Specific Contract is prepared and approved by the Contracting Authority, the EVAL Contractor signs the Contract, followed by the final signature of the Contract by the Contracting Authority.
The EVAL Contractor confirms that the EVAL Expert(s) have their EU Login details.

Once the above-listed steps are performed and the Specific Contract is signed by both parties, the Evaluation Manager will submit the Evaluation in OPSYS EVAL to the next status/step "CONTRACTED".

Contracted (3)

Evaluation Manager (internal), EVAL Contractor, EVAL Expert, EVAL Reference Group Members

In the Funding and Tenders Portal (external):

After the signing of the Specific Contract, the EVAL Contractor can access and download the signed Specific Contract from the Funding and Tenders Portal.
If requested, the bank guarantee is provided by the EVAL Contractor.

In OPSYS (internal):

If applicable, Pre-financing Payment process is initiated automatically by the Contracting Authority.

In OPSYS EVAL (internal):

The Evaluation Manager will verify the stakeholders' information (including the EVAL Reference Group Members) for the Evaluation, and update if needed.
The Evaluation Manager will then link the Specific Contract to the Evaluation.
The Evaluation Manager can verify from the Contract Information section of the Evaluation the Contract by using a direct URL link to the Contract.
If defined in the Specific Contract, the EVAL Contractor and EVAL Expert(s) are automatically retrieved from the Specific Contract to the Evaluation.
When the Specific Contract is linked to the Evaluation, the Evaluation Manager will link the required Contractual deliverables to their associated Evaluation "Inception", "Interim" and "Synthesis" statuses/steps.

Once the Specific Contract is linked to the Evaluation and all the above steps are completed, the Evaluation Manager will submit the Evaluation in OPSYS EVAL to the next status/step "INCEPTION".

Implementation Phase

Inception (4)

Evaluation Manager (internal), EVAL Contractor, EVAL Expert

In OPSYS EVAL (external):

The Evaluation is now accessible to the EVAL Contractor and EVAL Expert(s) from OPSYS EVAL (via Results > Monitoring & Evaluation > Evaluations section of the Funding and Tenders Portal).

In the Funding and Tenders Portal (external):

The management of Contract deliverables by the EVAL Contractor is performed from the Specific Contract.
During the "INCEPTION" status/step of the Evaluation, the EVAL Contractor uploads and submits an Inception Report deliverable before the set deadline.

In OPSYS (internal):

Once the deliverables are submitted by the EVAL Contractor, a notification is sent to the Contracting Authority to review the deliverables.
The Contracting Authority can add comments to the deliverable, and either send the deliverable back to the EVAL Contractor for revision, or approve the deliverable.

In OPSYS EVAL (internal):

When the deliverable is approved, the Evaluation Manager selects the relevant deliverable for the Evaluation to indicate that the deliverable has been correctly submitted, and the Evaluation can proceed to the next step.

After successfully selecting the Inception Report (and other(s) if applicable) deliverable in the Evaluation, the Evaluation Manager will submit the Evaluation in OPSYS EVAL to the next status/step "INTERIM".

The Evaluation’s inception phase has started

Interim (5)

Evaluation Manager (internal), EVAL Contractor, EVAL Expert

In the Funding and Tenders Portal (external):

The management of Contract deliverables by the EVAL Contractor is performed from the Specific Contract.
During the "INTERIM" status/step of the Evaluation, the EVAL Contractor uploads and submits an Interim Report deliverable (and any other required deliverable for this Evaluation status/step) before the set deadline.

In OPSYS (internal):

Once the deliverables are submitted by the EVAL Contractor, a notification is sent to the Contracting Authority to review the deliverables.
The Contracting Authority can add comments to the deliverable, and either send the deliverable back to the EVAL Contractor for revision, or approve the deliverable.

In OPSYS EVAL (internal):

When the deliverable is approved, the Evaluation Manager selects the relevant deliverable for the Evaluation to indicate that the deliverable has been correctly submitted, and the Evaluation can proceed to the next step.

After successfully selecting the Interim Report (and other(s) if applicable) deliverable in the Evaluation, the Evaluation Manager will submit the Evaluation in OPSYS EVAL to the next status/step "SYNTHESIS". The Contracting Authority manages the interim payment(s).

The Evaluation’s interim phase has started

Synthesis (6)

Evaluation Manager (internal), EVAL Contractor, EVAL Expert

In the Funding and Tenders Portal (external):

The management of Contract deliverables by the EVAL Contractor is performed from the Specific Contract.
During the "Synthesis" status/step of the Evaluation, the EVAL Contractor uploads and submits a Final Report deliverable (and any other required deliverable for this Evaluation status/step) before the set deadline.

In OPSYS EVAL (external):

A Draft conclusion, lessons learned and recommendations template to be completed becomes downloadable by the EVAL Contractor and EVAL Expert(s) for the Evaluation from OPSYS EVAL (via Results > Monitoring & Evaluation > Evaluations section of the Funding and Tenders Portal). Alternatively, the Draft conclusion, lessons learned and recommendations form can be completed online. For both the template and online form, conclusions, lessons learned and recommendations can be grouped and prioritised.
The EVAL Contractor confirms that the Conclusion, lessons learned and recommendations template or form is set as final for the Evaluation.

In OPSYS (internal):

Once the deliverables are submitted by the EVAL Contractor, a notification is sent to the Contracting Authority to review the deliverables.
The Contracting Authority can add comments to the deliverable, and either send the deliverable back to the EVAL Contractor for revision, or approve the deliverable.

In OPSYS EVAL (internal):

When the deliverable is approved, the Evaluation Manager selects the relevant deliverable for the Evaluation to indicate that the deliverable has been correctly submitted, and the Evaluation can proceed to the next step.

After successfully selecting the Final Report (and other(s) if applicable) deliverable in the Evaluation, and the Conclusions, lessons learned and recommendations are submitted by the EVAL Contractor or EVAL Expert(s), the Evaluation Manager will submit the Evaluation in OPSYS EVAL to the next status/step "COMPLETED". The Contracting Authority manages the final payment and clearing of the pre-financing (if applicable).

The Evaluation’s synthesis phase has started

Evaluation Completed

The Evaluation has been completed

Response of Service and Follow-up Phase

Response of the services
Report (7)

Evaluation Manager (internal)

In OPSYS EVAL (internal):

Once the Evaluation is completed, the Evaluation Manager fills in the online template Response of the services Report with the input of the services involved.
If this is the last step foreseen for the Evaluation (no Follow-up one year later is planned), the Evaluation is finalised.

In OPSYS (internal):

The Contracting Authority closes the Specific Contract.

The 'Response of the Services' has been completed

Follow-up one year later Report (8)

Evaluation Manager (internal)

In OPSYS EVAL:

A Follow-up phase is carried out only if applicable and one year later to monitor the impact of the inputs disseminated across the stakeholders.
The Evaluation Manager fill in the online template Follow-up on year later Report for the Evaluation with the input of the services and other actors involved.
The Evaluation is concluded.

1) Evaluation – Follow-up one year later (365 days passed since the final 'Response of the Services' notification

2) 'Follow-up one year later Report' has been completed

1.4 Roles and Responsibilities for EVAL

Units INTPA D4 (Performance, Results and Evaluation; Internal Communication, Knowledge Management and Collaborative Methods) and NEAR A4 (Coordination of financing instruments - performance, results, and evaluation) oversee the overall coordination of evaluations.

On the INTPA side, there are three levels of evaluation conducted:

EXACT External Wiki - EN > EVAL in OPSYS > image-2023-10-24_15-7-58.png (EVAL)

A meta-evaluation is a systematic analysis of evaluations conducted to bring together core learning on similar topics (EEAS and EC, 2014). It can focus on either intervention or strategic evaluations or a combination of both. Meta-evaluations synthesis recurring findings, conclusions and recommendations from different evaluation reports in order to provide inputs into strategic planning and institutional learning.

For any additional methodological support on the evaluation processes or documents, please contact:

For DG INTPA - INTPA EVALUATION SUPPORT
For DG NEAR - NEAR EVALUATION HELP DESK
For FPI - FPI EVALUATION SUPPORT

For technical support concerning OPSYS EVAL, please contact EC External Relations Application Support.

For technical support concerning the external Funding and Tender Opportunities Portal, please contact EC Funding and Tender Service Helpdesk.

For general IT support, please contact EC HELPDESK IT.

Also join the C4D "Evaluation Correspondent Network", which includes DG INTPA staff, who manage, plan or advise on evaluations conducted in EU Delegations and at Headquarters, as well as colleagues in DG ENEST, DG MENA and FPI.

EXACT External Wiki - EN > EVAL in OPSYS > image-2023-10-24_15-8-39.png (EVAL)

Please find below the list of roles and responsibilities involved in OPSYS EVAL:

Role	Responsibilities
Reference group members (Internal & External)	The reference group is presided over by the Evaluation Manager. The Reference Group (RG) is a kind of steering group for the evaluation that acts as an interface between the Evaluation Manager (Eu Delegation or EC/DG INTPA headquarter) and the Evaluation Team (to be recruited). Its members help the external evaluation team to identify and access data sources. They validate the Evaluation questions and discuss the findings and recommendations. The RG also allows the diversity of views on the intervention being evaluated to be expressed and usually takes collegial decisions. The RG is a small group consisting of the departments involved in the evaluated intervention and possibly other stakeholders, such as government departments, relevant development partners, some civil society organizations.
Evaluation Manager	The Evaluation Manager is a member of the EC service commissioning the Evaluation and is assigned by that service to manage the process on its behalf.
Evaluation Backup	Evaluation Backup acts as a substitute for the Evaluation Manager and has the corresponding permissions for all Evaluations.
Administrator	Administrator can perform all tasks/steps and all operations. The Administrator role will therefore not be indicated as able to perform a particular step/task in the workflow, as they are able to perform all by default.
Contractor	The Evaluation Contractors are the entities/consortia contracted and supervised by the Evaluation Manager to conduct the Evaluations on selected Interventions. They report the results of their Evaluations according to standardized procedures and assure quality of their outputs.
Team Leaders	External experts, responsible to conduct the Evaluations on selected Interventions.
Experts	External experts, responsible to conduct the Evaluations on selected Interventions.
M&E Focal Point	Each EUD/HQ Unit appoints a Focal Point (and backup) for monitoring (including ROM) and Evaluation. The M&E Focal Point has the corresponding permissions for all Evaluations in their Delegation or HQ Operational Unit as provided by the delegation/unit.
Head of Unit/Operations	The Head of Unit in HQ/Head of Cooperation/Operations in Delegations have access to OPSYS EVAL for oversight of the monitoring activities in the portfolio under their responsibility.

1.5 Operational Evaluation Planning

Evaluation planning is part of obligations of the EU Delegations and HQ Units. The Operational Evaluation Plan (OEP) will be submitted through EVAL module of OPSYS once a year, together with a preparation of Management Plan, and reported on within the framework of EAMR exercise. Except for INTPA, for which is no longer an obligation, as the INTPA 2022 simplification exercise removed the OEP exercise for the service.

The OEP is a specific list of Evaluations that are planned to be started, implemented by specific DG in Charge and Responsible Service. The decision of what to evaluate is in the hands of each Delegation / Unit and shall be based on their needs in terms of learning and accountability, which are to be indicated in their OEPs.

Planning is important to ensure:

That a significant portion of the Interventions portfolio is evaluated for accountability and transparency purposes, considering the legal obligation to evaluate all Interventions which entail significant spending.
That evaluations are conducted at the right time to be useful as learning and up take instruments.
That resources to be devoted to managing the evaluations are efficiently allocated within each Unit and Delegation.

EXACT External Wiki - EN > EVAL in OPSYS > image-2023-10-24_15-9-9.png (EVAL)

This internal process involves several Commission services and is supported by various information systems.
The Evaluation exercise starts with the creation of the Evaluations, which are going to be submitted as part of the Operational Evaluation Plan.

The OEP shall include all Evaluations that are planned for the particular year (i.e. in status "Planned", "TOR" or "Contracted" for the selected year):

To be managed by all Delegations and HQ Units.
Of all direct and indirect management types but to be launched by Delegations and HQ Units (with the exclusion then of Evaluations done by partners).
Of all types: Project/Programme Evaluations, Budget Support Evaluations, Country-level Evaluations, Thematic Evaluations at the delegation/unit level.
Evaluations which are already in the planned phase in OPSYS EVAL and that need to be updated (notably evaluations planned to be launched in the previous year and that they have not reached the TOR preparation phase on the year that EOP will be submitted).

Evaluations

EVAL Evaluations