ROM in OPSYS

Results-Oriented Monitoring (ROM) in OPSYS

6.1 Results-Oriented Monitoring (ROM) in OPSYS

6.1.1 Introduction to the ROM Module

The ROM module is a tool used by DG INTPA and DG NEAR (in HQ and in EU Delegations) for the effective management of Results-Oriented Monitoring (ROM) reviews on ongoing Interventions.

For detailed information on the entire ROM Review Process, please consult the ROM Handbook and Annexes.

The ROM module is now integrated into OPSYS, so as of March 2023 and following the issuing of the related instruction note, all new ROM reviews will be created and managed in OPSYS. All ROM reviews created in the legacy system ROM module will continue to be managed in that system until completion (ROM module documentation – legacy system).

The ROM module covers the full cycle of ROM reviews on Interventions including the preparation of the work plan, the implementation of ROM reviews, the delivery of ROM reports, and comments by Commission services.

The ROM module brings major improvements to the ROM Review Process (see above):

It reduces the workload for all actors, offering significant gains of time for ROM Contracts and ROM Experts (for completing the ROM reports and monitoring questions), as well as for Operational Managers (when commenting on ROM reports);
It supports the shift from a Contractor-driven ROM system to one driven by EC services;
It improves the quality of final ROM reports, by introducing the possibility for EC services to comment drafts;
It improves data quality as the information on Interventions subjected to ROM reviews are already encoded in OPSYS;
It offers an improved multi-criteria search function, based on Intervention and ROM review information.

The ROM module is accessible from:

Funding and Tender Opportunities Portal for ROM Contractors and ROM Experts; and
OPSYS – MyWorkplace for EC internal ROM users: ROM Coordinators, Operational Managers (OM), M&E Focal Points (M&E FPs), Heads of Cooperation/Heads of Section/Heads of Operations/Heads of Unit (HoC/HoO/HoS/HoU), Administrators, and EC users.

The tool is sub-divided into two sections in the Funding and Tenders Portal:

ROM Workplan – Where the Interventions are listed, from which ROM reviews are created.
ROM Reviews – Where all created ROM reviews are listed and managed.
ROM Experts – Where all ROM Experts are listed and managed.

This documentation for ROM Contractors and ROM Experts provides a description of the main operations to be performed during the ROM review workflow in the ROM module of OPSYS, but focuses on the step-by-step specific to ROM Contractors and ROM Experts.

6.1.2 Portfolio of Interventions for ROM Review

The ROM exercise starts with the establishment of the portfolio of Interventions to be submitted to the Contractors for ROM reviews. This internal process involves several Commission services and is supported by various information systems. It takes place every year and consists of:

The preparation of the list of all Interventions managed by Delegations and HQ Operational Units; and
The selection of the Interventions to be externally monitored (ROM reviewed) on indication of the internal monitoring and reporting activities.

The External Assistance Management Report (EAMR) is used as a reference for the list of Interventions to be subject to ROM review. With the information it contains on the performance of interventions (KPI 6 and KPI 7 in particular), the EAMR is also used to flag when a ROM review is needed. Finally, the ROM module in OPSYS is used to establish and manage the portfolio of Interventions for ROM review available to ROM Contractors, and subsequently to support the preparation and execution of the work plan. There exist additional inclusion/exclusion criteria for establishing the portfolio of Interventions for ROM review, which are covered in the ROM Handbook.

The External Assistance Management Report (EAMR)

Through the EAMR, Heads of Delegations (and HQ Directorates) report annually on the performance and results achieved in the implementation of EU external assistance the previous year. From the EAMR, several standard Key Performance Indicators (KPls) are computed. Two KPls are used to flag Interventions with problems and potentially in need of a ROM review, namely KPl-6 and KPl-7, through a "traffic light" scoring system used by OMs to report their view on the performance of the Intervention.

The first traffic light (KPl-6) relates to the following question: What is the expected level of scheduled resources the project will be able to use before the end of the project (< 75%, red; 75% - 90%, orange; > 90%, green)?

The second traffic light (KPI-7) relates to the following questions:

1. What is the likely level that the projects will achieve in terms of output targets (< 75%, red; 75% - 90%, orange; > 90%, green)?

2. What are the risks that - regardless of outputs achieved - the outcomes of the project will not be achieved (high, red; medium, orange; low, green)?

An Intervention is considered to have implementation problems and therefore in need of a ROM review, if at least one of the two KPls is flagged orange or red.

The EAMR is also used to indicate Interventions for which a ROM review is considered necessary by the Delegation/HQ Operational Unit, even if the traffic light on the two KPls is green. This is done through a specific question where a ROM review can be requested.

The OM in charge does not have to specify any longer in the EAMR the reasons for requesting a ROM, and only selects a yes/no option in the EAMR tool.

The portfolio of Interventions for ROM review includes four different categories:

On Sample – A representative sample of EU-funded interventions in their early stages of implementation. The sample is prepared by the ROM Coordination Units. For these Interventions, ROM reviews support the OM in confirming or updating the intervention logic and logframe matrix (including baselines and targets), reviewing the monitoring system of the IP, assessing risks, mitigation measures and fine-tuning the implementation set-up and calendar of activities, focusing on improving the performance of the Intervention.
EAMR Request – Interventions flagged as having implementation problems through the Commission’s internal reporting systems (in particular, the EAMR). In this case, the ROM review will assist the OM in addressing the main problems, identifying causes and options for solution, and making recommendations for follow-up actions.
On Demand – Other Interventions for which the Commission services request a ROM review. Several reasons can justify this request. For instance, a ROM review can be useful for Interventions which could not be visited by OMs or for which the necessary sector expertise at Delegation/HQ level may not be available at a given point in time.
Blending operations for which the Commission identifies the need for a ROM review. In OPSYS, a ROM review for Blending operations will be flagged (Yes/No value) during the creation of the ROM review, rather than being selected as a ROM review type.

Further details on the Interventions that each year become part of the ROM portfolio or are excluded from it, are available in the table below:

INCLUDED

EXCLUDED

Interventions with at least one orange or red flag for KPI-6 and/or KPI-7.
Interventions with green traffic lights in the EAMR, for which the Delegation/HQ Operational Unit requests a ROM review.
Interventions in early stages of implementation selected by HQ on a sample basis, irrespective of the traffic light they may already have on the EAMR.
Blending and Trust Fund interventions selected through an internal consultation process.
Interventions in a final stage of implementation (less than 8 months to the end) for which the Delegation/HQ Operational Unit has a justified request for a ROM review. The justification cannot be linked to delays in planning/implementing the ROM review.

Interventions that have just started and not included in the sample for ROM review in their early stage of implementation.
Interventions that are close to completion (less than 8 months to the end of the implementation phase/end date) and for which there is no justified need for ROM review.
Interventions with an orange or red traffic light, for which a mid-term-review (MTR) has recently taken place or is scheduled to take place shortly (evaluation take precedence over monitoring).
Interventions with an orange or red traffic light, for which the risks or reasons of underperformance are well known, and a ROM review would be unproductive. This can occur in cases of civil war, natural disaster, or other circumstances.

6.1.3 Overview of the ROM Review Workflow in OPSYS

Please find below an overview of the ROM Review Workflow in OPSYS:

1. The Interventions that could be subject to ROM reviews are listed in the internal OPSYS > Workplan. From this work plan, the Operational Manager (or M&E Focal Point or ROM Coordinator – see note below) will identify which Interventions require a ROM review, and then create the ROM reviews for these Interventions. Out of the list of ROM reviews created for the work plan, ROM Contractors via the external Funding & Tenders Portal can view and extract a list of planned ROM reviews (draft, confirmed, experts selected, experts confirmed), and then estimate the projects that they are able to review with the budget available under their contract.

Please note that a ROM review can only be created by:

Operational Managers or M&E Focal Points for Interventions in their Unit or Delegation;
ROM Coordinators for Interventions in their DG.

A ROM review can also only be created from an Intervention that is in “Ongoing” status.

2. Once the ROM review is created and saved, the Planning Phase of the ROM review starts with the draft ROM review. ROM Contractors can access draft ROM reviews under their responsibility from the ROM Reviews section in the Funding and Tenders Portal. ROM Core Team Experts can also access draft ROM reviews from the portal and for their respective lot.

3. The ROM Review Planning Phase consists of four statuses/steps in the system (Draft, Confirmed, Experts Selected and Experts Confirmed), each corresponding to the status of the ROM review in the workflow. Each step is managed by the ROM actor(s) responsible for the specific step. The responsible ROM actor(s) will complete the required operations for each step, and then submits the ROM review to the next step in the workflow.

During the ROM Review Planning Phase:

ROM Contractors can edit and update draft ROM reviews;
ROM Contractors will select ROM Experts for confirmed ROM reviews;
ROM Expert(s) can view the ROM reviews for which they are selected (only when the Expert is approved and the ROM review is confirmed);
ROM Core-Experts can view the ROM reviews for the respective lot for which the Experts are "core" (and only when the ROM review is confirmed).

4. Additional and optional actions are available at each step of the ROM review workflow, allowing the responsible ROM actor(s) for the specific step, and depending on the step and for examples, to manage, delete, cancel/resume, approve/reject the ROM review, or revert the ROM review to a previous step (the revert can only be performed by ROM Administrators).

5. The ROM Experts section is dedicated to the management of the pool of Experts engaged under the ROM contracts. ROM Experts are encoded by ROM Contractors, and must be approved by ROM Coordinators in DG INTPA/NEAR before they can be selected during the Planning Phase of a ROM review.

6. As soon as the ROM review is approved by a ROM Coordinator, the Implementation and Follow-up Phases for the execution of the ROM review begins, consisting of five statuses/steps in the system (Approved, Desk Phase Started, Draft Report, Draft Report Commented and Final Report). Again, each status/step is managed by the ROM actor(s) responsible for the specific step in the workflow. The responsible ROM actor(s) will complete the required operations for each step (for examples: the ROM Expert fills in the draft ROM Report and drafts answers to the Monitoring Questions; the Operational Manager or M&E Focal Point comment on the draft ROM Report and draft Monitoring Questions), and then submits the ROM review to the next step in the workflow.

During the ROM Review Implementation and Follow-up Phases:

ROM Contractors can edit and update draft ROM reviews;
ROM Contractors will transition ROM reviews to the desk phase (once approved);
ROM Contractors and/or assigned ROM Expert(s) will fill in the Draft ROM Report and Draft Monitoring Questions for the ROM review based on the data collected during the field phase;
ROM Contractors and/or assigned ROM Expert(s) can address any comments received (if necessary) on the Draft ROM Reports.

7. Additional and optional actions are also available at each step of the ROM review workflow during the Implementation and Follow-up Phase of the ROM review, allowing the responsible ROM actor(s) for the specific step, and depending on the step and for examples, to manage the ROM review, or revert the ROM review to a previous step (the revert can only be performed by ROM Administrators).

8. Following the submission of the Final Report, the ROM review is set as “Completed” in the system.

Please find below a graphical representation of the status transitions for ROM reviews and Expert administration in OPSYS:

6.1.4 Roles and Responsibilities for ROM

INTPA D4 (Performance, Results and Evaluation; Internal Communication, Knowledge Management and Collaborative Methods) and NEAR A4 (Coordination of financing instruments - performance, results, and evaluation), are in charge of the overall coordination of the ROM system.

Four service contracts have been signed with external ROM Contractors for the implementation of ROM reviews and support to results reporting, covering each one lot (geographic or thematic).

Lot:	Geographical Coverage:
1	DG INTPA Global
2	DG INTPA Sub-Saharan Africa
3	DG NEAR ENI (European Neighbourhood Instrument)
4	DG NEAR IPA (Instrument for Pre-Accession Assistance)

In each EU Delegation and HQ Operational Unit, one staff member has been designated as M&E Focal Point. They are the main interlocutor for the ROM Coordinators and facilitate the implementation of ROM reviews within their Delegation or HQ Operational Unit.

Please find below the list of roles and responsibilities involved in ROM:

Role:	Responsibilities:
Operational Manager	The OM is the Project Manager (in EUD or HQ) responsible for the intervention(s) subject to a ROM review.
M&E Focal Point	Each EUD/HQ Unit appoints a Focal Point (and backup) for monitoring (including ROM) and evaluation. Regarding ROM, their role is to ensure appropriate information flows and good coordination among all actors involved in ROM mission planning and execution, as well as to encourage follow-up. The M&E Focal Point acts as backup of all Operational Managers of their Delegation or HQ Operational Unit and has the corresponding permissions for all ROM reviews in their Delegation or HQ Operational Unit.
Head of Cooperation/Head of Section/Head of Operations/Head of Unit	The Head of Unit in HQ/Head of Cooperation/Operations in Delegations has access to the ROM module of OPSYS for oversight of the monitoring activities in the portfolio under his/her responsibility.
ROM Coordinator	HQ staff (in DG INTPA and DG NEAR) in charge of a ROM contract. Approves experts and ROM reviews.
ROM Contractor	The ROM Contractors are the entities/consortia contracted and supervised by the ROM Coordinators to carry out the ROM reviews on selected Interventions. They report the results of their reviews according to standardised procedures and assure quality of their outputs.
ROM Experts – Core Team	External experts, responsible for reviewing Interventions as well as for coordination activities within the ROM team. The same as ROM Expert with more privileges.
ROM Experts – Non-core	External experts, responsible for reviewing Interventions.
ROM QC Experts	The purpose of the role is to check quality of deliverables within a given ROM contract.

Expand all Collapse all

Page tree