ROM Reviews

6.3 ROM Reviews

6.3.1 Find and Access a ROM Review

ROM reviews can be found (once they have been created) in the ROM Reviews section in the external Funding & Tender Opportunities Portal:

• ROM Contractors have access to all ROM reviews in all statuses;
• ROM Core Team Experts have access to all ROM reviews in all statuses for their respective lot;
• ROM Experts have access to all ROM reviews for which they have been selected, and once the ROM review is approved.

The ROM Reviews section in the Funding and Tenders Portal displays all ROM reviews in all statuses (except deleted) of the ROM review preparation, implementation and follow-up phases of the ROM Review Workflow (Draft, Confirmed, Experts Selected, Experts Confirmed, Approved, Desk Phase Started, Draft Report, Draft Report Commented, Final Report, Completed or Cancelled).

How to Find and Access a ROM Review

To find and access a ROM review in the Funding and Tenders Portal, the ROM Contractor or ROM Expert will first access the Monitoring & Evaluation (1) component > ROM > ROM Reviews (2) section.

Use the Advanced Filter (3) to enter and select filter options to refine your search (by entering or selecting Intervention and/or ROM review search criteria), and then click on the “Apply” (4) button to apply the filter options. The list of ROM reviews based on the applied filter criteria are displayed in the Results (5).

Once you have found the ROM review, click on the ROM Review ID link (6) to access the page of the corresponding ROM review.

6.3.2 Manage and Submit a ROM Review

6.3.2.1 Manage a ROM Review

A ROM Contractor can manage a ROM review for which they are responsible for during the Draft, Confirmed, Approved, Desk Phase Started or Draft Report Commented statuses/steps of the ROM Review Workflow.

How to Manage a ROM Review

To manage a ROM review in the Funding and Tenders Portal, the ROM Contractor must first find and access the ROM review.

On the ROM Review page, click on the “Actions” button at the top right of the page, and then click on the “Manage” option to enable the edit mode.

In edit mode, apply the required changes to the ROM review, such as preferred times, target countries or language, and then click on the “Save” button at the bottom right of the page to save your modifications.

 If the ROM review was successfully saved, a green confirmation message will be displayed at the bottom right of the page.

Alternatively, click on the “Cancel” button before saving to cancel any modifications.

6.3.2.2 Submit a ROM Review

Once the ROM Contractor completes a status/step of the ROM review for which they are responsible for (i.e. for Confirmed, Approved, Desk Phase Started or Draft Report Commented, except Draft (which is submitted by the Operational Manager or M&E Focal Point)), the ROM Contractor must submit the ROM review to the next step of the ROM Review Workflow within the submission deadline.

Submission deadlines, as defined in the ROM Handbook, are set in the system. They indicate the maximum number of days for the corresponding status/step indicated, and are as follows:

Submission deadlines are global settings, so they are applicable to all ROM reviews.

How to Submit a ROM Review

To submit a ROM review in the Funding and Tenders Portal, the ROM Contractor must first find and access the ROM review before submission to the next step (see table above for ROM Contractor responsible steps).

On the ROM Review page, click on the “Actions” button at the top right of the page, and then click on the “Submit” option to submit the ROM review.

Any submission errors detected for this step of the ROM review workflow will be displayed before submission and will need to be corrected before proceeding with the submission.

If no submission errors are detected and in the Confirmation pop-up window (“Confirmed” to “Experts Selected” status for the below example), click on the “Continue” button to submit the ROM review.

If the ROM review was successfully submitted, a green confirmation message will be displayed at the bottom right of the page, and:

• You will be redirected back to the ROM Review page;
• The ROM review status is updated to the next step of the workflow.

Alternatively, click on the “Cancel” button before submission to cancel the operation.

6.3.3 ROM Review Planning Phase

6.3.3.1 Draft ROM Review

A ROM review is in “Draft” status once it has been created by an Operational Manager or M&E Focal Point for Interventions in their Unit or Delegation, or by a ROM Coordinators for Interventions in their DG.

Once the ROM reviews are created, the draft ROM reviews are shared with ROM Contractors via the ROM Reviews section of the Funding and Tenders Portal.

ROM Contractors and ROM Core Team Experts (only for their respective lot) can access draft ROM reviews.

6.3.3.1.1 Edit and Update a draft ROM Review

The Operational Manager, M&E Focal Point, and/or ROM Contractor are expected to fill in all the mandatory fields within an indicative timeframe of 14 calendar days.

When the draft ROM review is prepared and all the required fields completed, it is submitted and confirmed by the Operational Manager or M&E Focal Point.

To edit and update a draft ROM review in the Funding and Tenders Portal, the ROM Contractor must first find and access the ROM review.

Follow the same steps as in manage a ROM review above.

6.3.3.1.2 Delete a draft ROM Review

Only the Operational Manager, M&E Focal Point or ROM Coordinator can delete a ROM review, and only if the ROM review is in “Draft” status.

ROM reviews in other status during the planning phase can still be cancelled (if needed).

6.3.3.2 Confirm ROM Review

The ROM review transitions to “Confirmed” status as soon as the Operational Manager or M&E Focal Point submits and confirms the draft ROM review. The Expert Selection process by the ROM Contractor can begin when the ROM review is confirmed.

Out of the list of ROM reviews for the work plan, ROM Contractors can view and extract a list of planned ROM reviews (draft, confirmed, experts selected, experts confirmed) and estimate the projects that they are able to review with the budget available under their contract. ROM Contractors can cancel or resume a ROM review from the confirmed ROM review based on their planning (work plan). ROM Core Team Experts can also view confirmed ROM reviews for their respective lot.

Operational Managers, M&E Focal Points and ROM Coordinators can also cancel or resume ROM reviews from confirmed ROM reviews. In case of doubts about whether a ROM review should be cancelled or resumed, please contact your M&E Focal Point or ROM Coordinator.

6.3.3.3 Expert Selection for ROM Review

6.3.3.3.1 Manage Experts

The matching of pre-approved Expert(s) with specific ROM reviews is a responsibility of the ROM Contractor, and is performed directly from the ROM Review page via the Funding and Tenders Portal.

For details on how to manage ROM Experts, please view ROM Experts.

6.3.3.3.2 Expert Selection by Contractor

ROM Contractors can now plan the specific missions, assigning pre-approved Experts and proposing mission dates. They encode this information for each ROM review via the Funding and Tenders Portal. The indicative timeframe allocated to complete this step is 14 working days. ROM Contractors also select the Quality Control (QC) Expert in charge of supporting the mission as soon as possible, and in any case before the submission of the Draft Report. The ROM Contractor can also signal any conflict of interest at Consortium-level (if any).

How to Select Experts – ROM Contractor

To select the experts and propose mission dates in the Funding and Tenders Portal, the ROM Contractor must first find and access the ROM review.

On the ROM Review page, click on the “Actions” button at the top right of the page, and then click on the “Manage” option to enable the edit mode.

In edit mode, complete the following fields on the ROM Review page:

(*) required fields

Dates

• Planned date for Draft Report* – Enter the planned date for the submission of the Draft Report.

Selection of Experts and Conflict of Interest

• For each ROM Expert* (core or non-core) – Select the expert and enter the Field Phase Start Date* and Field Phase End Date*.
• QC Expert – If already known, select the Quality Control Expert for the ROM review.

For the selection of experts, click on the corresponding search button to activate the Search ROM Experts function. In the Search ROM Experts pop-up window, use the search criteria fields (and “Search” button) to filter the experts displayed in the below list of experts.

Click on the arrow icon to select the required expert from the list of experts (the selection of multiple experts possible – if needed). Deselect an expert (if required) from the list of experts by clicking on the bin icon.

Click on the “Save” button to confirm your selection of experts and return to the ROM Review page.

Alternatively, click on the “Cancel” button to cancel the operation and return to the ROM Review page.

How to signal a Conflict of Interest at Consortium-level (if any)

To signal a possible conflict of interest at Consortium-level, select and complete the following options/fields:

• Consortium Conflict of Interest – Select the option if there is a conflict of interest at consortium-level. If selected, enter the required Comment/Consortium conflict of interest* in the text box (max. 2000 characters).
• Solve Conflict of Interest at Consortium-level – Select the option if you have a proposition to solve the conflict of interest at consortium-level. If selected, enter the required Comment/Solve conflict of interest at consortium-level* in the text box (max. 2000 characters).

• Comment – Add any additional comments in the text box (if necessary).

Click on the “Save” button at the bottom right of the page to confirm the expert selection for the ROM review, or on the “Cancel” button to cancel the operation.

If the ROM review was successfully saved, a green confirmation message will be displayed at the bottom right of the page, and:

• The information added will be updated to the ROM review;
• You will be redirected back to the ROM Review

Alternatively, click on the “Cancel” button before saving to cancel the modification.

Once you have selected the experts for the ROM review, and all the information entered for this step and in the ROM review is correct, submit the ROM review to the next step.

6.3.3.3.3 Confirmation of Expert Selection by OM

ROM Contractors propose a suitable and available Expert for each Intervention subject to a ROM review. Should there be any concerns, e.g. for Interventions that require a rare expertise, the Operational Managers and M&E Focal Points can alert the ROM Contractor via e-mail or contact the ROM Coordinators, who will inform the Contractor accordingly.

Operational Managers and M&E Focal Points do not approve or reject the CV of the proposed Expert(s) based on the quality of the CV. They provide an opinion on any potential conflict of interest of the proposed Expert(s). The objectivity and quality of ROM Experts' judgments is crucial for the credibility of the system. Therefore, ROM Experts must not have been involved in the preparation or management of any part of the Intervention monitored. In case of doubts, the ROM Coordination Unit can share the Experts' CVs with the Unit or Delegation concerned.

ROM Contractors replace the ROM Experts in cases of conflict of interest. If there is no indication of conflict of interest within 14 working days from the submission of Expert Selection, the ROM Experts can be confirmed in the ROM module in OPSYS (tacit consent procedure), and the ROM Coordinator can approve the mission.

Once the Experts are confirmed, the next step is the approval of the mission by the ROM Coordinator.

6.3.3.4 Approve or Reject ROM Review

Once all previous steps are completed, the ROM Coordinator can approve (or reject) the ROM review, within an indicative timeframe of 14 working days following the confirmation of experts. While deciding whether to approve a mission or not, the ROM Coordination Unit will consider all elements, and in particular the association between Expert and the sector of the Intervention, any issues raised by the ROM Contractor or Operational Manager (including security matters), and any deviation from the ROM standards.

Only ROM Coordinators can approve or reject ROM reviews, and are able do so from the “Expert Selected” or “Experts Confirmed” status/step of the ROM review. ROM reviews can be approved in “Experts Selected” status/step if necessary, and when the tacit procedure for Expert confirmation is applied.

Once the mission is approved, the ROM review’s status will be updated to “Approved”, and the project documents can be shared with the ROM Experts (by the Operational Manager or M&E Focal Point), which may include:

• Country Programming documents
• Action Documents
• QRG assessments
• Financing/Contribution/Delegation/Administration Agreement or Grant Contract (or in some cases Service Contracts). In these documents, all annexes are to be included such as Technical and Administrative provisions, Implementation schedule, Logical Framework (updates included)
• Project Contracts/Programme Estimates
• In case of projects selected through Calls for Proposals: Guidelines for Calls for Proposals
• Amendments to all the documents listed above and their explanatory notes
• Annual and overall activity schedules/implementation plans
• Implementation progress reports
• Previous ROM Reports
• Mid-term evaluations
• Final evaluations of previous phases (if any)
• Thematic studies and consultancy reports related to the project

If the ROM review is rejected, the status of the ROM review is set to “Cancelled” with cancellation reason “Rejected”.

6.3.3.5 Revert ROM Review

Only ROM Administrators can revert a ROM review to any previous status/step in the ROM Review Workflow, except from “Draft” or “Completed” status, in both the planning and implementation and follow-up phases, or from the implementation and follow-up phases back to the planning phase (i.e. revert from “Approved” back to “Experts Confirmed” status).

In each step of the ROM review workflow, a ROM Administrator can revert the ROM review to a previous status/step by using the “Revert” operation.

The “Revert” operation is not limited to sending the ROM review back to the previous step, but can be sent back to any of the previously completed steps. For example, a ROM Administrator can send a ROM review from an “Expert Confirmed” status, all the way back to the “Draft” status, etc.

If the ROM review is reverted, the ROM review will be reverted to the status/step indicated during the revert action.

6.3.3.6 Cancel or Resume ROM Review

A ROM review can be cancelled during the “Confirmed”, “Experts Selected” or “Experts Confirmed” status/step during the planning phase of the ROM Review Workflow.

In each of these steps, the user that is responsible for submitting the ROM review to the next status/step, can also cancel the ROM review by using the “Cancel” operation.

ROM Contractors can therefore cancel a ROM review only from the "Confirmed" status/step.

The opposite of cancelling a ROM review is resuming a ROM review. ROM reviews can be resumed if the situation changes. If resumed, the ROM review will return to the status it had before the cancellation.

How to cancel a ROM Review

To cancel a ROM review in the Funding and Tenders Portal, the ROM Contractor must first find and access the ROM review (if not already done).

On the ROM Review page, click on the “Actions” button at the top right of the page, and then click on the “Cancel” option.

In the ROM Review Cancellation pop-up window, select the required cancellation reason (not enough budget, security issue, natural disaster, covid-19, health risk, or other*).

If needed, further describe the reason for the cancellation in the text box, which is required for the (*) reasons selected previously, and click on the “Continue” button.

In the following Confirmation pop-up window, click on the “Yes” button to confirm the cancellation of the ROM review.

If the ROM review was successfully cancelled, a green confirmation message will be displayed at the bottom right of the page, and:

• You will be redirected back to the ROM Review page;
• The ROM review status is set to “Cancelled”.

 Alternatively, click on the “No” button before cancellation to cancel the operation.

How to resume a ROM Review

Only an internal ROM user responsible for the ROM review can resume a cancelled ROM review.

To resume a ROM review (for example, if a situation changes), the ROM Contractor must contact the M&E Focal Point or Operational Manager to request the resume.

If resumed, the ROM review status is resumed to the status/step it had before cancellation.

6.3.4 ROM Review Implementation and Follow-up Phases

ROM review missions are organised around three main phases:

• The Desk Phase, involving document and logframe analysis;
• The Field Phase, in which the main stakeholders are consulted, and briefing and debriefing sessions are conducted with the Operational Manager in charge; and
• The Reporting Phase, in which the quantitative and qualitative analysis of the evidence gathered, with the guidance provided by a set of Monitoring Questions (MQs), converges into a ROM Report, which is the key deliverable for each ROM review.

Each ROM review takes approximately three months (from the approval of the ROM review), including the internal Quality Control (QC) and comments by the Commission services on the draft and final versions of the ROM Report.

6.3.4.1 ROM Review Desk Phase

Once the ROM review is approved and the ROM mission has started, the ROM Contractor will transition the ROM review from "Approved" to the “Desk Phase Started” status/step (within the indicative timeframe: 14 calendar days).

Following the start of the Desk Phase (and after the field phase has taken place), the ROM Contractor and/or assigned ROM Expert(s) will access the ROM review, fill in the Draft ROM Report and Draft Monitoring Questions based on the data collected during the field phase, and submit the ROM review to status “Draft Report”. The Draft ROM Report and Draft Monitoring Questions must be filled in no later than 14 calendar days after the end of the field phase.

During the Desk Phase (and if necessary), the ROM Contractor, Operational Manager and M&E Focal Point can manage (update) the ROM review.

How to start the Desk Phase (ROM Contractor)

To start the Desk Phase in the Funding and Tenders Portal, the ROM Contractor must first find and access the ROM review.

On the ROM Review page, submit the ROM review to start the Desk Phase.

 How to fill-in the Draft ROM Report and Draft Monitoring Questions (ROM Contractor or ROM Expert)

To fill-in the Draft ROM Report and Draft Monitoring Questions in the Funding and Tenders Portal, the ROM Contractor or assigned ROM Expert must first find and access the ROM review (if not already the case).

On the ROM Review page, click on the “Actions” button at the top right of the page, and then select either:

• If you want to complete the Draft ROM Report or ROM Monitoring Questions via the online form (1), choose respectively:

- the “Fill Draft ROM Report” option and complete the online form; and

- the “Fill ROM Monitoring Questions” option and complete the online form.

• Or, if you want to complete the Draft ROM Report or ROM Monitoring Questions via the template (editable PDF file) (2), choose respectively:

- the “Download ROM Report Template” option to download the PDF file to your computer and complete the file; and

- the “Download Monitoring Questions Template”  option to download the PDF file to your computer and complete the file.

Via the online form

Complete the following fields on the corresponding page and sections.

Draft ROM Report page

Click on the section title to expand/collapse the corresponding section and fields to complete (draft values).

(*) required fields

Executive Summary section:

• Executive Summary*

Project Synopsis section

• Description of the context with a focus on the problematic to be addressed by the action*
• Description of the intervention logic*
• Description of the target group(s) and financial beneficiaries*

Findings section:

• Relevance*
  
• Coordination, complementarity, and EU added value*
• Intervention logic, Monitoring & Learning*
• Efficiency*
• Effectiveness*
• Sustainability*
• Cross-cutting issues*
• Communication and visibility*

Conclusions section:

• Conclusions* – Enter your conclusions. If needed, click on the plus button to add additional conclusions, and/or the delete button to remove conclusions.
• Recommendations – Enter the recommendations:
  • Recommendation*
  • Priority*
  • Importance*
  • To Whom*

If needed, click on the plus button to add additional recommendations, and/or the delete button to remove recommendations.

 While completing the Draft ROM Report form and at the bottom right of the page, you can either click on the “Save” button to save your changes, or click on the “Complete” button once you have completed the form.

The confirmation message is displayed when the Draft ROM Report is successfully saved or completed.

Once the Draft ROM Report form has been successfully filled-in, it will also be indicated as completed under the “Actions” of the ROM review.

ROM Monitoring Questions page

Reply to the questions on the ROM Monitoring Questions page. It is mandatory to reply to all questions marked with an asterisk (*).

1. Depending on the geographical implementation and lot for the ROM review, you may need to select the ROM Monitoring Questions template (Global or Country). If "Country", select the country from the additional drop-down list;

2. Click on the section header to expand the questions underneath;

3. Reply to the questions in the corresponding textboxes;

4. Choose the appropriate traffic light: green = good/very good, orange = issues to be addressed, red = off-track/serious deficiencies;

5. At any moment, you can cancel or save the questionnaire, or complete the questionnaire once all the required questions are completed.

Questions related to the List of Documents Analysed section (section 9) are in the form of “Yes”/”No”/”N/A” answers, and need to be replied by selecting the appropriate choice from the corresponding drop-down list.

To add additional analysed document(s) to the list, click on the "Add Document" button at the end of the list and enter the name (description) of the document(s) analysed.

Add the list of person(s) interviewed for the ROM Monitoring Questions under the List of persons interviewed section.

Click on the “Add a new person” button to add additional person(s) to the list. Manage the list (if required) by clicking on delete button to remove person(s) from the list.

Any validation errors detected for the completion of the form will be displayed before completion, and will need to be corrected before proceeding with the completion.

If the ROM Monitoring Questions were successfully filled in, a green confirmation message will be displayed at the bottom right of the page, and:

• You will be redirected back to the ROM Review page;
• Both the ROM Review forms (Draft ROM Report/ROM Monitoring Questions) are indicated as “Completed”.

Alternatively, click on the “Cancel” button before saving or completion to cancel the operation.

Via the PDF template

 In each of the downloaded Draft ROM Report or Monitoring Questions PDF templates:

1. Open the downloaded PDF file template;

2. Complete the required fields and questions in each template file (using the same procedure as described above for the online form);

3. Once completed, save each template file to your computer;

4. Once the template files are saved, return to the ROM Review page in the system, click on the “Actions” button at the top right of the page and choose respectively:

- the “Upload ROM Report Template” option to upload the completed PDF file template from your computer to the system; and

- the “Upload Monitoring Questions Template”  option to upload the completed PDF file template from your computer to the system.

Any validation errors detected for the completion of the templates will be displayed, and will need to be corrected in the respective template and uploaded before proceeding.

If the Draft ROM Report or ROM Monitoring Questions templates were successfully filled in, respectively, a green confirmation message will be displayed at the bottom right of the page, and:

• You will be redirected back to the ROM Review page;
• Both the ROM Review forms (Draft ROM Report/ROM Monitoring Questions) are indicated as “Completed”.

How to download the Draft Reports

The Draft ROM Report/Draft Monitoring Questions/Draft Consolidated ROM Report can be downloaded (PDF version) by all ROM actors (internal or external) from the ROM review once the forms are completed.

To download the draft reports, click on the “Actions” button of the ROM review and select the appropriate option.

The system will automatically generate the Draft Report PDF file from the information entered from the corresponding form(s), and you will be able to download and view the file on your computer.

The Draft Consolidated ROM Report is a consolidated PDF file version of the Draft ROM Report and scores from the Draft Monitoring Questions.

Once the ROM Contractor or ROM Expert(s) have filled-in the Draft ROM Report and Draft Monitoring Questions for the ROM review, and all the information entered for this step and in the ROM review is correct, the ROM review must be submitted the ROM review to the next status/step in the workflow: “Draft Report”.

6.3.4.2 ROM Review Draft Report

6.3.4.2.1 Comment Draft Report

Following the submission of the ROM review from the “Desk Phase Started” to “Draft Report” status/step, the Operational Manager and M&E Focal Points have 21 calendar days to download/view/comment on the Draft ROM Report submitted by the ROM Contractor, and submit the ROM review to status “Draft Report Commented”.

During the “Draft Report” status/step (and if necessary), the ROM Contractor, Operational Manager and M&E Focal Point can manage (update) the ROM review.

Once the Draft ROM Report has been commented and graded, the ROM review is submitted by the Operational Manager or M&E Focal Point to the next “Draft Report Commented” status/step.

6.3.4.2.2 Update Draft Report

Once the ROM review is in “Draft Report Commented” status/step, the ROM Contractor and selected ROM Expert(s) have 14 calendar days to address the comments received from the Operational Manager/M&E Focal Point on the Draft ROM Report, make changes (if necessary), and submit the ROM review to status “Final Report”.

6.3.4.3 ROM Review Final Report

In this step, the Operational Manager or M&E Focal Point will comment on the Final ROM Report.

The Final ROM Report can be downloaded from the corresponding ROM review.

6.3.4.4 ROM Review Completed

In this step, the Operational Manager or M&E Focal Point will submit the ROM review with the Final ROM Report, and the status of the ROM review will be updated to “Completed”.

The Final ROM Report can be downloaded from the corresponding ROM review.