Methodology

Background

We synthetise large amount of data to assess whether farming practices have positive or negative effects on the environment, climate and productivity. Our starting point is the relatively large number of meta-analyses (MAs) in agricultural science. A meta-analysis is the systematic statistical synthesis of the results of many independent individual experiments. Therefore, MAs allow to explore general trends beyond the context-dependence of large numbers of experimental studies and identify key moderating factors. MA has become a gold standard method for quantitative research synthesis, and scientists can use the growing number of MAs available to inform decisions of policy makers.

Method

We have developed a methodological framework for assessing the impacts of farming practices on the environment and climate and crop productivity based on a Systematic review of published meta-analyses. The framework helps to report the results and quality of MAs in a rigorous and transparent manner. In addition, the framework can be implemented quickly - within weeks - to be operational and compatible with the time constraints of modern policymaking processes.

The framework includes four main steps: (1) systematic literature search of existing MAs, (2) screening and selection of MAs, (3) data extraction and quality assessment, and (4) reporting.

Quality criteria

We assess each MA using 16 quality criteria covering different aspects of the MAs

• Scoping (definition of the objective of the MA)
• Search (search of potentially relevant individual published studies)
• Study selection (selection of the relevant individual studies)
• Data extraction (extraction of the experimental data useful for computing the effect sizes)
• Statistical analysis (analysis of the data to estimate the mean effect size and analyse between-study variability)
• Bias and uncertainty (analysis of the uncertainty of the results and analysis of the risk of bias).

Results

We assess the overall effect of each farming practice on a specific impact based on the scientific evidence found in the published MAs. As each MA involves a number of individual studies, the assessment of impacts relies on the statistical analysis of a large number of results obtained mainly in field experiments, and sometimes in lab experiments or from model simulations.

According to the statistical analysis provided by MAs, we categorize the results as ‘Positive’, ‘Negative’ or ‘No-effect’, where proper statistical analysis is available, testing the effect of a farming practice, as compared to a control. When statistical analysis is not available we rate results as ‘Uncertain’.

Three types of reports are generated from the extracted data:

• General Fiches: reports summarising the results of all MAs exploring environmental, climatic or crop production impacts of each farming practice analysed.
• Single-Impact fiches: reports summarising each of the impacts of a given farming practice on a specific environmental, climate, or production outcome.
• Summaries of the meta-analyses: reports summarising the details (i.e.; objectives, methods, and main results) of each MAs for each impact.

Advantages

Since 2020, the framework is successfully applied by a group of scientific experts to support decisions of EU policy makers, and examine the large diversity of single farming practices presented in this WIKI in a relatively short time. It provides an operational tool for scientists who want to supply policymakers with scientific evidence based on large numbers of experiments, in a timely and reproducible manner. In addition, the reported results provide a wealth of information to those interested in the environmental and climate impacts of agricultural practices that Member States can apply through their Common Agricultural Policy Strategic Plans.

Reference: DOI: 10.31219/osf.io/byuw9