Introduction
The SDMX initiative sets standards to facilitate the exchange of statistical data and metadata using modern information technology. SDMX has also been published as an ISO International Standard (ISO 17369).
The operations in this API supports SMDX-3.0 artefacts and implements the REST API specification v2.0.
To make the most of this guide, a basic knowledge of XML and REST webservices is required.
The main elements are refered to as SDMX artefacts. Below are some terms that are used in SDMX and their definitions:
Dataset: a collection of related observations, organized according to a predefined structure
Data Structure Definition (DSD): metadata describing the structure and organization of a dataset, the statistical concepts and attached to them code lists used within the dataset
Dimensions: concepts that determine the dataset’s "physical" structure
Codelist: a code list is a predefined list from which some statistical coded concepts take their values. Each code list has the following properties:
identifier (it provides a unique identification within the set of code lists specified by a structural definitions maintenance agency);
name (also unique);
description (a description of the purpose of the code list); and
code value length (either an exact or a maximum number of characters and a type, i.e. numeric or alphanumeric).
Attributes: give additional information about the concepts used and do not affect the dataset structure itself
Dataflow: a structure which describes, categorizes and constrains the allowable content of a dataset that providers supply for different reference periods
Concept scheme: the descriptive information for an arrangement or division of concepts into groups based on characteristics, which the objects have in common. A concept scheme is a maintained list of concepts that are used in key family and metadata structure definitions (Definitions from EUROSTAT SDMX info space and OECD Glossary of statistical terms)
For in depth details, check as well the learning section of SDMX.org or the formal definition of the SDMX information model.
About versioned artefacts
While all SDMX artefacts could be versioned, currently only the following structural artefacts are versioned: Code lists (CL), Concept Schemes (CS), Data structure definitions (DSD).
It means that such artefacts are final and identified by a version number and safe to be copied/cached for further reference.
Other artefacts : Dataflow, ContentConstraint will always have the default version '1.0' and need to be requested again for updates.
Retrieving Structural metadata artefacts
Starting from the online data code of a dataset of choice, it is possible to query the API for detailed metadata on this data
Call in this guide, explicitly request not compressed response but it is recommended to use compression in automated or recurrent calls.
Looking up in the metadata of a dataset
in the SDMX Dataflow
Taking the dataset ISOC_CI_ID_H as example, its main information are available in its Dataflow SDMX artefact
This resource is not versioned, so 1.0 or + (meaning latest) can be used interchangeably
The minimal response would always contains the dataset label and the reference to the versioned DSD currently used by the dataset
<s:Dataflow id="ISOC_CI_ID_H" urn="urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ESTAT:ISOC_CI_ID_H(1.0)" agencyID="ESTAT" version="1.0"> <c:Name xml:lang="de">Haushalte - Verfügbarkeit von Internet-Geräten</c:Name> <c:Name xml:lang="en">Households - devices to access the internet</c:Name> <c:Name xml:lang="fr">Ménages - dispositifs pour accéder à l'internet</c:Name> <s:Structure>urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=ESTAT:ISOC_CI_ID_H(28.0)</s:Structure> </s:Dataflow>
Additionally a set of annotations (omitted in previous example, please expand full XML below) would provide additional information
| Annotation type | Description | Value(s) (in AnnotationTitle or multi-lingual AnnotationText) |
|---|---|---|
| OBS_COUNT | Number of statisticals observations in the dataset | 95814 |
| OBS_PERIOD_OVERALL_OLDEST | Oldest TIME position reported in an observation | 2002 |
| OBS_PERIOD_OVERALL_LATEST | Latest TIME position reported in an observation | 2014 |
| UPDATE_STRUCTURE | Timestamp when the dataset structure last changed
| 2021-02-08T23:00:00+0100 |
| UPDATE_DATA | Timestamp when the dataset data last changed | 2023-05-10T11:00:00+0200 |
| ESMS_HTML | Link to Reference Metadata page | https://ec.europa.eu/eurostat/cache/metadata/en/isoc_i_esms.htm |
| ESMS_SDMX | Link to Reference Metadata archive | https://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing?file=metadata/isoc_i_esms.sdmx.zip |
| SOURCE_INSTITUTIONS | Source institution | Eurostat |
in the referenced data structure definition (DSD)
From the reference present in the dataflow, it is possible to query for the corresponding SDMX Data Structure Definition
These resources are versioned, so version present in the reference must be used to ensure consistency.
This definition is informing about the list of dimensions used in the definition of the time-series of the dataset.
The order of dimensions will help build key filtering in the Data query later
For each dimension a reference is provided
to the to the concept holding the dimension label
to the code lists holding the code and labels for the dimension positions
These code lists are reference metadata and may contains more code and labels that the one used by a specific dataset.
To known the list of positions present in the dataset, please refer to the Content Constraint artefact (next section).
Additionally the DSD defines
- the mandatory TIME_PERIOD time-dimension where the value are expressed using ISO8601
- the primary measure OBS_VALUE holding the statistical value observation the
the optional value attribute OBS_FLAG hodling the statistical status (also refered as flags)
in the SDMX Data Constraint
| Content Constraint Link | https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/dataconstraint/ESTAT/ISOC_CI_ID_H/1.0?compress=false |
|---|
In the API, SDMX Data Constraints artefacts are used to define available positions, it lists for each dimension code the list of position code that are used at least once to refer to a statistical observation.
From example below we can see that ISOC_CI_ID_H dataset
- provides annual data ( freq = A)
- provides data for 14 indicators
- provides data in 2 units
- provides a breakdown on hhtyp ("Type of Household")
- provides data for EU aggregates and countries
- provides from 2002 to 2010 and 2014
retrieve several artefacts in a single response
It is not necessary to do these calls one by one.
Starting back from the Dataflow it is possible to include the referenced artefacts, at two different level
| Scope | Link |
|---|---|
| Dataflow + DSD | https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/dataflow/ESTAT/ISOC_CI_ID_H/1.0?references=children&compress=false |
| Dataflow + DSD + CS and CL | https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/dataflow/ESTAT/ISOC_CI_ID_H/1.0?references=descendants&compress=false |
Dataflow + DSD + CS and CL filtered on the constraints | https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/dataflow/ESTAT/ISOC_CI_ID_H/1.0?references=descendants&detail=referencepartial&compress=false |
Also from the DSD, it is possible to include the referenced artefacts :
https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/datastructure/ESTAT/ISOC_CI_ID_H/28.0?compress=false&references=children
Special case of Dataset listing
Instead of specifying a dataset code in the dataflow request the ALL keyword can be used to retrieve a list of all Eurostat datasets in one request
It is recommended to retrieve the compressed version
| ALL DATAFLOWS link | https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/dataflow/ESTAT/* |
|---|
Special case of Metadata harvesting
Similarly to the request on ALL dataflows it is possible to get the latest version for all artefacts for a specified type
It is recommended to retrieve the compressed version
| Scope | Link |
|---|---|
| All code lists | https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/codelist/ESTAT/* |
| All concept schemes | https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/conceptscheme/ESTAT/* |
| All data structure definitions | https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/datastructure/ESTAT/* |
Data query
in SDMX 3.0, the data query requires the dataflow complete identifier : type/agency/ID/version
Above link retrieves the complete dataset in default format : SDMX-ML 3.0 Structure Specific Data. (please remind taht Generic Data is now removed)
The data file is compose of time-series identified by a series-keys containing Observation as the one show below
<Series freq="A" geo="EA" hhtyp="TOTAL" indic_is="H_IPC" unit="PC_HH_IACC"> <Obs TIME_PERIOD="2003" OBS_VALUE="96.27"/> <Obs TIME_PERIOD="2004" OBS_VALUE="96.02"/> <Obs TIME_PERIOD="2005" OBS_VALUE="96.27"/> <Obs TIME_PERIOD="2006" OBS_VALUE="96.23"/> <Obs TIME_PERIOD="2007" OBS_VALUE="96.59"/> <Obs TIME_PERIOD="2008" OBS_VALUE="84.89"/> <Obs TIME_PERIOD="2009" OBS_VALUE="96.81"/> <Obs TIME_PERIOD="2010" OBS_VALUE="97.16"/> <Obs TIME_PERIOD="2014" OBS_VALUE="95.63"/> </Series>
It is possible to further customize the query to retrieve only the needed data or to request a different output format :
Filtering data
In SDMX 3.0 REST API, there are tow kind of filtering mechanism
- Filtering on series-keys : it is possible to define for each dimension one value or a wildcard to scope the data extraction ; this filter must follow dimensions order
- Filter by component value : it is possible to define for each dimension a list of value to filter on, and TIME_PERIOD dimension also allow range ; this filter is not impacted by dimensions order
These two filtering mechanisms can be combined with the logical limitation that a dimension first specified on the series-keys filter should not be repeated in the component value filter.
While all filtering could be done on component value filter, using series-keys filter can ease readability of the data slices targeted by a data query.
Filtering on series-keys
First level of filtering is about selecting how the dataset could be sliced and is done by filtering on the series-keys following the dimension order as specified in the DSD
In the example of ISOC_CI_ID_H, the series-keys template is as follow
FREQ.INDIC_IS.UNIT.HHTYP.GEO
with the following syntax:
- '*' is a wildacard means no filtering for this dimension
- A value can be set for a dimension
- Partial key is supported and assume wildcarding for remaining dimension, see example below
Scope | Details on the series-keys filter | Link | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Single time-series fully specified |
| https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/data/dataflow/ESTAT/ISOC_CI_ID_H/1.0/A.H_IPC.PC_HH_IACC.TOTAL.EA?compress=false | ||||||||||||
Single indicator + Single Unit All Type and countries |
|
This can prove useful on FREQ dimension, in dataset holding data for multiple frequency , to ensure to retrieve a single type of data (annual, quarterly, monthly...) per request. | ||||||||||||
Single GEO data | As the GEO dimension is the last, previous dimension must be wildcarded *.*.*.*.EU27_2020 |
Using partial key support means that the remaining dimension .*.* can be omitted from the URL so it would be equivalent to Filtering on component value
Component value filtering has the following syntax c[dim_id]=value:
- dim_id is dimension id as found in the DSD
- value can list several positions separated by a comma
- both dim_id and value are case insensitive
| Scope | Description | Link |
|---|---|---|
Single time-series fully specified | Alternative query equivalent to series-keys filtering above | |
Single indicator + Single Unit All Type and countries | Alternative query equivalent to series-keys filtering above | |
Single GEO data | Alternative query equivalent to series-keys filtering above Not needing to know of the dimension list and order | |
All indicators and unit TOTAL type Only EU+EA | The GEO list of value can only be expressed as component value filtering Single position HHTYP = TOTAL can be expressed either way | It could be expressed alternatively In combination with series-keys filter for hhtyp dimension |
Filtering on time period
Filtering the observations to be returned based on their TIME_PERIOD value is also controller via component value filtering as other dimension but additionally support value range definition
Following SDMX syntax
Reusing above single time-series example, it can be restricted on TIME_PERIOD
| Scope | Description | Link |
|---|---|---|
| Single time value | Get value for 2010 only | or to align with other examples below by specifying the equals operator eq |
| Rolling time | Get value after 2010 | Inclusive operator : "greater than or equals" : ge |
| Until time | Get value before 2010 | Inclusive operator : "lower than or equals" : le |
| From - to | By combining previous operator with a '+' it is possible to specify a time range |
Supported format
An additional format parameter allows to request a response in a different semantic format
Compression
An additional compress parameter allows to disable compression. It is recommended to use it only if necessary.
| Uncompressed TSV data link | https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/data/dataflow/ESTAT/ISOC_CI_ID_H/1.0?format=tsv&compress=false |
|---|
Retrieving Navigation artefacts
It is worth to mention that secondary artefacts exists to represent as SDMX artefacts a classification of dataset in categories (also refered as Navigation Tree in Eurostat)
- Category Scheme : Hierarchy of categories
- Categorisation : one categorisation is referencing one dataset into a category of a Category Scheme
| Scope | Link |
|---|---|
| All category schemes | https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/categoryscheme/ESTAT/* |
| All categorisations | https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/categorisation/ESTAT/* |