Eurostat Online Help for Data Browser
Child pages
  • API - Migrating from Bulk Download Listing urls to API urls

Overview

Since October 2023, a new URL for files retrieval from Eurostat API is available (https://ec.europa.eu/eurostat/api/dissemination/files) and phasing out of the Bulk Download Listing (https://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing) is planned in the coming weeks.

Currently both URL endpoints remains available and distribute the same content without change.

At the end of the transition phase :

  • the existing Bulk Download Listing URL endpoint will not be accessible anymore
  • (warning) some files will not be produced anymore 
    • /data folder will not exist (TSV and SDMX-2.0)
    • /dic/{en,fr,de} will not exist (codelists files)
  • the new URL (https://ec.europa.eu/eurostat/api/dissemination/files) will become the main one and focused on automation usability machine to machine files retrieval 
    • serving content for
      • /comext
      • /metadata 
      • /codelist-mappings (new folder)
    • (warning) keeping basic HTML responses for directory listing to offer extended support to existing web-crawler clients, but expecting that client will migrate to the CSV responses 

Migration path is different for

  1. manual browsing and downloading by users that are invited to use the Downloads section of Databrowser
  2. automation solutions that should
    1. at minimum update the target URL for continued content (/comext and /metadata)
    2. change to request data and code list content from the API

How content is accessible from the Data Browser

The Downloads section of Databrowser is listing the content available via Eurostat Dissemination Chain and allow to download one element, a batch of selected elements (max 50) or retrieve an inventory with download links for use in scripts

Migrating from /data content to API

Listing datasets

Option 1 (SDMX) : Retrieve all dataflows "stubs"(only references and title) in XML https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/dataflow/ESTAT/all/1.0?detail=allstubs 

Option 2 (TXT) :  Retrieve the inventory listing of data https://ec.europa.eu/eurostat/api/dissemination/files/inventory?type=data

Downloading a dataset

Transitioning from Eurostat Bulk Download to API introduces some structural and data changes in the TSV and SDMX file formats.

They are presented in detail in the following documents:


Migrating from /dic code lists content to API

Listing code lists

Option 1 : Retrieve all code lists "stubs"(only references and title) in XML https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/codelist/ESTAT/all/latest?detail=allstubs

Option 2 : Retrieve the inventory listing of code lists https://ec.europa.eu/eurostat/api/dissemination/files/inventory?type=codelist

Downloading a code list

For a known code list code the download URL must be changed to 

https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/codelist/ESTAT/<CODELIST_CODE>/latest?format=TSV&lang=en

Examples :


Migrating from special files to API

/metabase.txt.gz

Option 1 : New URL https://ec.europa.eu/eurostat/api/dissemination/catalogue/metabase.txt.gz. Please consult API - Detailed guidelines - Catalogue API - Metabase for further details

Option 2 : Migrating_to_API_data_constraints.pdf explains how to get similar information per dataset from SDMX artefacts.

/dic/all_dic.zip

Option 1 : Retrieve all codelists in XML https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/codelist/ESTAT/all/latest

Option 2 : Retrieve the inventory listing of code lists https://ec.europa.eu/eurostat/api/dissemination/files/inventory?type=codelist and request the TSV content for each code list as explained above in Downloading a code list or in API - Examples - periodic data download

Code lists are versioned SDMX artefacts, in case the version mentioned in the inventory listing did not change, there is no need to re-download the content as it would be identical.

/dic/en/table_dic.dic

As mentioned in above section in for "Migrating from /data content to API"

Option 1 in XML : Retrieve all dataflows "stubs"(only references and title) in XML https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/dataflow/ESTAT/all/1.0?detail=allstubs 

Option 2 in TSV : Retrieve the inventory listing of data https://ec.europa.eu/eurostat/api/dissemination/files/inventory?type=data 

/dic/en/dimlst.dic

(info) As in Eurobase, one code list is defined for each dimension the "dimension list" is equivalent to the list of code lists.

Option 1 : Retrieve all code lists in XML https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/codelist/ESTAT/all/latest

Option 2 : Retrieve the inventory listing of codelists https://ec.europa.eu/eurostat/api/dissemination/files/inventory?type=codelist

/table_of_contents.xml

In case table of contents was mainly used to check for updates on a specific list of datasets, please consider looking up directly the dataflow for each dataset where the information is available in SDMX Annotations

Example: https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/dataflow/ESTAT/DEMO_GIND/latest

If necessary the listing for ALL dataflows can also be retrieved in SDMX-ML format https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/dataflow/ESTAT/all/latest 

Option 1 : Retrieve new XML table of contents from API catalogue https://webgate.acceptance.ec.europa.eu/eurostat/api/dissemination/catalogue/toc/xml. Please consult API - Getting started with catalogue API for further details

Option 2 : Retrieve backward compatible XML table of contents from API catalogue https://webgate.acceptance.ec.europa.eu/eurostat/api/dissemination/catalogue/toc/legacy_xml.

How files are accessible with the new API Files

File download

The file download keeps the same basis as previously with specifying the relative path to content as a query parameter named file.

https://ec.europa.eu/eurostat/api/dissemination/files?file=<relative-path-to-content>

In case a file does not exists HTTP response will be a 404 Not found response

Examples:

  • https://ec.europa.eu/eurostat/api/dissemination/files?file=comext/COMEXT_DATA/PRODUCTS/full202305.7z
  • https://ec.europa.eu/eurostat/api/dissemination/files?file=metadata/aact_esms.sdmx.zip

About other parameters

Other parameters are safely ignored so in case parameters from previous Bulk Download Listing such as sort 

changing the base URL to

would work.

Directory listing in CSV

Listing of a directory content in CSV

https://ec.europa.eu/eurostat/api/dissemination/files?format=csv&dir=

Additional options

  • sort
    • name (default)
    • date
    • size
  • order
    • asc (default)
    • desc
  • hierarchy
    • true
    • false (default)
  • dateFormat
    • ISO (default)
    • UNIX
    • READABLE
  • sizeFormat
    • NONE (default)
    • READABLE


When hierarchy=true, sort/order will be ignored

The CSV will have the following columns :

Column nameDescription
NAME

Directory or file name

when hierarchy=true, it is prefixed with the relative path to the dir requested

SIZE

Not specified for folders

Formatted according to sizeFormat parameter

NONE : raw number of bytes, easier for handling by clients (sorting

READABLE: formatted with unit (13.06 KB or 9.74 MB)


TYPE

DIR for directory

Extension from filename (TXT,DOC...)

DATE

Last modified date from filesystem

Formatted according to dateFormat parameter

ISO: ISO8601 format (2023-08-02T04:45:34+00:00)

UNIX:  number of seconds since EPOCH TIME (JAN 01 1970)

READABLE: Eurostat date time format ( 01/08/2023 22:50:05)

(info) A download URL can be computed with the file download base URL + the current folder listed + the content of the NAME column



Example for COMEXT_OTHER_DATA with hierarchy=false, sizeFormat=READABLE, dateFormat=READABLE

https://ec.europa.eu/eurostat/api/dissemination/files?format=csv&dir=comext&hierarchy=false&sizeFormat=READABLE&dateFormat=READABLE

NAME,SIZE,TYPE,DATE
EUROPROMS,,DIR,24/11/2017 12:09:15
NON_EU_DATASETS,,DIR,30/05/2022 11:06:16
TEC exercise for CA 2014.zip,122.52 KB,zip,18/07/2019 13:05:02
TEC exercise for VN and JP 2017.zip,1.54 MB,zip,16/07/2018 10:11:15

So the download URL for TEC exercise for CA 2014.zip would be https://ec.europa.eu/eurostat/api/dissemination/files?file=comext/TEC exercise for CA 2014.zip

Example for COMEXT_OTHER_DATA with hierarchy=true

https://ec.europa.eu/eurostat/api/dissemination/files?format=csv&dir=comext&hierarchy=true&sizeFormat=READABLE&dateFormat=READABLE

NAME,SIZE,TYPE,DATE
EUROPROMS,,DIR,24/11/2017 12:09:15
EUROPROMS/Data availability Comext 01#2017.doc,52.00 KB,doc,24/11/2017 12:09:13
EUROPROMS/epannsold-r2.zip,29.98 MB,zip,24/11/2017 12:09:15
EUROPROMS/epannsold.zip,29.64 MB,zip,24/11/2017 12:09:15
EUROPROMS/epanntotal-r2.zip,1.10 MB,zip,24/11/2017 12:09:15
EUROPROMS/epanntotal.zip,1.12 MB,zip,24/11/2017 12:09:15
EUROPROMS/epmens.zip,1.13 MB,zip,24/11/2017 12:09:15
EUROPROMS/Footnotes2017-01.zip,12.40 MB,zip,24/11/2017 12:09:14
NON_EU_DATASETS,,DIR,30/05/2022 11:06:16
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020,,DIR,14/03/2023 15:22:44
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/full_GB202002.7z,2.16 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/full_GB202003.7z,2.17 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/full_GB202004.7z,1.87 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/full_GB202005.7z,1.90 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/full_GB202006.7z,2.03 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/full_GB202007.7z,2.12 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/full_GB202008.7z,2.06 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/full_GB202009.7z,2.13 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/full_GB202010.7z,2.18 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/lines.txt,648 bytes,txt,16/11/2021 12:04:21
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/sizes.txt,1.78 KB,txt,16/11/2021 12:04:22
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/trhs_GB202002.7z,1.54 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/trhs_GB202003.7z,1.56 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/trhs_GB202004.7z,1.33 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/trhs_GB202005.7z,1.35 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/trhs_GB202006.7z,1.44 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/trhs_GB202007.7z,1.52 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/trhs_GB202008.7z,1.47 MB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/trhs_GB202009.7z,1.52 MB,7z,15/02/2021 08:44:11
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/trhs_GB202010.7z,1.56 MB,7z,15/02/2021 08:44:11
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/tr_GB202002.7z,541.24 KB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/tr_GB202003.7z,577.40 KB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/tr_GB202004.7z,484.27 KB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/tr_GB202005.7z,468.86 KB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/tr_GB202006.7z,485.66 KB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/tr_GB202007.7z,532.73 KB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/tr_GB202008.7z,520.44 KB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/tr_GB202009.7z,541.66 KB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/UK_TRADE_SINCE_FEB_2020/tr_GB202010.7z,559.01 KB,7z,15/02/2021 08:44:10
NON_EU_DATASETS/XI_TRADE_SINCE_2021,,DIR,15/06/2023 10:43:10
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202101.7z,165.85 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202102.7z,170.21 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202103.7z,180.65 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202104.7z,171.61 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202105.7z,165.48 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202106.7z,167.22 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202107.7z,166.25 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202108.7z,161.77 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202109.7z,168.25 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202110.7z,165.61 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202111.7z,163.37 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202112.7z,157.64 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202152.7z,395.44 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202201.7z,188.97 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202202.7z,196.21 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202203.7z,207.53 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202204.7z,198.96 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202205.7z,201.73 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202206.7z,202.43 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202207.7z,192.55 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202208.7z,195.37 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202209.7z,203.61 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202210.7z,202.78 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202211.7z,203.41 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202212.7z,188.06 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202252.7z,439.72 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202301.7z,188.48 KB,7z,15/06/2023 10:43:10
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202302.7z,190.42 KB,7z,15/06/2023 10:43:10
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202303.7z,196.31 KB,7z,15/06/2023 10:43:10
NON_EU_DATASETS/XI_TRADE_SINCE_2021/full_XI202304.7z,30.43 KB,7z,15/06/2023 10:43:10
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202101.7z,113.17 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202102.7z,116.69 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202103.7z,123.44 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202104.7z,116.90 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202105.7z,111.85 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202106.7z,112.29 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202107.7z,112.18 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202108.7z,108.00 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202109.7z,112.53 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202110.7z,111.40 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202111.7z,109.48 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202112.7z,105.66 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202152.7z,279.05 KB,7z,15/02/2023 10:20:51
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202201.7z,122.84 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202202.7z,128.78 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202203.7z,136.96 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202204.7z,130.95 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202205.7z,133.00 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202206.7z,133.34 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202207.7z,126.76 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202208.7z,129.02 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202209.7z,134.67 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202210.7z,134.54 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202211.7z,135.06 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202212.7z,125.16 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202252.7z,303.07 KB,7z,20/04/2023 10:38:23
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202301.7z,125.36 KB,7z,15/06/2023 10:43:10
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202302.7z,126.98 KB,7z,15/06/2023 10:43:10
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202303.7z,132.04 KB,7z,15/06/2023 10:43:10
NON_EU_DATASETS/XI_TRADE_SINCE_2021/trhs_XI202304.7z,22.90 KB,7z,15/06/2023 10:43:10
TEC exercise for CA 2014.zip,122.52 KB,zip,18/07/2019 13:05:02
TEC exercise for VN and JP 2017.zip,1.54 MB,zip,16/07/2018 10:11:15

So the download URL for EUROPROMS/epannsold.zip would be https://ec.europa.eu/eurostat/api/dissemination/files?file=comext/EUROPROMS/epannsold.zip

Directory listing in HTML for /comext and /metadata

For backward compatibility with web-crawlers on /comext and /metadata, API Files will continue to serve HTML content (when format=html is specified as query parameter)


Expand all   Collapse all
  • No labels
_