Data and Metadata Upload System

The Data and Metadata Upload System is made up of the following components:

  • Delivery user interface (deliver, browse and track project data and metadata)
  • Validation process
  • Completeness calculations

  • Search tool with Facets

  • Project archive and Indicator package generation

The User manual for Project Manager is available here: How to deliver my data.

Delivery user interface

This Web user interface allows Projects Managers to deliver their Data and Metadata across the validation Process (see section bellow) by providing the following elements:

  • Various entry forms which allow to input Metadata, Data and switch the validation process states.
  • A comprehensive navigation which allows to browse across the structure of the delivered data and its validation process states.
  • A faceted search to inspect across different features the delivered data.

Validation Process

The validation process allows a Project Manager to deliver their Data jointly with a Project Reviewer across steps with ensures that the material delivered reaches a certain level of “normalization” and quality before being published via the ESPON Database Portal User Interface.

This is done through a workflow that controls Project, Main Data (Data + Metadata) and Other Data delivery. See the following page for details: Project validation process.

Completeness calculations

For indicators, a specific completeness calculation is applied, based on the spatial extent declared for each given indicator (e.g. EU28, EU28+4+CC, Alpine Region, Adriatic). The system calculates a ratio between the number of data available for each nomenclature, level, version and the number of expected data according to the spatial extent declared:

The calculation takes in account only the data of the expected territorial units according to the spatial extent declared :

Indicator 2 in the above figure illustrates the case of a few indicators from former ESPON 2013 activities, which have data for territorial units beyond the declared spatial extent. For the new indicators, the upload checking process restricts the loading of values for territorial units that are not in the declared spatial extension.

To enrich this first completeness calculation based on frequencies, three other metrics are provided based on the weighting of the number of available data by respectively the area, population (2016) and GDP (2011) of each territorial unit. According to these three weighted metrics, a missing value of a big territorial unit will have more influence on the completeness score than the missing value of a small territorial unit.

The information on completeness is currently delivered as Web Services (e.g. https://database.espon.eu/api/public/indicators/) as well as in the upload interface (see e.g. https://database.espon.eu/indicator/321/):

Search tool with Facets

This tool is available at https://database.espon.eu/search/ to all user authenticated to the Upload System: Project Managers, Project Reviewers and Project Approvers (ESPON EGTC).

It allows the user to find elements across the database (Main Data, Other Data, Indicators, Resources, etc, ...) using Facets which expose some chosen "dimensions" of the data set (e.g.: Keywords, Spatial nomenclature, Validation process state, Content type, etc, ...). This allows to discover and find items by drilling down the result while clicking the Facets items links and observing their count states.

There are two groups of facets:

  • General Facets are toward the whole set of items
  • Indicator Facets are specific to Indicators

For example, Project Reviewers or Project Approver (ESPON EGTC) might use ‘Validation Process (Datasets)’ Facet to find Datasets and their inner Indicators given their state within the delivery workflow.

Depending on the Facet, a hierarchical display allows to explore the nature of a particular "dimension" (e.g. the spatial nomenclatures). Finally, this search includes a 'keyword(s)' box, which searches across all fields.

Each returned item shows its name but also 2 different links:

  • A link to its display in public version (read only)
  • A link to its display in editing mode (available when authenticated and accessible only if permission is granted)

The implementation uses Apache Solr, a free software implementing a search engine. This allows to expose a complex database structure without actually making any modification on the requirements or complexity of that database.

Project archive and Indicator package generation

Project archives and Indicator packages are built as ZIP files during dedicated phases of the Validation Process and are made available via the User Interface of the ESPON 2020 Database Portal. See the following page for more information: Project archive and Indicator package generation.