Developing an Open Data Portal for the ESA Climate Change Initiative
We introduce the rationale for, and architecture of, the European Space Agency Climate Change Initiative (CCI) Open Data Portal (http://cci.esa.int/data/). The Open Data Portal hosts a set of richly diverse datasets – 13 “Essential Climate Variables” – from the CCI programme in a consistent and harm...
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Ubiquity Press
2020-04-01
|
| Series: | Data Science Journal |
| Subjects: | |
| Online Access: | https://datascience.codata.org/articles/1137 |
| _version_ | 1811322394314276864 |
|---|---|
| author | Philip Kershaw Kevin Halsall Bryan N. Lawrence Victoria Bennett Steve Donegan Alan Iwi Martin Juckes Eduardo Pechorro Ruth Petrie Joe Singleton Ag Stephens Alison Waterfall Antony Wilson Alexander Wood |
| author_facet | Philip Kershaw Kevin Halsall Bryan N. Lawrence Victoria Bennett Steve Donegan Alan Iwi Martin Juckes Eduardo Pechorro Ruth Petrie Joe Singleton Ag Stephens Alison Waterfall Antony Wilson Alexander Wood |
| author_sort | Philip Kershaw |
| collection | DOAJ |
| description | We introduce the rationale for, and architecture of, the European Space Agency Climate Change Initiative (CCI) Open Data Portal (http://cci.esa.int/data/). The Open Data Portal hosts a set of richly diverse datasets – 13 “Essential Climate Variables” – from the CCI programme in a consistent and harmonised form and to provides a single point of access for the (>100 TB) data for broad dissemination to an international user community. These data have been produced by a range of different institutions and vary across both scientific and spatio-temporal characteristics. This heterogeneity of the data together with the range of services to be supported presented significant technical challenges. An iterative development methodology was key to tackling these challenges: the system developed exploits a workflow which takes data that conforms to the CCI data specification, ingests it into a managed archive and uses both manual and automatically generated metadata to support data discovery, browse, and delivery services. It utilises both Earth System Grid Federation (ESGF) data nodes and the Open Geospatial Consortium Catalogue Service for the Web (OGC-CSW) interface, serving data into both the ESGF and the Global Earth Observation System of Systems (GEOSS). A key part of the system is a new vocabulary server, populated with CCI specific terms and relationships which integrates OGC-CSW and ESGF search services together, developed as part of a dialogue between domain scientists and linked data specialists. These services have enabled the development of a unified user interface for graphical search and visualisation – the CCI Open Data Portal Web Presence. |
| first_indexed | 2024-04-13T13:35:35Z |
| format | Article |
| id | doaj.art-25a3b1082b8c43a0b5bbe54a3985e876 |
| institution | Directory Open Access Journal |
| issn | 1683-1470 |
| language | English |
| last_indexed | 2024-04-13T13:35:35Z |
| publishDate | 2020-04-01 |
| publisher | Ubiquity Press |
| record_format | Article |
| series | Data Science Journal |
| spelling | doaj.art-25a3b1082b8c43a0b5bbe54a3985e8762022-12-22T02:44:48ZengUbiquity PressData Science Journal1683-14702020-04-0119110.5334/dsj-2020-016771Developing an Open Data Portal for the ESA Climate Change InitiativePhilip Kershaw0Kevin Halsall1Bryan N. Lawrence2Victoria Bennett3Steve Donegan4Alan Iwi5Martin Juckes6Eduardo Pechorro7Ruth Petrie8Joe Singleton9Ag Stephens10Alison Waterfall11Antony Wilson12Alexander Wood13Centre for Environmental Data Analysis, RAL Space, STFC Rutherford Appleton Laboratory, Chilton, Didcot; National Centre for Earth ObservationTelespazio VEGA UK Ltd, 350 Capability Green, Luton, BedfordshireNational Centre for Atmospheric Science; Department of Computer Science, University of Reading, ReadingCentre for Environmental Data Analysis, RAL Space, STFC Rutherford Appleton Laboratory, Chilton, Didcot; National Centre for Earth ObservationCentre for Environmental Data Analysis, RAL Space, STFC Rutherford Appleton Laboratory, Chilton, Didcot; National Centre for Earth ObservationCentre for Environmental Data Analysis, RAL Space, STFC Rutherford Appleton Laboratory, Chilton, Didcot; National Centre for Atmospheric ScienceCentre for Environmental Data Analysis, RAL Space, STFC Rutherford Appleton Laboratory, Chilton, Didcot; National Centre for Atmospheric ScienceESA Climate Office, ESCAT, Harwell Campus, Chilton, DidcotCentre for Environmental Data Analysis, RAL Space, STFC Rutherford Appleton Laboratory, Chilton, Didcot; National Centre for Atmospheric ScienceCentre for Environmental Data Analysis, RAL Space, STFC Rutherford Appleton Laboratory, Chilton, DidcotCentre for Environmental Data Analysis, RAL Space, STFC Rutherford Appleton Laboratory, Chilton, Didcot; National Centre for Atmospheric ScienceCentre for Environmental Data Analysis, RAL Space, STFC Rutherford Appleton Laboratory, Chilton, Didcot; National Centre for Earth ObservationScientific Computing, STFC Rutherford Appleton Laboratory, Chilton, DidcotCGI Space, Defence and Intelligence, CGI, Leatherhead, SurreyWe introduce the rationale for, and architecture of, the European Space Agency Climate Change Initiative (CCI) Open Data Portal (http://cci.esa.int/data/). The Open Data Portal hosts a set of richly diverse datasets – 13 “Essential Climate Variables” – from the CCI programme in a consistent and harmonised form and to provides a single point of access for the (>100 TB) data for broad dissemination to an international user community. These data have been produced by a range of different institutions and vary across both scientific and spatio-temporal characteristics. This heterogeneity of the data together with the range of services to be supported presented significant technical challenges. An iterative development methodology was key to tackling these challenges: the system developed exploits a workflow which takes data that conforms to the CCI data specification, ingests it into a managed archive and uses both manual and automatically generated metadata to support data discovery, browse, and delivery services. It utilises both Earth System Grid Federation (ESGF) data nodes and the Open Geospatial Consortium Catalogue Service for the Web (OGC-CSW) interface, serving data into both the ESGF and the Global Earth Observation System of Systems (GEOSS). A key part of the system is a new vocabulary server, populated with CCI specific terms and relationships which integrates OGC-CSW and ESGF search services together, developed as part of a dialogue between domain scientists and linked data specialists. These services have enabled the development of a unified user interface for graphical search and visualisation – the CCI Open Data Portal Web Presence.https://datascience.codata.org/articles/1137essential climate variablescontrolled vocabularieslinked datacf-netcdfopendapogc services |
| spellingShingle | Philip Kershaw Kevin Halsall Bryan N. Lawrence Victoria Bennett Steve Donegan Alan Iwi Martin Juckes Eduardo Pechorro Ruth Petrie Joe Singleton Ag Stephens Alison Waterfall Antony Wilson Alexander Wood Developing an Open Data Portal for the ESA Climate Change Initiative Data Science Journal essential climate variables controlled vocabularies linked data cf-netcdf opendap ogc services |
| title | Developing an Open Data Portal for the ESA Climate Change Initiative |
| title_full | Developing an Open Data Portal for the ESA Climate Change Initiative |
| title_fullStr | Developing an Open Data Portal for the ESA Climate Change Initiative |
| title_full_unstemmed | Developing an Open Data Portal for the ESA Climate Change Initiative |
| title_short | Developing an Open Data Portal for the ESA Climate Change Initiative |
| title_sort | developing an open data portal for the esa climate change initiative |
| topic | essential climate variables controlled vocabularies linked data cf-netcdf opendap ogc services |
| url | https://datascience.codata.org/articles/1137 |
| work_keys_str_mv | AT philipkershaw developinganopendataportalfortheesaclimatechangeinitiative AT kevinhalsall developinganopendataportalfortheesaclimatechangeinitiative AT bryannlawrence developinganopendataportalfortheesaclimatechangeinitiative AT victoriabennett developinganopendataportalfortheesaclimatechangeinitiative AT stevedonegan developinganopendataportalfortheesaclimatechangeinitiative AT alaniwi developinganopendataportalfortheesaclimatechangeinitiative AT martinjuckes developinganopendataportalfortheesaclimatechangeinitiative AT eduardopechorro developinganopendataportalfortheesaclimatechangeinitiative AT ruthpetrie developinganopendataportalfortheesaclimatechangeinitiative AT joesingleton developinganopendataportalfortheesaclimatechangeinitiative AT agstephens developinganopendataportalfortheesaclimatechangeinitiative AT alisonwaterfall developinganopendataportalfortheesaclimatechangeinitiative AT antonywilson developinganopendataportalfortheesaclimatechangeinitiative AT alexanderwood developinganopendataportalfortheesaclimatechangeinitiative |