Recommendations to Improve Downloads of Large Earth Observation Data

With the volume of Earth observation data expanding rapidly, cloud computing is quickly changing the way these data are processed, analyzed, and visualized. Collocating freely available Earth observation data on a cloud computing infrastructure may create opportunities unforeseen by the original dat...

Full description

Bibliographic Details
Main Authors: Rahul Ramachandran, Christopher Lynnes, Kathleen Baynes, Kevin Murphy, Jamie Baker, Jamie Kinney, Ariel Gold, Jed Sundwall, Mark Korver, Allison Lieber, William Vambenepe, Matthew Hancher, Rebecca Moore, Tyler Erickson, Josh Henretig, Brant Zwiefel, Heather Patrick-Ahlstrom, Matthew J. Smith
Format: Article
Language:English
Published: Ubiquity Press 2018-01-01
Series:Data Science Journal
Subjects:
Online Access:https://datascience.codata.org/articles/732
_version_ 1811242718769184768
author Rahul Ramachandran
Christopher Lynnes
Kathleen Baynes
Kevin Murphy
Jamie Baker
Jamie Kinney
Ariel Gold
Jed Sundwall
Mark Korver
Allison Lieber
William Vambenepe
Matthew Hancher
Rebecca Moore
Tyler Erickson
Josh Henretig
Brant Zwiefel
Heather Patrick-Ahlstrom
Matthew J. Smith
author_facet Rahul Ramachandran
Christopher Lynnes
Kathleen Baynes
Kevin Murphy
Jamie Baker
Jamie Kinney
Ariel Gold
Jed Sundwall
Mark Korver
Allison Lieber
William Vambenepe
Matthew Hancher
Rebecca Moore
Tyler Erickson
Josh Henretig
Brant Zwiefel
Heather Patrick-Ahlstrom
Matthew J. Smith
author_sort Rahul Ramachandran
collection DOAJ
description With the volume of Earth observation data expanding rapidly, cloud computing is quickly changing the way these data are processed, analyzed, and visualized. Collocating freely available Earth observation data on a cloud computing infrastructure may create opportunities unforeseen by the original data provider for innovation and value-added data re-use, but existing systems at data centers are not designed for supporting requests for large data transfers. A lack of common methodology necessitates that each data center handle such requests from different cloud vendors differently. Guidelines are needed to support enabling all cloud vendors to utilize a common methodology for bulk-downloading data from data centers, thus preventing the providers from building custom capabilities to meet the needs of individual vendors. This paper presents recommendations distilled from use cases provided by three cloud vendors (Amazon, Google, and Microsoft) and are based on the vendors’ interactions with data systems at different Federal agencies and organizations. These specific recommendations range from obvious steps for improving data usability (such as ensuring the use of standard data formats and commonly supported projections) to non-obvious undertakings important for enabling bulk data downloads at scale. These recommendations can be used to evaluate and improve existing data systems for high-volume data transfers, and their adoption can lead to cloud vendors utilizing a common methodology.
first_indexed 2024-04-12T13:56:10Z
format Article
id doaj.art-95baa21293804e29b961215520ef8c14
institution Directory Open Access Journal
issn 1683-1470
language English
last_indexed 2024-04-12T13:56:10Z
publishDate 2018-01-01
publisher Ubiquity Press
record_format Article
series Data Science Journal
spelling doaj.art-95baa21293804e29b961215520ef8c142022-12-22T03:30:23ZengUbiquity PressData Science Journal1683-14702018-01-011710.5334/dsj-2018-002660Recommendations to Improve Downloads of Large Earth Observation DataRahul Ramachandran0Christopher Lynnes1Kathleen Baynes2Kevin Murphy3Jamie Baker4Jamie Kinney5Ariel Gold6Jed Sundwall7Mark Korver8Allison Lieber9William Vambenepe10Matthew Hancher11Rebecca Moore12Tyler Erickson13Josh Henretig14Brant Zwiefel15Heather Patrick-Ahlstrom16Matthew J. Smith17NASA Marshall Space Flight CenterNASA Goddard Space Flight CenterNASA Goddard Space Flight CenterNASA HeadquartersAmazon Web ServicesAmazon Web ServicesAmazon Web ServicesAmazon Web ServicesAmazon Web ServicesGoogleGoogleGoogleGoogleGoogleMicrosoftMicrosoftMicrosoftMicrosoftWith the volume of Earth observation data expanding rapidly, cloud computing is quickly changing the way these data are processed, analyzed, and visualized. Collocating freely available Earth observation data on a cloud computing infrastructure may create opportunities unforeseen by the original data provider for innovation and value-added data re-use, but existing systems at data centers are not designed for supporting requests for large data transfers. A lack of common methodology necessitates that each data center handle such requests from different cloud vendors differently. Guidelines are needed to support enabling all cloud vendors to utilize a common methodology for bulk-downloading data from data centers, thus preventing the providers from building custom capabilities to meet the needs of individual vendors. This paper presents recommendations distilled from use cases provided by three cloud vendors (Amazon, Google, and Microsoft) and are based on the vendors’ interactions with data systems at different Federal agencies and organizations. These specific recommendations range from obvious steps for improving data usability (such as ensuring the use of standard data formats and commonly supported projections) to non-obvious undertakings important for enabling bulk data downloads at scale. These recommendations can be used to evaluate and improve existing data systems for high-volume data transfers, and their adoption can lead to cloud vendors utilizing a common methodology.https://datascience.codata.org/articles/732Earth Observation DataLarge Data TransfersCloudBest Practices
spellingShingle Rahul Ramachandran
Christopher Lynnes
Kathleen Baynes
Kevin Murphy
Jamie Baker
Jamie Kinney
Ariel Gold
Jed Sundwall
Mark Korver
Allison Lieber
William Vambenepe
Matthew Hancher
Rebecca Moore
Tyler Erickson
Josh Henretig
Brant Zwiefel
Heather Patrick-Ahlstrom
Matthew J. Smith
Recommendations to Improve Downloads of Large Earth Observation Data
Data Science Journal
Earth Observation Data
Large Data Transfers
Cloud
Best Practices
title Recommendations to Improve Downloads of Large Earth Observation Data
title_full Recommendations to Improve Downloads of Large Earth Observation Data
title_fullStr Recommendations to Improve Downloads of Large Earth Observation Data
title_full_unstemmed Recommendations to Improve Downloads of Large Earth Observation Data
title_short Recommendations to Improve Downloads of Large Earth Observation Data
title_sort recommendations to improve downloads of large earth observation data
topic Earth Observation Data
Large Data Transfers
Cloud
Best Practices
url https://datascience.codata.org/articles/732
work_keys_str_mv AT rahulramachandran recommendationstoimprovedownloadsoflargeearthobservationdata
AT christopherlynnes recommendationstoimprovedownloadsoflargeearthobservationdata
AT kathleenbaynes recommendationstoimprovedownloadsoflargeearthobservationdata
AT kevinmurphy recommendationstoimprovedownloadsoflargeearthobservationdata
AT jamiebaker recommendationstoimprovedownloadsoflargeearthobservationdata
AT jamiekinney recommendationstoimprovedownloadsoflargeearthobservationdata
AT arielgold recommendationstoimprovedownloadsoflargeearthobservationdata
AT jedsundwall recommendationstoimprovedownloadsoflargeearthobservationdata
AT markkorver recommendationstoimprovedownloadsoflargeearthobservationdata
AT allisonlieber recommendationstoimprovedownloadsoflargeearthobservationdata
AT williamvambenepe recommendationstoimprovedownloadsoflargeearthobservationdata
AT matthewhancher recommendationstoimprovedownloadsoflargeearthobservationdata
AT rebeccamoore recommendationstoimprovedownloadsoflargeearthobservationdata
AT tylererickson recommendationstoimprovedownloadsoflargeearthobservationdata
AT joshhenretig recommendationstoimprovedownloadsoflargeearthobservationdata
AT brantzwiefel recommendationstoimprovedownloadsoflargeearthobservationdata
AT heatherpatrickahlstrom recommendationstoimprovedownloadsoflargeearthobservationdata
AT matthewjsmith recommendationstoimprovedownloadsoflargeearthobservationdata