Recommendations to Improve Downloads of Large Earth Observation Data
With the volume of Earth observation data expanding rapidly, cloud computing is quickly changing the way these data are processed, analyzed, and visualized. Collocating freely available Earth observation data on a cloud computing infrastructure may create opportunities unforeseen by the original dat...
Main Authors: | , , , , , , , , , , , , , , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
Ubiquity Press
2018-01-01
|
Series: | Data Science Journal |
Subjects: | |
Online Access: | https://datascience.codata.org/articles/732 |
_version_ | 1811242718769184768 |
---|---|
author | Rahul Ramachandran Christopher Lynnes Kathleen Baynes Kevin Murphy Jamie Baker Jamie Kinney Ariel Gold Jed Sundwall Mark Korver Allison Lieber William Vambenepe Matthew Hancher Rebecca Moore Tyler Erickson Josh Henretig Brant Zwiefel Heather Patrick-Ahlstrom Matthew J. Smith |
author_facet | Rahul Ramachandran Christopher Lynnes Kathleen Baynes Kevin Murphy Jamie Baker Jamie Kinney Ariel Gold Jed Sundwall Mark Korver Allison Lieber William Vambenepe Matthew Hancher Rebecca Moore Tyler Erickson Josh Henretig Brant Zwiefel Heather Patrick-Ahlstrom Matthew J. Smith |
author_sort | Rahul Ramachandran |
collection | DOAJ |
description | With the volume of Earth observation data expanding rapidly, cloud computing is quickly changing the way these data are processed, analyzed, and visualized. Collocating freely available Earth observation data on a cloud computing infrastructure may create opportunities unforeseen by the original data provider for innovation and value-added data re-use, but existing systems at data centers are not designed for supporting requests for large data transfers. A lack of common methodology necessitates that each data center handle such requests from different cloud vendors differently. Guidelines are needed to support enabling all cloud vendors to utilize a common methodology for bulk-downloading data from data centers, thus preventing the providers from building custom capabilities to meet the needs of individual vendors. This paper presents recommendations distilled from use cases provided by three cloud vendors (Amazon, Google, and Microsoft) and are based on the vendors’ interactions with data systems at different Federal agencies and organizations. These specific recommendations range from obvious steps for improving data usability (such as ensuring the use of standard data formats and commonly supported projections) to non-obvious undertakings important for enabling bulk data downloads at scale. These recommendations can be used to evaluate and improve existing data systems for high-volume data transfers, and their adoption can lead to cloud vendors utilizing a common methodology. |
first_indexed | 2024-04-12T13:56:10Z |
format | Article |
id | doaj.art-95baa21293804e29b961215520ef8c14 |
institution | Directory Open Access Journal |
issn | 1683-1470 |
language | English |
last_indexed | 2024-04-12T13:56:10Z |
publishDate | 2018-01-01 |
publisher | Ubiquity Press |
record_format | Article |
series | Data Science Journal |
spelling | doaj.art-95baa21293804e29b961215520ef8c142022-12-22T03:30:23ZengUbiquity PressData Science Journal1683-14702018-01-011710.5334/dsj-2018-002660Recommendations to Improve Downloads of Large Earth Observation DataRahul Ramachandran0Christopher Lynnes1Kathleen Baynes2Kevin Murphy3Jamie Baker4Jamie Kinney5Ariel Gold6Jed Sundwall7Mark Korver8Allison Lieber9William Vambenepe10Matthew Hancher11Rebecca Moore12Tyler Erickson13Josh Henretig14Brant Zwiefel15Heather Patrick-Ahlstrom16Matthew J. Smith17NASA Marshall Space Flight CenterNASA Goddard Space Flight CenterNASA Goddard Space Flight CenterNASA HeadquartersAmazon Web ServicesAmazon Web ServicesAmazon Web ServicesAmazon Web ServicesAmazon Web ServicesGoogleGoogleGoogleGoogleGoogleMicrosoftMicrosoftMicrosoftMicrosoftWith the volume of Earth observation data expanding rapidly, cloud computing is quickly changing the way these data are processed, analyzed, and visualized. Collocating freely available Earth observation data on a cloud computing infrastructure may create opportunities unforeseen by the original data provider for innovation and value-added data re-use, but existing systems at data centers are not designed for supporting requests for large data transfers. A lack of common methodology necessitates that each data center handle such requests from different cloud vendors differently. Guidelines are needed to support enabling all cloud vendors to utilize a common methodology for bulk-downloading data from data centers, thus preventing the providers from building custom capabilities to meet the needs of individual vendors. This paper presents recommendations distilled from use cases provided by three cloud vendors (Amazon, Google, and Microsoft) and are based on the vendors’ interactions with data systems at different Federal agencies and organizations. These specific recommendations range from obvious steps for improving data usability (such as ensuring the use of standard data formats and commonly supported projections) to non-obvious undertakings important for enabling bulk data downloads at scale. These recommendations can be used to evaluate and improve existing data systems for high-volume data transfers, and their adoption can lead to cloud vendors utilizing a common methodology.https://datascience.codata.org/articles/732Earth Observation DataLarge Data TransfersCloudBest Practices |
spellingShingle | Rahul Ramachandran Christopher Lynnes Kathleen Baynes Kevin Murphy Jamie Baker Jamie Kinney Ariel Gold Jed Sundwall Mark Korver Allison Lieber William Vambenepe Matthew Hancher Rebecca Moore Tyler Erickson Josh Henretig Brant Zwiefel Heather Patrick-Ahlstrom Matthew J. Smith Recommendations to Improve Downloads of Large Earth Observation Data Data Science Journal Earth Observation Data Large Data Transfers Cloud Best Practices |
title | Recommendations to Improve Downloads of Large Earth Observation Data |
title_full | Recommendations to Improve Downloads of Large Earth Observation Data |
title_fullStr | Recommendations to Improve Downloads of Large Earth Observation Data |
title_full_unstemmed | Recommendations to Improve Downloads of Large Earth Observation Data |
title_short | Recommendations to Improve Downloads of Large Earth Observation Data |
title_sort | recommendations to improve downloads of large earth observation data |
topic | Earth Observation Data Large Data Transfers Cloud Best Practices |
url | https://datascience.codata.org/articles/732 |
work_keys_str_mv | AT rahulramachandran recommendationstoimprovedownloadsoflargeearthobservationdata AT christopherlynnes recommendationstoimprovedownloadsoflargeearthobservationdata AT kathleenbaynes recommendationstoimprovedownloadsoflargeearthobservationdata AT kevinmurphy recommendationstoimprovedownloadsoflargeearthobservationdata AT jamiebaker recommendationstoimprovedownloadsoflargeearthobservationdata AT jamiekinney recommendationstoimprovedownloadsoflargeearthobservationdata AT arielgold recommendationstoimprovedownloadsoflargeearthobservationdata AT jedsundwall recommendationstoimprovedownloadsoflargeearthobservationdata AT markkorver recommendationstoimprovedownloadsoflargeearthobservationdata AT allisonlieber recommendationstoimprovedownloadsoflargeearthobservationdata AT williamvambenepe recommendationstoimprovedownloadsoflargeearthobservationdata AT matthewhancher recommendationstoimprovedownloadsoflargeearthobservationdata AT rebeccamoore recommendationstoimprovedownloadsoflargeearthobservationdata AT tylererickson recommendationstoimprovedownloadsoflargeearthobservationdata AT joshhenretig recommendationstoimprovedownloadsoflargeearthobservationdata AT brantzwiefel recommendationstoimprovedownloadsoflargeearthobservationdata AT heatherpatrickahlstrom recommendationstoimprovedownloadsoflargeearthobservationdata AT matthewjsmith recommendationstoimprovedownloadsoflargeearthobservationdata |