Aurum: A Data Discovery System
© 2018 IEEE. Organizations face a data discovery problem when their analysts spend more time looking for relevant data than analyzing it. This problem has become commonplace in modern organizations as: i) data is stored across multiple storage systems, from databases to data lakes, to the cloud; ii)...
| Main Authors: | , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021
|
| Online Access: | https://hdl.handle.net/1721.1/137860 |
| _version_ | 1811085877536882688 |
|---|---|
| author | Castro Fernandez, Raul Abedjan, Ziawasch Koko, Famien Yuan, Gina Madden, Samuel Stonebraker, Michael |
| author2 | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory |
| author_facet | Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Castro Fernandez, Raul Abedjan, Ziawasch Koko, Famien Yuan, Gina Madden, Samuel Stonebraker, Michael |
| author_sort | Castro Fernandez, Raul |
| collection | MIT |
| description | © 2018 IEEE. Organizations face a data discovery problem when their analysts spend more time looking for relevant data than analyzing it. This problem has become commonplace in modern organizations as: i) data is stored across multiple storage systems, from databases to data lakes, to the cloud; ii) data scientists do not operate within the limits of well-defined schemas or a small number of data sources-instead, to answer complex questions they must access data spread across thousands of data sources. To address this problem, we capture relationships between datasets in an enterprise knowledge graph (EKG), which helps users to navigate among disparate sources. The contribution of this paper is AURUM, a system to build, maintain and query the EKG. To build the EKG, we introduce a Two-step process which scales to large datasets and requires only one-pass over the data, avoiding overloading the source systems. To maintain the EKG without re-reading all data every time, we introduce a resource-efficient sampling signature (RESS) method which works by only using a small sample of the data. Finally, to query the EKG, we introduce a collection of composable primitives, thus allowing users to define many different types of discovery queries. We describe our experience using AURUM in three corporate scenarios and do a performance evaluation of each component. |
| first_indexed | 2024-09-23T13:16:58Z |
| format | Article |
| id | mit-1721.1/137860 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T13:16:58Z |
| publishDate | 2021 |
| publisher | IEEE |
| record_format | dspace |
| spelling | mit-1721.1/1378602023-04-10T14:37:42Z Aurum: A Data Discovery System Castro Fernandez, Raul Abedjan, Ziawasch Koko, Famien Yuan, Gina Madden, Samuel Stonebraker, Michael Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory © 2018 IEEE. Organizations face a data discovery problem when their analysts spend more time looking for relevant data than analyzing it. This problem has become commonplace in modern organizations as: i) data is stored across multiple storage systems, from databases to data lakes, to the cloud; ii) data scientists do not operate within the limits of well-defined schemas or a small number of data sources-instead, to answer complex questions they must access data spread across thousands of data sources. To address this problem, we capture relationships between datasets in an enterprise knowledge graph (EKG), which helps users to navigate among disparate sources. The contribution of this paper is AURUM, a system to build, maintain and query the EKG. To build the EKG, we introduce a Two-step process which scales to large datasets and requires only one-pass over the data, avoiding overloading the source systems. To maintain the EKG without re-reading all data every time, we introduce a resource-efficient sampling signature (RESS) method which works by only using a small sample of the data. Finally, to query the EKG, we introduce a collection of composable primitives, thus allowing users to define many different types of discovery queries. We describe our experience using AURUM in three corporate scenarios and do a performance evaluation of each component. 2021-11-09T13:30:46Z 2021-11-09T13:30:46Z 2018-04 2019-06-18T16:47:11Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/137860 Castro Fernandez, Raul, Abedjan, Ziawasch, Koko, Famien, Yuan, Gina, Madden, Samuel et al. 2018. "Aurum: A Data Discovery System." en 10.1109/icde.2018.00094 Creative Commons Attribution-Noncommercial-Share Alike http://creativecommons.org/licenses/by-nc-sa/4.0/ application/pdf IEEE website |
| spellingShingle | Castro Fernandez, Raul Abedjan, Ziawasch Koko, Famien Yuan, Gina Madden, Samuel Stonebraker, Michael Aurum: A Data Discovery System |
| title | Aurum: A Data Discovery System |
| title_full | Aurum: A Data Discovery System |
| title_fullStr | Aurum: A Data Discovery System |
| title_full_unstemmed | Aurum: A Data Discovery System |
| title_short | Aurum: A Data Discovery System |
| title_sort | aurum a data discovery system |
| url | https://hdl.handle.net/1721.1/137860 |
| work_keys_str_mv | AT castrofernandezraul aurumadatadiscoverysystem AT abedjanziawasch aurumadatadiscoverysystem AT kokofamien aurumadatadiscoverysystem AT yuangina aurumadatadiscoverysystem AT maddensamuel aurumadatadiscoverysystem AT stonebrakermichael aurumadatadiscoverysystem |