Exploring the use of topological data analysis to automatically detect data quality faults
Data quality problems may occur in various forms in structured and semi-structured data sources. This paper details an unsupervised method of analyzing data quality that is agnostic to the semantics of the data, the format of the encoding, or the internal structure of the dataset. A distance functio...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-12-01
|
| Series: | Frontiers in Big Data |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fdata.2022.931398/full |
| _version_ | 1828095535108587520 |
|---|---|
| author | M. Eduard Tudoreanu |
| author_facet | M. Eduard Tudoreanu |
| author_sort | M. Eduard Tudoreanu |
| collection | DOAJ |
| description | Data quality problems may occur in various forms in structured and semi-structured data sources. This paper details an unsupervised method of analyzing data quality that is agnostic to the semantics of the data, the format of the encoding, or the internal structure of the dataset. A distance function is used to transform each record of a dataset into an n-dimensional vector of real numbers, which effectively transforms the original data into a high-dimensional point cloud. The shape of the point cloud is then efficiently examined via topological data analysis to find high-dimensional anomalies that may signal quality issues. The specific quality faults examined in this paper are the detection of records that, while not exactly the same, refer to the same entity. Our algorithm, based on topological data analysis, provides similar accuracy for both higher and lower quality data and performs better than a baseline approach for data with poor quality. |
| first_indexed | 2024-04-11T07:14:57Z |
| format | Article |
| id | doaj.art-d4a287f4027c466ba1aba8a780f322c1 |
| institution | Directory Open Access Journal |
| issn | 2624-909X |
| language | English |
| last_indexed | 2024-04-11T07:14:57Z |
| publishDate | 2022-12-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Big Data |
| spelling | doaj.art-d4a287f4027c466ba1aba8a780f322c12022-12-22T04:38:02ZengFrontiers Media S.A.Frontiers in Big Data2624-909X2022-12-01510.3389/fdata.2022.931398931398Exploring the use of topological data analysis to automatically detect data quality faultsM. Eduard TudoreanuData quality problems may occur in various forms in structured and semi-structured data sources. This paper details an unsupervised method of analyzing data quality that is agnostic to the semantics of the data, the format of the encoding, or the internal structure of the dataset. A distance function is used to transform each record of a dataset into an n-dimensional vector of real numbers, which effectively transforms the original data into a high-dimensional point cloud. The shape of the point cloud is then efficiently examined via topological data analysis to find high-dimensional anomalies that may signal quality issues. The specific quality faults examined in this paper are the detection of records that, while not exactly the same, refer to the same entity. Our algorithm, based on topological data analysis, provides similar accuracy for both higher and lower quality data and performs better than a baseline approach for data with poor quality.https://www.frontiersin.org/articles/10.3389/fdata.2022.931398/fulltopological data analysisMorse-Smale complexentity resolutionunsuperviseddistance-based point cloud |
| spellingShingle | M. Eduard Tudoreanu Exploring the use of topological data analysis to automatically detect data quality faults Frontiers in Big Data topological data analysis Morse-Smale complex entity resolution unsupervised distance-based point cloud |
| title | Exploring the use of topological data analysis to automatically detect data quality faults |
| title_full | Exploring the use of topological data analysis to automatically detect data quality faults |
| title_fullStr | Exploring the use of topological data analysis to automatically detect data quality faults |
| title_full_unstemmed | Exploring the use of topological data analysis to automatically detect data quality faults |
| title_short | Exploring the use of topological data analysis to automatically detect data quality faults |
| title_sort | exploring the use of topological data analysis to automatically detect data quality faults |
| topic | topological data analysis Morse-Smale complex entity resolution unsupervised distance-based point cloud |
| url | https://www.frontiersin.org/articles/10.3389/fdata.2022.931398/full |
| work_keys_str_mv | AT meduardtudoreanu exploringtheuseoftopologicaldataanalysistoautomaticallydetectdataqualityfaults |