Automated Data Quality Assessment of Marine Sensors
The automated collection of data (e.g., through sensor networks) has led to a massive increase in the quantity of environmental and other data available. The sheer quantity of data and growing need for real-time ingestion of sensor data (e.g., alerts and forecasts from physical models) means that au...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2011-10-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | http://www.mdpi.com/1424-8220/11/10/9589/ |
| _version_ | 1811184267202396160 |
|---|---|
| author | Daniel V. Smith Leon Reznik Paulo A. de Souza Greg P. Timms |
| author_facet | Daniel V. Smith Leon Reznik Paulo A. de Souza Greg P. Timms |
| author_sort | Daniel V. Smith |
| collection | DOAJ |
| description | The automated collection of data (e.g., through sensor networks) has led to a massive increase in the quantity of environmental and other data available. The sheer quantity of data and growing need for real-time ingestion of sensor data (e.g., alerts and forecasts from physical models) means that automated Quality Assurance/Quality Control (QA/QC) is necessary to ensure that the data collected is fit for purpose. Current automated QA/QC approaches provide assessments based upon hard classifications of the gathered data; often as a binary decision of good or bad data that fails to quantify our confidence in the data for use in different applications. We propose a novel framework for automated data quality assessments that uses Fuzzy Logic to provide a continuous scale of data quality. This continuous quality scale is then used to compute error bars upon the data, which quantify the data uncertainty and provide a more meaningful measure of the data’s fitness for purpose in a particular application compared with hard quality classifications. The design principles of the framework are presented and enable both data statistics and expert knowledge to be incorporated into the uncertainty assessment. We have implemented and tested the framework upon a real time platform of temperature and conductivity sensors that have been deployed to monitor the Derwent Estuary in Hobart, Australia. Results indicate that the error bars generated from the Fuzzy QA/QC implementation are in good agreement with the error bars manually encoded by a domain expert. |
| first_indexed | 2024-04-11T13:09:32Z |
| format | Article |
| id | doaj.art-a7d2968933cd432da61d9d9a53a15444 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-04-11T13:09:32Z |
| publishDate | 2011-10-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-a7d2968933cd432da61d9d9a53a154442022-12-22T04:22:37ZengMDPI AGSensors1424-82202011-10-0111109589960210.3390/s111009589Automated Data Quality Assessment of Marine SensorsDaniel V. SmithLeon ReznikPaulo A. de SouzaGreg P. TimmsThe automated collection of data (e.g., through sensor networks) has led to a massive increase in the quantity of environmental and other data available. The sheer quantity of data and growing need for real-time ingestion of sensor data (e.g., alerts and forecasts from physical models) means that automated Quality Assurance/Quality Control (QA/QC) is necessary to ensure that the data collected is fit for purpose. Current automated QA/QC approaches provide assessments based upon hard classifications of the gathered data; often as a binary decision of good or bad data that fails to quantify our confidence in the data for use in different applications. We propose a novel framework for automated data quality assessments that uses Fuzzy Logic to provide a continuous scale of data quality. This continuous quality scale is then used to compute error bars upon the data, which quantify the data uncertainty and provide a more meaningful measure of the data’s fitness for purpose in a particular application compared with hard quality classifications. The design principles of the framework are presented and enable both data statistics and expert knowledge to be incorporated into the uncertainty assessment. We have implemented and tested the framework upon a real time platform of temperature and conductivity sensors that have been deployed to monitor the Derwent Estuary in Hobart, Australia. Results indicate that the error bars generated from the Fuzzy QA/QC implementation are in good agreement with the error bars manually encoded by a domain expert.http://www.mdpi.com/1424-8220/11/10/9589/sensorsmeasurement resultsqualityfuzzy logic |
| spellingShingle | Daniel V. Smith Leon Reznik Paulo A. de Souza Greg P. Timms Automated Data Quality Assessment of Marine Sensors Sensors sensors measurement results quality fuzzy logic |
| title | Automated Data Quality Assessment of Marine Sensors |
| title_full | Automated Data Quality Assessment of Marine Sensors |
| title_fullStr | Automated Data Quality Assessment of Marine Sensors |
| title_full_unstemmed | Automated Data Quality Assessment of Marine Sensors |
| title_short | Automated Data Quality Assessment of Marine Sensors |
| title_sort | automated data quality assessment of marine sensors |
| topic | sensors measurement results quality fuzzy logic |
| url | http://www.mdpi.com/1424-8220/11/10/9589/ |
| work_keys_str_mv | AT danielvsmith automateddataqualityassessmentofmarinesensors AT leonreznik automateddataqualityassessmentofmarinesensors AT pauloadesouza automateddataqualityassessmentofmarinesensors AT gregptimms automateddataqualityassessmentofmarinesensors |