The <i>T</i> Index: Measuring the Reliability of Accuracy Estimates Obtained from Non-Probability Samples
In remote sensing, the term accuracy typically expresses the degree of correctness of a map. Best practices in accuracy assessment have been widely researched and include guidelines on how to select validation data using probability sampling designs. In practice, however, probability samples may be...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2020-08-01
|
| Series: | Remote Sensing |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2072-4292/12/15/2483 |
| _version_ | 1797560466912837632 |
|---|---|
| author | François Waldner |
| author_facet | François Waldner |
| author_sort | François Waldner |
| collection | DOAJ |
| description | In remote sensing, the term accuracy typically expresses the degree of correctness of a map. Best practices in accuracy assessment have been widely researched and include guidelines on how to select validation data using probability sampling designs. In practice, however, probability samples may be lacking and, instead, cross-validation using non-probability samples is common. This practice is risky because the resulting accuracy estimates can easily be mistaken for map accuracy. The following question arises: to what extent are accuracy estimates obtained from non-probability samples representative of map accuracy? This letter introduces the <i>T</i> index to answer this question. Certain cross-validation designs (such as the common single-split or hold-out validation) provide representative accuracy estimates when hold-out sets are simple random samples of the map population. The <i>T</i> index essentially measures the probability of a hold-out set of unknown sampling design to be a simple random sample. To that aim, we compare its spread in the feature space against the spread of random unlabelled samples of the same size. Data spread is measured by a variant of Moran’s <i>I</i> autocorrelation index. Consistent interpretation of the <i>T</i> index is proposed through the prism of significance testing, with <i>T</i> values < 0.05 indicating unreliable accuracy estimates. Its relevance and interpretation guidelines are also illustrated in a case study on crop-type mapping. Uptake of the <i>T</i> index by the remote-sensing community will help inform about—and sometimes caution against—the representativeness of accuracy estimates obtained by cross-validation, so that users can better decide whether a map is fit for their purpose or how its accuracy impacts their application. Subsequently, the <i>T</i> index will build trust and improve the transparency of accuracy assessment in conditions which deviate from best practices. |
| first_indexed | 2024-03-10T18:00:57Z |
| format | Article |
| id | doaj.art-5e2bb7e3b8994c3c8402b8f0c4e42b6d |
| institution | Directory Open Access Journal |
| issn | 2072-4292 |
| language | English |
| last_indexed | 2024-03-10T18:00:57Z |
| publishDate | 2020-08-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Remote Sensing |
| spelling | doaj.art-5e2bb7e3b8994c3c8402b8f0c4e42b6d2023-11-20T08:54:13ZengMDPI AGRemote Sensing2072-42922020-08-011215248310.3390/rs12152483The <i>T</i> Index: Measuring the Reliability of Accuracy Estimates Obtained from Non-Probability SamplesFrançois Waldner0CSIRO Agriculture & Food, 306 Carmody Road, St Lucia 4067, AustraliaIn remote sensing, the term accuracy typically expresses the degree of correctness of a map. Best practices in accuracy assessment have been widely researched and include guidelines on how to select validation data using probability sampling designs. In practice, however, probability samples may be lacking and, instead, cross-validation using non-probability samples is common. This practice is risky because the resulting accuracy estimates can easily be mistaken for map accuracy. The following question arises: to what extent are accuracy estimates obtained from non-probability samples representative of map accuracy? This letter introduces the <i>T</i> index to answer this question. Certain cross-validation designs (such as the common single-split or hold-out validation) provide representative accuracy estimates when hold-out sets are simple random samples of the map population. The <i>T</i> index essentially measures the probability of a hold-out set of unknown sampling design to be a simple random sample. To that aim, we compare its spread in the feature space against the spread of random unlabelled samples of the same size. Data spread is measured by a variant of Moran’s <i>I</i> autocorrelation index. Consistent interpretation of the <i>T</i> index is proposed through the prism of significance testing, with <i>T</i> values < 0.05 indicating unreliable accuracy estimates. Its relevance and interpretation guidelines are also illustrated in a case study on crop-type mapping. Uptake of the <i>T</i> index by the remote-sensing community will help inform about—and sometimes caution against—the representativeness of accuracy estimates obtained by cross-validation, so that users can better decide whether a map is fit for their purpose or how its accuracy impacts their application. Subsequently, the <i>T</i> index will build trust and improve the transparency of accuracy assessment in conditions which deviate from best practices.https://www.mdpi.com/2072-4292/12/15/2483accuracy assessmentvalidationclassificationspatial balanceunlabelled datasample selection bias |
| spellingShingle | François Waldner The <i>T</i> Index: Measuring the Reliability of Accuracy Estimates Obtained from Non-Probability Samples Remote Sensing accuracy assessment validation classification spatial balance unlabelled data sample selection bias |
| title | The <i>T</i> Index: Measuring the Reliability of Accuracy Estimates Obtained from Non-Probability Samples |
| title_full | The <i>T</i> Index: Measuring the Reliability of Accuracy Estimates Obtained from Non-Probability Samples |
| title_fullStr | The <i>T</i> Index: Measuring the Reliability of Accuracy Estimates Obtained from Non-Probability Samples |
| title_full_unstemmed | The <i>T</i> Index: Measuring the Reliability of Accuracy Estimates Obtained from Non-Probability Samples |
| title_short | The <i>T</i> Index: Measuring the Reliability of Accuracy Estimates Obtained from Non-Probability Samples |
| title_sort | i t i index measuring the reliability of accuracy estimates obtained from non probability samples |
| topic | accuracy assessment validation classification spatial balance unlabelled data sample selection bias |
| url | https://www.mdpi.com/2072-4292/12/15/2483 |
| work_keys_str_mv | AT francoiswaldner theitiindexmeasuringthereliabilityofaccuracyestimatesobtainedfromnonprobabilitysamples AT francoiswaldner itiindexmeasuringthereliabilityofaccuracyestimatesobtainedfromnonprobabilitysamples |