Should We Gain Confidence from the Similarity of Results between Methods?
Confirming the result of a calculation by a calculation with a different method is often seen as a validity check. However, when the methods considered are all subject to the same (systematic) errors, this practice fails. Using a statistical approach, we define measures for <i>reliability</...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-02-01
|
| Series: | Computation |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2079-3197/10/2/27 |
| _version_ | 1797481552097050624 |
|---|---|
| author | Pascal Pernot Andreas Savin |
| author_facet | Pascal Pernot Andreas Savin |
| author_sort | Pascal Pernot |
| collection | DOAJ |
| description | Confirming the result of a calculation by a calculation with a different method is often seen as a validity check. However, when the methods considered are all subject to the same (systematic) errors, this practice fails. Using a statistical approach, we define measures for <i>reliability</i> and <i>similarity</i>, and we explore the extent to which the similarity of results can help improve our judgment of the validity of data. This method is illustrated on synthetic data and applied to two benchmark datasets extracted from the literature: band gaps of solids estimated by various density functional approximations, and effective atomization energies estimated by <i>ab initio</i> and machine-learning methods. Depending on the levels of bias and correlation of the datasets, we found that similarity may provide a null-to-marginal improvement in reliability and was mostly effective in eliminating large errors. |
| first_indexed | 2024-03-09T22:16:14Z |
| format | Article |
| id | doaj.art-67d5b5f40b4848008feaef43c32e40c1 |
| institution | Directory Open Access Journal |
| issn | 2079-3197 |
| language | English |
| last_indexed | 2024-03-09T22:16:14Z |
| publishDate | 2022-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Computation |
| spelling | doaj.art-67d5b5f40b4848008feaef43c32e40c12023-11-23T19:22:31ZengMDPI AGComputation2079-31972022-02-011022710.3390/computation10020027Should We Gain Confidence from the Similarity of Results between Methods?Pascal Pernot0Andreas Savin1Institut de Chimie Physique, UMR8000, CNRS, Université Paris-Saclay, 91405 Orsay, FranceLaboratoire de Chimie Théorique, CNRS and UPMC Université Paris 06, Sorbonne Universités, 75252 Paris, FranceConfirming the result of a calculation by a calculation with a different method is often seen as a validity check. However, when the methods considered are all subject to the same (systematic) errors, this practice fails. Using a statistical approach, we define measures for <i>reliability</i> and <i>similarity</i>, and we explore the extent to which the similarity of results can help improve our judgment of the validity of data. This method is illustrated on synthetic data and applied to two benchmark datasets extracted from the literature: band gaps of solids estimated by various density functional approximations, and effective atomization energies estimated by <i>ab initio</i> and machine-learning methods. Depending on the levels of bias and correlation of the datasets, we found that similarity may provide a null-to-marginal improvement in reliability and was mostly effective in eliminating large errors.https://www.mdpi.com/2079-3197/10/2/27statisticsmethods comparisonbenchmarkingband gapsatomization energy |
| spellingShingle | Pascal Pernot Andreas Savin Should We Gain Confidence from the Similarity of Results between Methods? Computation statistics methods comparison benchmarking band gaps atomization energy |
| title | Should We Gain Confidence from the Similarity of Results between Methods? |
| title_full | Should We Gain Confidence from the Similarity of Results between Methods? |
| title_fullStr | Should We Gain Confidence from the Similarity of Results between Methods? |
| title_full_unstemmed | Should We Gain Confidence from the Similarity of Results between Methods? |
| title_short | Should We Gain Confidence from the Similarity of Results between Methods? |
| title_sort | should we gain confidence from the similarity of results between methods |
| topic | statistics methods comparison benchmarking band gaps atomization energy |
| url | https://www.mdpi.com/2079-3197/10/2/27 |
| work_keys_str_mv | AT pascalpernot shouldwegainconfidencefromthesimilarityofresultsbetweenmethods AT andreassavin shouldwegainconfidencefromthesimilarityofresultsbetweenmethods |