Mapping global dynamics of benchmark creation and saturation in artificial intelligence
Recent studies raised concerns over the state of AI benchmarking, reporting issues such as benchmark overfitting, benchmark saturation and increasing centralization of benchmark dataset creation. To facilitate monitoring of the health of the AI benchmarking ecosystem, the authors introduce methodolo...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2022-11-01
|
| Series: | Nature Communications |
| Online Access: | https://doi.org/10.1038/s41467-022-34591-0 |
| _version_ | 1798019129027854336 |
|---|---|
| author | Simon Ott Adriano Barbosa-Silva Kathrin Blagec Jan Brauner Matthias Samwald |
| author_facet | Simon Ott Adriano Barbosa-Silva Kathrin Blagec Jan Brauner Matthias Samwald |
| author_sort | Simon Ott |
| collection | DOAJ |
| description | Recent studies raised concerns over the state of AI benchmarking, reporting issues such as benchmark overfitting, benchmark saturation and increasing centralization of benchmark dataset creation. To facilitate monitoring of the health of the AI benchmarking ecosystem, the authors introduce methodologies for creating condensed maps of the global dynamics of benchmark. |
| first_indexed | 2024-04-11T16:35:42Z |
| format | Article |
| id | doaj.art-357a20a54edd4dfabde61f00141c7e58 |
| institution | Directory Open Access Journal |
| issn | 2041-1723 |
| language | English |
| last_indexed | 2024-04-11T16:35:42Z |
| publishDate | 2022-11-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Nature Communications |
| spelling | doaj.art-357a20a54edd4dfabde61f00141c7e582022-12-22T04:13:51ZengNature PortfolioNature Communications2041-17232022-11-0113111110.1038/s41467-022-34591-0Mapping global dynamics of benchmark creation and saturation in artificial intelligenceSimon Ott0Adriano Barbosa-Silva1Kathrin Blagec2Jan Brauner3Matthias Samwald4Institute of Artificial Intelligence, Medical University of Vienna. Währingerstraße 25aInstitute of Artificial Intelligence, Medical University of Vienna. Währingerstraße 25aInstitute of Artificial Intelligence, Medical University of Vienna. Währingerstraße 25aOxford Applied and Theoretical Machine Learning (OATML) Group, Department of Computer Science, University of OxfordInstitute of Artificial Intelligence, Medical University of Vienna. Währingerstraße 25aRecent studies raised concerns over the state of AI benchmarking, reporting issues such as benchmark overfitting, benchmark saturation and increasing centralization of benchmark dataset creation. To facilitate monitoring of the health of the AI benchmarking ecosystem, the authors introduce methodologies for creating condensed maps of the global dynamics of benchmark.https://doi.org/10.1038/s41467-022-34591-0 |
| spellingShingle | Simon Ott Adriano Barbosa-Silva Kathrin Blagec Jan Brauner Matthias Samwald Mapping global dynamics of benchmark creation and saturation in artificial intelligence Nature Communications |
| title | Mapping global dynamics of benchmark creation and saturation in artificial intelligence |
| title_full | Mapping global dynamics of benchmark creation and saturation in artificial intelligence |
| title_fullStr | Mapping global dynamics of benchmark creation and saturation in artificial intelligence |
| title_full_unstemmed | Mapping global dynamics of benchmark creation and saturation in artificial intelligence |
| title_short | Mapping global dynamics of benchmark creation and saturation in artificial intelligence |
| title_sort | mapping global dynamics of benchmark creation and saturation in artificial intelligence |
| url | https://doi.org/10.1038/s41467-022-34591-0 |
| work_keys_str_mv | AT simonott mappingglobaldynamicsofbenchmarkcreationandsaturationinartificialintelligence AT adrianobarbosasilva mappingglobaldynamicsofbenchmarkcreationandsaturationinartificialintelligence AT kathrinblagec mappingglobaldynamicsofbenchmarkcreationandsaturationinartificialintelligence AT janbrauner mappingglobaldynamicsofbenchmarkcreationandsaturationinartificialintelligence AT matthiassamwald mappingglobaldynamicsofbenchmarkcreationandsaturationinartificialintelligence |