Challenges of machine learning model validation using correlated behaviour data: Evaluation of cross-validation strategies and accuracy measures.
Automated monitoring of the movements and behaviour of animals is a valuable research tool. Recently, machine learning tools were applied to many species to classify units of behaviour. For the monitoring of wild species, collecting enough data for training models might be problematic, thus we exami...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2020-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://doi.org/10.1371/journal.pone.0236092 |
| _version_ | 1819016283408039936 |
|---|---|
| author | Bence Ferdinandy Linda Gerencsér Luca Corrieri Paula Perez Dóra Újváry Gábor Csizmadia Ádám Miklósi |
| author_facet | Bence Ferdinandy Linda Gerencsér Luca Corrieri Paula Perez Dóra Újváry Gábor Csizmadia Ádám Miklósi |
| author_sort | Bence Ferdinandy |
| collection | DOAJ |
| description | Automated monitoring of the movements and behaviour of animals is a valuable research tool. Recently, machine learning tools were applied to many species to classify units of behaviour. For the monitoring of wild species, collecting enough data for training models might be problematic, thus we examine how machine learning models trained on one species can be applied to another closely related species with similar behavioural conformation. We contrast two ways to calculate accuracies, termed here as overall and threshold accuracy, because the field has yet to define solid standards for reporting and measuring classification performances. We measure 21 dogs and 7 wolves, and find that overall accuracies are between 51 and 60% for classifying 8 behaviours (lay, sit, stand, walk, trot, run, eat, drink) when training and testing data are from the same species and between 41 and 51% when training and testing is cross-species. We show that using data from dogs to predict the behaviour of wolves is feasible. We also show that optimising the model for overall accuracy leads to similar overall and threshold accuracies, while optimizing for threshold accuracy leads to threshold accuracies well above 80%, but yielding very low overall accuracies, often below the chance level. Moreover, we show that the most common method for dividing the data between training and testing data (random selection of test data) overestimates the accuracy of models when applied to data of new specimens. Consequently, we argue that for the most common goals of animal behaviour recognition overall accuracy should be the preferred metric. Considering, that often the goal is to collect movement data without other methods of observation, we argue that training data and testing data should be divided by individual and not randomly. |
| first_indexed | 2024-12-21T02:45:10Z |
| format | Article |
| id | doaj.art-bc7214afd8a34cd5b84e996b6b666be2 |
| institution | Directory Open Access Journal |
| issn | 1932-6203 |
| language | English |
| last_indexed | 2024-12-21T02:45:10Z |
| publishDate | 2020-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj.art-bc7214afd8a34cd5b84e996b6b666be22022-12-21T19:18:34ZengPublic Library of Science (PLoS)PLoS ONE1932-62032020-01-01157e023609210.1371/journal.pone.0236092Challenges of machine learning model validation using correlated behaviour data: Evaluation of cross-validation strategies and accuracy measures.Bence FerdinandyLinda GerencsérLuca CorrieriPaula PerezDóra ÚjváryGábor CsizmadiaÁdám MiklósiAutomated monitoring of the movements and behaviour of animals is a valuable research tool. Recently, machine learning tools were applied to many species to classify units of behaviour. For the monitoring of wild species, collecting enough data for training models might be problematic, thus we examine how machine learning models trained on one species can be applied to another closely related species with similar behavioural conformation. We contrast two ways to calculate accuracies, termed here as overall and threshold accuracy, because the field has yet to define solid standards for reporting and measuring classification performances. We measure 21 dogs and 7 wolves, and find that overall accuracies are between 51 and 60% for classifying 8 behaviours (lay, sit, stand, walk, trot, run, eat, drink) when training and testing data are from the same species and between 41 and 51% when training and testing is cross-species. We show that using data from dogs to predict the behaviour of wolves is feasible. We also show that optimising the model for overall accuracy leads to similar overall and threshold accuracies, while optimizing for threshold accuracy leads to threshold accuracies well above 80%, but yielding very low overall accuracies, often below the chance level. Moreover, we show that the most common method for dividing the data between training and testing data (random selection of test data) overestimates the accuracy of models when applied to data of new specimens. Consequently, we argue that for the most common goals of animal behaviour recognition overall accuracy should be the preferred metric. Considering, that often the goal is to collect movement data without other methods of observation, we argue that training data and testing data should be divided by individual and not randomly.https://doi.org/10.1371/journal.pone.0236092 |
| spellingShingle | Bence Ferdinandy Linda Gerencsér Luca Corrieri Paula Perez Dóra Újváry Gábor Csizmadia Ádám Miklósi Challenges of machine learning model validation using correlated behaviour data: Evaluation of cross-validation strategies and accuracy measures. PLoS ONE |
| title | Challenges of machine learning model validation using correlated behaviour data: Evaluation of cross-validation strategies and accuracy measures. |
| title_full | Challenges of machine learning model validation using correlated behaviour data: Evaluation of cross-validation strategies and accuracy measures. |
| title_fullStr | Challenges of machine learning model validation using correlated behaviour data: Evaluation of cross-validation strategies and accuracy measures. |
| title_full_unstemmed | Challenges of machine learning model validation using correlated behaviour data: Evaluation of cross-validation strategies and accuracy measures. |
| title_short | Challenges of machine learning model validation using correlated behaviour data: Evaluation of cross-validation strategies and accuracy measures. |
| title_sort | challenges of machine learning model validation using correlated behaviour data evaluation of cross validation strategies and accuracy measures |
| url | https://doi.org/10.1371/journal.pone.0236092 |
| work_keys_str_mv | AT benceferdinandy challengesofmachinelearningmodelvalidationusingcorrelatedbehaviourdataevaluationofcrossvalidationstrategiesandaccuracymeasures AT lindagerencser challengesofmachinelearningmodelvalidationusingcorrelatedbehaviourdataevaluationofcrossvalidationstrategiesandaccuracymeasures AT lucacorrieri challengesofmachinelearningmodelvalidationusingcorrelatedbehaviourdataevaluationofcrossvalidationstrategiesandaccuracymeasures AT paulaperez challengesofmachinelearningmodelvalidationusingcorrelatedbehaviourdataevaluationofcrossvalidationstrategiesandaccuracymeasures AT doraujvary challengesofmachinelearningmodelvalidationusingcorrelatedbehaviourdataevaluationofcrossvalidationstrategiesandaccuracymeasures AT gaborcsizmadia challengesofmachinelearningmodelvalidationusingcorrelatedbehaviourdataevaluationofcrossvalidationstrategiesandaccuracymeasures AT adammiklosi challengesofmachinelearningmodelvalidationusingcorrelatedbehaviourdataevaluationofcrossvalidationstrategiesandaccuracymeasures |