Multi-animal pose estimation, identification and tracking with DeepLabCut
<jats:title>Abstract</jats:title><jats:p>Estimating the pose of multiple animals is a challenging computer vision problem: frequent interactions cause occlusions and complicate the association of detected keypoints to the correct individuals, as well as having highly similar lookin...
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Springer Science and Business Media LLC
2023
|
| Online Access: | https://hdl.handle.net/1721.1/148780 |
| _version_ | 1826202012097708032 |
|---|---|
| author | Lauer, Jessy Zhou, Mu Ye, Shaokai Menegas, William Schneider, Steffen Nath, Tanmay Rahman, Mohammed Mostafizur Di Santo, Valentina Soberanes, Daniel Feng, Guoping Murthy, Venkatesh N Lauder, George Dulac, Catherine Mathis, Mackenzie Weygandt Mathis, Alexander |
| author2 | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences |
| author_facet | Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences Lauer, Jessy Zhou, Mu Ye, Shaokai Menegas, William Schneider, Steffen Nath, Tanmay Rahman, Mohammed Mostafizur Di Santo, Valentina Soberanes, Daniel Feng, Guoping Murthy, Venkatesh N Lauder, George Dulac, Catherine Mathis, Mackenzie Weygandt Mathis, Alexander |
| author_sort | Lauer, Jessy |
| collection | MIT |
| description | <jats:title>Abstract</jats:title><jats:p>Estimating the pose of multiple animals is a challenging computer vision problem: frequent interactions cause occlusions and complicate the association of detected keypoints to the correct individuals, as well as having highly similar looking animals that interact more closely than in typical multi-human scenarios. To take up this challenge, we build on DeepLabCut, an open-source pose estimation toolbox, and provide high-performance animal assembly and tracking—features required for multi-animal scenarios. Furthermore, we integrate the ability to predict an animal’s identity to assist tracking (in case of occlusions). We illustrate the power of this framework with four datasets varying in complexity, which we release to serve as a benchmark for future algorithm development.</jats:p> |
| first_indexed | 2024-09-23T12:00:45Z |
| format | Article |
| id | mit-1721.1/148780 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T12:00:45Z |
| publishDate | 2023 |
| publisher | Springer Science and Business Media LLC |
| record_format | dspace |
| spelling | mit-1721.1/1487802023-03-28T03:28:17Z Multi-animal pose estimation, identification and tracking with DeepLabCut Lauer, Jessy Zhou, Mu Ye, Shaokai Menegas, William Schneider, Steffen Nath, Tanmay Rahman, Mohammed Mostafizur Di Santo, Valentina Soberanes, Daniel Feng, Guoping Murthy, Venkatesh N Lauder, George Dulac, Catherine Mathis, Mackenzie Weygandt Mathis, Alexander Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences <jats:title>Abstract</jats:title><jats:p>Estimating the pose of multiple animals is a challenging computer vision problem: frequent interactions cause occlusions and complicate the association of detected keypoints to the correct individuals, as well as having highly similar looking animals that interact more closely than in typical multi-human scenarios. To take up this challenge, we build on DeepLabCut, an open-source pose estimation toolbox, and provide high-performance animal assembly and tracking—features required for multi-animal scenarios. Furthermore, we integrate the ability to predict an animal’s identity to assist tracking (in case of occlusions). We illustrate the power of this framework with four datasets varying in complexity, which we release to serve as a benchmark for future algorithm development.</jats:p> 2023-03-27T14:26:22Z 2023-03-27T14:26:22Z 2022 2023-03-27T14:17:44Z Article http://purl.org/eprint/type/JournalArticle https://hdl.handle.net/1721.1/148780 Lauer, Jessy, Zhou, Mu, Ye, Shaokai, Menegas, William, Schneider, Steffen et al. 2022. "Multi-animal pose estimation, identification and tracking with DeepLabCut." Nature Methods, 19 (4). en 10.1038/S41592-022-01443-0 Nature Methods Creative Commons Attribution 4.0 International license https://creativecommons.org/licenses/by/4.0/ application/pdf Springer Science and Business Media LLC Nature |
| spellingShingle | Lauer, Jessy Zhou, Mu Ye, Shaokai Menegas, William Schneider, Steffen Nath, Tanmay Rahman, Mohammed Mostafizur Di Santo, Valentina Soberanes, Daniel Feng, Guoping Murthy, Venkatesh N Lauder, George Dulac, Catherine Mathis, Mackenzie Weygandt Mathis, Alexander Multi-animal pose estimation, identification and tracking with DeepLabCut |
| title | Multi-animal pose estimation, identification and tracking with DeepLabCut |
| title_full | Multi-animal pose estimation, identification and tracking with DeepLabCut |
| title_fullStr | Multi-animal pose estimation, identification and tracking with DeepLabCut |
| title_full_unstemmed | Multi-animal pose estimation, identification and tracking with DeepLabCut |
| title_short | Multi-animal pose estimation, identification and tracking with DeepLabCut |
| title_sort | multi animal pose estimation identification and tracking with deeplabcut |
| url | https://hdl.handle.net/1721.1/148780 |
| work_keys_str_mv | AT lauerjessy multianimalposeestimationidentificationandtrackingwithdeeplabcut AT zhoumu multianimalposeestimationidentificationandtrackingwithdeeplabcut AT yeshaokai multianimalposeestimationidentificationandtrackingwithdeeplabcut AT menegaswilliam multianimalposeestimationidentificationandtrackingwithdeeplabcut AT schneidersteffen multianimalposeestimationidentificationandtrackingwithdeeplabcut AT nathtanmay multianimalposeestimationidentificationandtrackingwithdeeplabcut AT rahmanmohammedmostafizur multianimalposeestimationidentificationandtrackingwithdeeplabcut AT disantovalentina multianimalposeestimationidentificationandtrackingwithdeeplabcut AT soberanesdaniel multianimalposeestimationidentificationandtrackingwithdeeplabcut AT fengguoping multianimalposeestimationidentificationandtrackingwithdeeplabcut AT murthyvenkateshn multianimalposeestimationidentificationandtrackingwithdeeplabcut AT laudergeorge multianimalposeestimationidentificationandtrackingwithdeeplabcut AT dulaccatherine multianimalposeestimationidentificationandtrackingwithdeeplabcut AT mathismackenzieweygandt multianimalposeestimationidentificationandtrackingwithdeeplabcut AT mathisalexander multianimalposeestimationidentificationandtrackingwithdeeplabcut |