Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL
Abstract Understandings of the three-dimensional social behaviors of freely moving large-size mammals are valuable for both agriculture and life science, yet challenging due to occlusions in close interactions. Although existing animal pose estimation methods captured keypoint trajectories, they ign...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2023-11-01
|
| Series: | Nature Communications |
| Online Access: | https://doi.org/10.1038/s41467-023-43483-w |
| _version_ | 1797452082499813376 |
|---|---|
| author | Liang An Jilong Ren Tao Yu Tang Hai Yichang Jia Yebin Liu |
| author_facet | Liang An Jilong Ren Tao Yu Tang Hai Yichang Jia Yebin Liu |
| author_sort | Liang An |
| collection | DOAJ |
| description | Abstract Understandings of the three-dimensional social behaviors of freely moving large-size mammals are valuable for both agriculture and life science, yet challenging due to occlusions in close interactions. Although existing animal pose estimation methods captured keypoint trajectories, they ignored deformable surfaces which contained geometric information essential for social interaction prediction and for dealing with the occlusions. In this study, we develop a Multi-Animal Mesh Model Alignment (MAMMAL) system based on an articulated surface mesh model. Our self-designed MAMMAL algorithms automatically enable us to align multi-view images into our mesh model and to capture 3D surface motions of multiple animals, which display better performance upon severe occlusions compared to traditional triangulation and allow complex social analysis. By utilizing MAMMAL, we are able to quantitatively analyze the locomotion, postures, animal-scene interactions, social interactions, as well as detailed tail motions of pigs. Furthermore, experiments on mouse and Beagle dogs demonstrate the generalizability of MAMMAL across different environments and mammal species. |
| first_indexed | 2024-03-09T15:03:37Z |
| format | Article |
| id | doaj.art-0ce42767e8804d979c1391f15beb5161 |
| institution | Directory Open Access Journal |
| issn | 2041-1723 |
| language | English |
| last_indexed | 2024-03-09T15:03:37Z |
| publishDate | 2023-11-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Nature Communications |
| spelling | doaj.art-0ce42767e8804d979c1391f15beb51612023-11-26T13:44:44ZengNature PortfolioNature Communications2041-17232023-11-0114111410.1038/s41467-023-43483-wThree-dimensional surface motion capture of multiple freely moving pigs using MAMMALLiang An0Jilong Ren1Tao Yu2Tang Hai3Yichang Jia4Yebin Liu5Department of Automation, Tsinghua UniversityState Key Laboratory of Stem Cell and Reproductive Biology, Institute of Zoology, Chinese Academy of SciencesDepartment of Automation, Tsinghua UniversityState Key Laboratory of Stem Cell and Reproductive Biology, Institute of Zoology, Chinese Academy of SciencesSchool of Medicine, Tsinghua UniversityDepartment of Automation, Tsinghua UniversityAbstract Understandings of the three-dimensional social behaviors of freely moving large-size mammals are valuable for both agriculture and life science, yet challenging due to occlusions in close interactions. Although existing animal pose estimation methods captured keypoint trajectories, they ignored deformable surfaces which contained geometric information essential for social interaction prediction and for dealing with the occlusions. In this study, we develop a Multi-Animal Mesh Model Alignment (MAMMAL) system based on an articulated surface mesh model. Our self-designed MAMMAL algorithms automatically enable us to align multi-view images into our mesh model and to capture 3D surface motions of multiple animals, which display better performance upon severe occlusions compared to traditional triangulation and allow complex social analysis. By utilizing MAMMAL, we are able to quantitatively analyze the locomotion, postures, animal-scene interactions, social interactions, as well as detailed tail motions of pigs. Furthermore, experiments on mouse and Beagle dogs demonstrate the generalizability of MAMMAL across different environments and mammal species.https://doi.org/10.1038/s41467-023-43483-w |
| spellingShingle | Liang An Jilong Ren Tao Yu Tang Hai Yichang Jia Yebin Liu Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL Nature Communications |
| title | Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL |
| title_full | Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL |
| title_fullStr | Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL |
| title_full_unstemmed | Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL |
| title_short | Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL |
| title_sort | three dimensional surface motion capture of multiple freely moving pigs using mammal |
| url | https://doi.org/10.1038/s41467-023-43483-w |
| work_keys_str_mv | AT liangan threedimensionalsurfacemotioncaptureofmultiplefreelymovingpigsusingmammal AT jilongren threedimensionalsurfacemotioncaptureofmultiplefreelymovingpigsusingmammal AT taoyu threedimensionalsurfacemotioncaptureofmultiplefreelymovingpigsusingmammal AT tanghai threedimensionalsurfacemotioncaptureofmultiplefreelymovingpigsusingmammal AT yichangjia threedimensionalsurfacemotioncaptureofmultiplefreelymovingpigsusingmammal AT yebinliu threedimensionalsurfacemotioncaptureofmultiplefreelymovingpigsusingmammal |