DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila
Studying how neural circuits orchestrate limbed behaviors requires the precise measurement of the positions of each appendage in three-dimensional (3D) space. Deep neural networks can estimate two-dimensional (2D) pose in freely behaving and tethered animals. However, the unique challenges associate...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
eLife Sciences Publications Ltd
2019-10-01
|
| Series: | eLife |
| Subjects: | |
| Online Access: | https://elifesciences.org/articles/48571 |
| _version_ | 1829096895878791168 |
|---|---|
| author | Semih Günel Helge Rhodin Daniel Morales João Campagnolo Pavan Ramdya Pascal Fua |
| author_facet | Semih Günel Helge Rhodin Daniel Morales João Campagnolo Pavan Ramdya Pascal Fua |
| author_sort | Semih Günel |
| collection | DOAJ |
| description | Studying how neural circuits orchestrate limbed behaviors requires the precise measurement of the positions of each appendage in three-dimensional (3D) space. Deep neural networks can estimate two-dimensional (2D) pose in freely behaving and tethered animals. However, the unique challenges associated with transforming these 2D measurements into reliable and precise 3D poses have not been addressed for small animals including the fly, Drosophila melanogaster. Here, we present DeepFly3D, a software that infers the 3D pose of tethered, adult Drosophila using multiple camera images. DeepFly3D does not require manual calibration, uses pictorial structures to automatically detect and correct pose estimation errors, and uses active learning to iteratively improve performance. We demonstrate more accurate unsupervised behavioral embedding using 3D joint angles rather than commonly used 2D pose data. Thus, DeepFly3D enables the automated acquisition of Drosophila behavioral measurements at an unprecedented level of detail for a variety of biological applications. |
| first_indexed | 2024-04-11T09:05:05Z |
| format | Article |
| id | doaj.art-907e62ec3678464d8e70da61e5a6f446 |
| institution | Directory Open Access Journal |
| issn | 2050-084X |
| language | English |
| last_indexed | 2024-04-11T09:05:05Z |
| publishDate | 2019-10-01 |
| publisher | eLife Sciences Publications Ltd |
| record_format | Article |
| series | eLife |
| spelling | doaj.art-907e62ec3678464d8e70da61e5a6f4462022-12-22T04:32:40ZengeLife Sciences Publications LtdeLife2050-084X2019-10-01810.7554/eLife.48571DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult DrosophilaSemih Günel0Helge Rhodin1https://orcid.org/0000-0003-2692-0801Daniel Morales2https://orcid.org/0000-0002-7469-0898João Campagnolo3Pavan Ramdya4https://orcid.org/0000-0001-5425-4610Pascal Fua5Computer Vision Laboratory, School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland; Neuroengineering Laboratory, Brain Mind Institute & Interfaculty Institute of Bioengineering, School of Life Sciences, EPFL, Lausanne, SwitzerlandComputer Vision Laboratory, School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland; Department of Computer Science, UBC, Vancouver, CanadaNeuroengineering Laboratory, Brain Mind Institute & Interfaculty Institute of Bioengineering, School of Life Sciences, EPFL, Lausanne, SwitzerlandNeuroengineering Laboratory, Brain Mind Institute & Interfaculty Institute of Bioengineering, School of Life Sciences, EPFL, Lausanne, SwitzerlandNeuroengineering Laboratory, Brain Mind Institute & Interfaculty Institute of Bioengineering, School of Life Sciences, EPFL, Lausanne, SwitzerlandComputer Vision Laboratory, School of Computer and Communication Sciences, EPFL, Lausanne, SwitzerlandStudying how neural circuits orchestrate limbed behaviors requires the precise measurement of the positions of each appendage in three-dimensional (3D) space. Deep neural networks can estimate two-dimensional (2D) pose in freely behaving and tethered animals. However, the unique challenges associated with transforming these 2D measurements into reliable and precise 3D poses have not been addressed for small animals including the fly, Drosophila melanogaster. Here, we present DeepFly3D, a software that infers the 3D pose of tethered, adult Drosophila using multiple camera images. DeepFly3D does not require manual calibration, uses pictorial structures to automatically detect and correct pose estimation errors, and uses active learning to iteratively improve performance. We demonstrate more accurate unsupervised behavioral embedding using 3D joint angles rather than commonly used 2D pose data. Thus, DeepFly3D enables the automated acquisition of Drosophila behavioral measurements at an unprecedented level of detail for a variety of biological applications.https://elifesciences.org/articles/485713D pose estimationanimal behaviordeep learningcomputer visionunsupervised classification |
| spellingShingle | Semih Günel Helge Rhodin Daniel Morales João Campagnolo Pavan Ramdya Pascal Fua DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila eLife 3D pose estimation animal behavior deep learning computer vision unsupervised classification |
| title | DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila |
| title_full | DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila |
| title_fullStr | DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila |
| title_full_unstemmed | DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila |
| title_short | DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila |
| title_sort | deepfly3d a deep learning based approach for 3d limb and appendage tracking in tethered adult drosophila |
| topic | 3D pose estimation animal behavior deep learning computer vision unsupervised classification |
| url | https://elifesciences.org/articles/48571 |
| work_keys_str_mv | AT semihgunel deepfly3dadeeplearningbasedapproachfor3dlimbandappendagetrackingintetheredadultdrosophila AT helgerhodin deepfly3dadeeplearningbasedapproachfor3dlimbandappendagetrackingintetheredadultdrosophila AT danielmorales deepfly3dadeeplearningbasedapproachfor3dlimbandappendagetrackingintetheredadultdrosophila AT joaocampagnolo deepfly3dadeeplearningbasedapproachfor3dlimbandappendagetrackingintetheredadultdrosophila AT pavanramdya deepfly3dadeeplearningbasedapproachfor3dlimbandappendagetrackingintetheredadultdrosophila AT pascalfua deepfly3dadeeplearningbasedapproachfor3dlimbandappendagetrackingintetheredadultdrosophila |