Rapid automated 3-D pose estimation of larval zebrafish using a physical model-trained neural network.
Quantitative ethology requires an accurate estimation of an organism's postural dynamics in three dimensions plus time. Technological progress over the last decade has made animal pose estimation in challenging scenarios possible with unprecedented detail. Here, we present (i) a fast automated...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2023-10-01
|
| Series: | PLoS Computational Biology |
| Online Access: | https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1011566&type=printable |
| _version_ | 1797633315811885056 |
|---|---|
| author | Aniket Ravan Ruopei Feng Martin Gruebele Yann R Chemla |
| author_facet | Aniket Ravan Ruopei Feng Martin Gruebele Yann R Chemla |
| author_sort | Aniket Ravan |
| collection | DOAJ |
| description | Quantitative ethology requires an accurate estimation of an organism's postural dynamics in three dimensions plus time. Technological progress over the last decade has made animal pose estimation in challenging scenarios possible with unprecedented detail. Here, we present (i) a fast automated method to record and track the pose of individual larval zebrafish in a 3-D environment, applicable when accurate human labeling is not possible; (ii) a rich annotated dataset of 3-D larval poses for ethologists and the general zebrafish and machine learning community; and (iii) a technique to generate realistic, annotated larval images in different behavioral contexts. Using a three-camera system calibrated with refraction correction, we record diverse larval swims under free swimming conditions and in response to acoustic and optical stimuli. We then employ a convolutional neural network to estimate 3-D larval poses from video images. The network is trained against a set of synthetic larval images rendered using a 3-D physical model of larvae. This 3-D model samples from a distribution of realistic larval poses that we estimate a priori using a template-based pose estimation of a small number of swim bouts. Our network model, trained without any human annotation, performs larval pose estimation three orders of magnitude faster and with accuracy comparable to the template-based approach, capturing detailed kinematics of 3-D larval swims. It also applies accurately to other datasets collected under different imaging conditions and containing behavioral contexts not included in our training. |
| first_indexed | 2024-03-11T11:52:23Z |
| format | Article |
| id | doaj.art-e6abd2a3b56b4bcbbba69953e6e3c07f |
| institution | Directory Open Access Journal |
| issn | 1553-734X 1553-7358 |
| language | English |
| last_indexed | 2024-03-11T11:52:23Z |
| publishDate | 2023-10-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS Computational Biology |
| spelling | doaj.art-e6abd2a3b56b4bcbbba69953e6e3c07f2023-11-09T05:31:22ZengPublic Library of Science (PLoS)PLoS Computational Biology1553-734X1553-73582023-10-011910e101156610.1371/journal.pcbi.1011566Rapid automated 3-D pose estimation of larval zebrafish using a physical model-trained neural network.Aniket RavanRuopei FengMartin GruebeleYann R ChemlaQuantitative ethology requires an accurate estimation of an organism's postural dynamics in three dimensions plus time. Technological progress over the last decade has made animal pose estimation in challenging scenarios possible with unprecedented detail. Here, we present (i) a fast automated method to record and track the pose of individual larval zebrafish in a 3-D environment, applicable when accurate human labeling is not possible; (ii) a rich annotated dataset of 3-D larval poses for ethologists and the general zebrafish and machine learning community; and (iii) a technique to generate realistic, annotated larval images in different behavioral contexts. Using a three-camera system calibrated with refraction correction, we record diverse larval swims under free swimming conditions and in response to acoustic and optical stimuli. We then employ a convolutional neural network to estimate 3-D larval poses from video images. The network is trained against a set of synthetic larval images rendered using a 3-D physical model of larvae. This 3-D model samples from a distribution of realistic larval poses that we estimate a priori using a template-based pose estimation of a small number of swim bouts. Our network model, trained without any human annotation, performs larval pose estimation three orders of magnitude faster and with accuracy comparable to the template-based approach, capturing detailed kinematics of 3-D larval swims. It also applies accurately to other datasets collected under different imaging conditions and containing behavioral contexts not included in our training.https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1011566&type=printable |
| spellingShingle | Aniket Ravan Ruopei Feng Martin Gruebele Yann R Chemla Rapid automated 3-D pose estimation of larval zebrafish using a physical model-trained neural network. PLoS Computational Biology |
| title | Rapid automated 3-D pose estimation of larval zebrafish using a physical model-trained neural network. |
| title_full | Rapid automated 3-D pose estimation of larval zebrafish using a physical model-trained neural network. |
| title_fullStr | Rapid automated 3-D pose estimation of larval zebrafish using a physical model-trained neural network. |
| title_full_unstemmed | Rapid automated 3-D pose estimation of larval zebrafish using a physical model-trained neural network. |
| title_short | Rapid automated 3-D pose estimation of larval zebrafish using a physical model-trained neural network. |
| title_sort | rapid automated 3 d pose estimation of larval zebrafish using a physical model trained neural network |
| url | https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1011566&type=printable |
| work_keys_str_mv | AT aniketravan rapidautomated3dposeestimationoflarvalzebrafishusingaphysicalmodeltrainedneuralnetwork AT ruopeifeng rapidautomated3dposeestimationoflarvalzebrafishusingaphysicalmodeltrainedneuralnetwork AT martingruebele rapidautomated3dposeestimationoflarvalzebrafishusingaphysicalmodeltrainedneuralnetwork AT yannrchemla rapidautomated3dposeestimationoflarvalzebrafishusingaphysicalmodeltrainedneuralnetwork |