DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels
Videos of animal behavior are used to quantify researcher-defined behaviors of interest to study neural function, gene mutations, and pharmacological therapies. Behaviors of interest are often scored manually, which is time-consuming, limited to few behaviors, and variable across researchers. We cre...
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
eLife Sciences Publications Ltd
2021-09-01
|
| Series: | eLife |
| Subjects: | |
| Online Access: | https://elifesciences.org/articles/63377 |
| _version_ | 1811227611202846720 |
|---|---|
| author | James P Bohnslav Nivanthika K Wimalasena Kelsey J Clausing Yu Y Dai David A Yarmolinsky Tomás Cruz Adam D Kashlan M Eugenia Chiappe Lauren L Orefice Clifford J Woolf Christopher D Harvey |
| author_facet | James P Bohnslav Nivanthika K Wimalasena Kelsey J Clausing Yu Y Dai David A Yarmolinsky Tomás Cruz Adam D Kashlan M Eugenia Chiappe Lauren L Orefice Clifford J Woolf Christopher D Harvey |
| author_sort | James P Bohnslav |
| collection | DOAJ |
| description | Videos of animal behavior are used to quantify researcher-defined behaviors of interest to study neural function, gene mutations, and pharmacological therapies. Behaviors of interest are often scored manually, which is time-consuming, limited to few behaviors, and variable across researchers. We created DeepEthogram: software that uses supervised machine learning to convert raw video pixels into an ethogram, the behaviors of interest present in each video frame. DeepEthogram is designed to be general-purpose and applicable across species, behaviors, and video-recording hardware. It uses convolutional neural networks to compute motion, extract features from motion and images, and classify features into behaviors. Behaviors are classified with above 90% accuracy on single frames in videos of mice and flies, matching expert-level human performance. DeepEthogram accurately predicts rare behaviors, requires little training data, and generalizes across subjects. A graphical interface allows beginning-to-end analysis without end-user programming. DeepEthogram’s rapid, automatic, and reproducible labeling of researcher-defined behaviors of interest may accelerate and enhance supervised behavior analysis. Code is available at: https://github.com/jbohnslav/deepethogram. |
| first_indexed | 2024-04-12T09:44:58Z |
| format | Article |
| id | doaj.art-0490bc2a85ba4831a27480dbae549b72 |
| institution | Directory Open Access Journal |
| issn | 2050-084X |
| language | English |
| last_indexed | 2024-04-12T09:44:58Z |
| publishDate | 2021-09-01 |
| publisher | eLife Sciences Publications Ltd |
| record_format | Article |
| series | eLife |
| spelling | doaj.art-0490bc2a85ba4831a27480dbae549b722022-12-22T03:37:58ZengeLife Sciences Publications LtdeLife2050-084X2021-09-011010.7554/eLife.63377DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixelsJames P Bohnslav0https://orcid.org/0000-0002-9359-8907Nivanthika K Wimalasena1Kelsey J Clausing2Yu Y Dai3David A Yarmolinsky4Tomás Cruz5Adam D Kashlan6M Eugenia Chiappe7https://orcid.org/0000-0003-1761-0457Lauren L Orefice8Clifford J Woolf9Christopher D Harvey10https://orcid.org/0000-0001-9850-2268Department of Neurobiology, Harvard Medical School, Boston, United StatesDepartment of Neurobiology, Harvard Medical School, Boston, United States; F.M. Kirby Neurobiology Center, Boston Children’s Hospital, Boston, United StatesDepartment of Molecular Biology, Massachusetts General Hospital, Boston, United States; Department of Genetics, Harvard Medical School, Boston, United StatesDepartment of Molecular Biology, Massachusetts General Hospital, Boston, United States; Department of Genetics, Harvard Medical School, Boston, United StatesDepartment of Neurobiology, Harvard Medical School, Boston, United States; F.M. Kirby Neurobiology Center, Boston Children’s Hospital, Boston, United StatesChampalimaud Neuroscience Programme, Champalimaud Center for the Unknown, Lisbon, PortugalDepartment of Neurobiology, Harvard Medical School, Boston, United States; F.M. Kirby Neurobiology Center, Boston Children’s Hospital, Boston, United StatesChampalimaud Neuroscience Programme, Champalimaud Center for the Unknown, Lisbon, PortugalDepartment of Molecular Biology, Massachusetts General Hospital, Boston, United States; Department of Genetics, Harvard Medical School, Boston, United StatesDepartment of Neurobiology, Harvard Medical School, Boston, United States; F.M. Kirby Neurobiology Center, Boston Children’s Hospital, Boston, United StatesDepartment of Neurobiology, Harvard Medical School, Boston, United StatesVideos of animal behavior are used to quantify researcher-defined behaviors of interest to study neural function, gene mutations, and pharmacological therapies. Behaviors of interest are often scored manually, which is time-consuming, limited to few behaviors, and variable across researchers. We created DeepEthogram: software that uses supervised machine learning to convert raw video pixels into an ethogram, the behaviors of interest present in each video frame. DeepEthogram is designed to be general-purpose and applicable across species, behaviors, and video-recording hardware. It uses convolutional neural networks to compute motion, extract features from motion and images, and classify features into behaviors. Behaviors are classified with above 90% accuracy on single frames in videos of mice and flies, matching expert-level human performance. DeepEthogram accurately predicts rare behaviors, requires little training data, and generalizes across subjects. A graphical interface allows beginning-to-end analysis without end-user programming. DeepEthogram’s rapid, automatic, and reproducible labeling of researcher-defined behaviors of interest may accelerate and enhance supervised behavior analysis. Code is available at: https://github.com/jbohnslav/deepethogram.https://elifesciences.org/articles/63377behavior analysisdeep learningcomputer vision |
| spellingShingle | James P Bohnslav Nivanthika K Wimalasena Kelsey J Clausing Yu Y Dai David A Yarmolinsky Tomás Cruz Adam D Kashlan M Eugenia Chiappe Lauren L Orefice Clifford J Woolf Christopher D Harvey DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels eLife behavior analysis deep learning computer vision |
| title | DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels |
| title_full | DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels |
| title_fullStr | DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels |
| title_full_unstemmed | DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels |
| title_short | DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels |
| title_sort | deepethogram a machine learning pipeline for supervised behavior classification from raw pixels |
| topic | behavior analysis deep learning computer vision |
| url | https://elifesciences.org/articles/63377 |
| work_keys_str_mv | AT jamespbohnslav deepethogramamachinelearningpipelineforsupervisedbehaviorclassificationfromrawpixels AT nivanthikakwimalasena deepethogramamachinelearningpipelineforsupervisedbehaviorclassificationfromrawpixels AT kelseyjclausing deepethogramamachinelearningpipelineforsupervisedbehaviorclassificationfromrawpixels AT yuydai deepethogramamachinelearningpipelineforsupervisedbehaviorclassificationfromrawpixels AT davidayarmolinsky deepethogramamachinelearningpipelineforsupervisedbehaviorclassificationfromrawpixels AT tomascruz deepethogramamachinelearningpipelineforsupervisedbehaviorclassificationfromrawpixels AT adamdkashlan deepethogramamachinelearningpipelineforsupervisedbehaviorclassificationfromrawpixels AT meugeniachiappe deepethogramamachinelearningpipelineforsupervisedbehaviorclassificationfromrawpixels AT laurenlorefice deepethogramamachinelearningpipelineforsupervisedbehaviorclassificationfromrawpixels AT cliffordjwoolf deepethogramamachinelearningpipelineforsupervisedbehaviorclassificationfromrawpixels AT christopherdharvey deepethogramamachinelearningpipelineforsupervisedbehaviorclassificationfromrawpixels |