Automatic landmark detection and mapping for 2D/3D registration with BoneNet
The 3D musculoskeletal motion of animals is of interest for various biological studies and can be derived from X-ray fluoroscopy acquisitions by means of image matching or manual landmark annotation and mapping. While the image matching method requires a robust similarity measure (intensity-based) o...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2022-08-01
|
| Series: | Frontiers in Veterinary Science |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fvets.2022.923449/full |
| _version_ | 1798039050479730688 |
|---|---|
| author | Van Nguyen Luis F. Alves Pereira Luis F. Alves Pereira Zhihua Liang Falk Mielke Falk Mielke Jeroen Van Houtte Jan Sijbers Jan De Beenhouwer |
| author_facet | Van Nguyen Luis F. Alves Pereira Luis F. Alves Pereira Zhihua Liang Falk Mielke Falk Mielke Jeroen Van Houtte Jan Sijbers Jan De Beenhouwer |
| author_sort | Van Nguyen |
| collection | DOAJ |
| description | The 3D musculoskeletal motion of animals is of interest for various biological studies and can be derived from X-ray fluoroscopy acquisitions by means of image matching or manual landmark annotation and mapping. While the image matching method requires a robust similarity measure (intensity-based) or an expensive computation (tomographic reconstruction-based), the manual annotation method depends on the experience of operators. In this paper, we tackle these challenges by a strategic approach that consists of two building blocks: an automated 3D landmark extraction technique and a deep neural network for 2D landmarks detection. For 3D landmark extraction, we propose a technique based on the shortest voxel coordinate variance to extract the 3D landmarks from the 3D tomographic reconstruction of an object. For 2D landmark detection, we propose a customized ResNet18-based neural network, BoneNet, to automatically detect geometrical landmarks on X-ray fluoroscopy images. With a deeper network architecture in comparison to the original ResNet18 model, BoneNet can extract and propagate feature vectors for accurate 2D landmark inference. The 3D poses of the animal are then reconstructed by aligning the extracted 2D landmarks from X-ray radiographs and the corresponding 3D landmarks in a 3D object reference model. Our proposed method is validated on X-ray images, simulated from a real piglet hindlimb 3D computed tomography scan and does not require manual annotation of landmark positions. The simulation results show that BoneNet is able to accurately detect the 2D landmarks in simulated, noisy 2D X-ray images, resulting in promising rigid and articulated parameter estimations. |
| first_indexed | 2024-04-11T21:48:34Z |
| format | Article |
| id | doaj.art-716f03ceb1f14e1bb41676bc3f3f6b61 |
| institution | Directory Open Access Journal |
| issn | 2297-1769 |
| language | English |
| last_indexed | 2024-04-11T21:48:34Z |
| publishDate | 2022-08-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Veterinary Science |
| spelling | doaj.art-716f03ceb1f14e1bb41676bc3f3f6b612022-12-22T04:01:18ZengFrontiers Media S.A.Frontiers in Veterinary Science2297-17692022-08-01910.3389/fvets.2022.923449923449Automatic landmark detection and mapping for 2D/3D registration with BoneNetVan Nguyen0Luis F. Alves Pereira1Luis F. Alves Pereira2Zhihua Liang3Falk Mielke4Falk Mielke5Jeroen Van Houtte6Jan Sijbers7Jan De Beenhouwer8Imec—Vision Lab, Department of Physics, University of Antwerp, Antwerp, BelgiumImec—Vision Lab, Department of Physics, University of Antwerp, Antwerp, BelgiumDepartamento de Ciência da Computação, Universidade Federal do Agreste de Pernambuco, Garanhuns, BrazilImec—Vision Lab, Department of Physics, University of Antwerp, Antwerp, BelgiumImec—Vision Lab, Department of Physics, University of Antwerp, Antwerp, BelgiumDepartment of Biology, University of Antwerp, Antwerp, BelgiumImec—Vision Lab, Department of Physics, University of Antwerp, Antwerp, BelgiumImec—Vision Lab, Department of Physics, University of Antwerp, Antwerp, BelgiumImec—Vision Lab, Department of Physics, University of Antwerp, Antwerp, BelgiumThe 3D musculoskeletal motion of animals is of interest for various biological studies and can be derived from X-ray fluoroscopy acquisitions by means of image matching or manual landmark annotation and mapping. While the image matching method requires a robust similarity measure (intensity-based) or an expensive computation (tomographic reconstruction-based), the manual annotation method depends on the experience of operators. In this paper, we tackle these challenges by a strategic approach that consists of two building blocks: an automated 3D landmark extraction technique and a deep neural network for 2D landmarks detection. For 3D landmark extraction, we propose a technique based on the shortest voxel coordinate variance to extract the 3D landmarks from the 3D tomographic reconstruction of an object. For 2D landmark detection, we propose a customized ResNet18-based neural network, BoneNet, to automatically detect geometrical landmarks on X-ray fluoroscopy images. With a deeper network architecture in comparison to the original ResNet18 model, BoneNet can extract and propagate feature vectors for accurate 2D landmark inference. The 3D poses of the animal are then reconstructed by aligning the extracted 2D landmarks from X-ray radiographs and the corresponding 3D landmarks in a 3D object reference model. Our proposed method is validated on X-ray images, simulated from a real piglet hindlimb 3D computed tomography scan and does not require manual annotation of landmark positions. The simulation results show that BoneNet is able to accurately detect the 2D landmarks in simulated, noisy 2D X-ray images, resulting in promising rigid and articulated parameter estimations.https://www.frontiersin.org/articles/10.3389/fvets.2022.923449/full2D/3D registrationlandmark-based registrationpose estimationautomatic landmark detectiondeep learning |
| spellingShingle | Van Nguyen Luis F. Alves Pereira Luis F. Alves Pereira Zhihua Liang Falk Mielke Falk Mielke Jeroen Van Houtte Jan Sijbers Jan De Beenhouwer Automatic landmark detection and mapping for 2D/3D registration with BoneNet Frontiers in Veterinary Science 2D/3D registration landmark-based registration pose estimation automatic landmark detection deep learning |
| title | Automatic landmark detection and mapping for 2D/3D registration with BoneNet |
| title_full | Automatic landmark detection and mapping for 2D/3D registration with BoneNet |
| title_fullStr | Automatic landmark detection and mapping for 2D/3D registration with BoneNet |
| title_full_unstemmed | Automatic landmark detection and mapping for 2D/3D registration with BoneNet |
| title_short | Automatic landmark detection and mapping for 2D/3D registration with BoneNet |
| title_sort | automatic landmark detection and mapping for 2d 3d registration with bonenet |
| topic | 2D/3D registration landmark-based registration pose estimation automatic landmark detection deep learning |
| url | https://www.frontiersin.org/articles/10.3389/fvets.2022.923449/full |
| work_keys_str_mv | AT vannguyen automaticlandmarkdetectionandmappingfor2d3dregistrationwithbonenet AT luisfalvespereira automaticlandmarkdetectionandmappingfor2d3dregistrationwithbonenet AT luisfalvespereira automaticlandmarkdetectionandmappingfor2d3dregistrationwithbonenet AT zhihualiang automaticlandmarkdetectionandmappingfor2d3dregistrationwithbonenet AT falkmielke automaticlandmarkdetectionandmappingfor2d3dregistrationwithbonenet AT falkmielke automaticlandmarkdetectionandmappingfor2d3dregistrationwithbonenet AT jeroenvanhoutte automaticlandmarkdetectionandmappingfor2d3dregistrationwithbonenet AT jansijbers automaticlandmarkdetectionandmappingfor2d3dregistrationwithbonenet AT jandebeenhouwer automaticlandmarkdetectionandmappingfor2d3dregistrationwithbonenet |