Multimodal-GuideNet: gaze-probe bidirectional guidance in obstetric ultrasound scanning
Eye trackers can provide visual guidance to sonographers during ultrasound (US) scanning. Such guidance is potentially valuable for less experienced operators to improve their scanning skills on how to manipulate the probe to achieve the desired plane. In this paper, a multimodal guidance approach (...
| Main Authors: | , , , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
Springer
2022
|
| _version_ | 1797109539264266240 |
|---|---|
| author | Men, Q Teng, C Drukker, L Papageorghiou, AT Noble, JA |
| author_facet | Men, Q Teng, C Drukker, L Papageorghiou, AT Noble, JA |
| author_sort | Men, Q |
| collection | OXFORD |
| description | Eye trackers can provide visual guidance to sonographers during ultrasound (US) scanning. Such guidance is potentially valuable for less experienced operators to improve their scanning skills on how to manipulate the probe to achieve the desired plane. In this paper, a multimodal guidance approach (Multimodal-GuideNet) is proposed to capture the stepwise dependency between a real-world US video signal, synchronized gaze, and probe motion within a unified framework. To understand the causal relationship between gaze movement and probe motion, our model exploits multitask learning to jointly learn two related tasks: predicting gaze movements and probe signals that an experienced sonographer would perform in routine obstetric scanning. The two tasks are associated by a modality-aware spatial graph to detect the co-occurrence among the multi-modality inputs and share useful cross-modal information. Instead of a deterministic scanning path, Multimodal-GuideNet allows for scanning diversity by estimating the probability distribution of real scans. Experiments performed with three typical obstetric scanning examinations show that the new approach outperforms single-task learning for both probe motion guidance and gaze movement prediction. Multimodal-GuideNet also provides a visual guidance signal with an error rate of less than 10 pixels for a 224 × 288 US image. |
| first_indexed | 2024-03-07T07:41:37Z |
| format | Conference item |
| id | oxford-uuid:e33a26aa-a949-4afa-bd3a-20e3f740ee55 |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T07:41:37Z |
| publishDate | 2022 |
| publisher | Springer |
| record_format | dspace |
| spelling | oxford-uuid:e33a26aa-a949-4afa-bd3a-20e3f740ee552023-04-28T11:02:38ZMultimodal-GuideNet: gaze-probe bidirectional guidance in obstetric ultrasound scanningConference itemhttp://purl.org/coar/resource_type/c_5794uuid:e33a26aa-a949-4afa-bd3a-20e3f740ee55EnglishSymplectic ElementsSpringer2022Men, QTeng, CDrukker, LPapageorghiou, ATNoble, JAEye trackers can provide visual guidance to sonographers during ultrasound (US) scanning. Such guidance is potentially valuable for less experienced operators to improve their scanning skills on how to manipulate the probe to achieve the desired plane. In this paper, a multimodal guidance approach (Multimodal-GuideNet) is proposed to capture the stepwise dependency between a real-world US video signal, synchronized gaze, and probe motion within a unified framework. To understand the causal relationship between gaze movement and probe motion, our model exploits multitask learning to jointly learn two related tasks: predicting gaze movements and probe signals that an experienced sonographer would perform in routine obstetric scanning. The two tasks are associated by a modality-aware spatial graph to detect the co-occurrence among the multi-modality inputs and share useful cross-modal information. Instead of a deterministic scanning path, Multimodal-GuideNet allows for scanning diversity by estimating the probability distribution of real scans. Experiments performed with three typical obstetric scanning examinations show that the new approach outperforms single-task learning for both probe motion guidance and gaze movement prediction. Multimodal-GuideNet also provides a visual guidance signal with an error rate of less than 10 pixels for a 224 × 288 US image. |
| spellingShingle | Men, Q Teng, C Drukker, L Papageorghiou, AT Noble, JA Multimodal-GuideNet: gaze-probe bidirectional guidance in obstetric ultrasound scanning |
| title | Multimodal-GuideNet: gaze-probe bidirectional guidance in obstetric ultrasound scanning |
| title_full | Multimodal-GuideNet: gaze-probe bidirectional guidance in obstetric ultrasound scanning |
| title_fullStr | Multimodal-GuideNet: gaze-probe bidirectional guidance in obstetric ultrasound scanning |
| title_full_unstemmed | Multimodal-GuideNet: gaze-probe bidirectional guidance in obstetric ultrasound scanning |
| title_short | Multimodal-GuideNet: gaze-probe bidirectional guidance in obstetric ultrasound scanning |
| title_sort | multimodal guidenet gaze probe bidirectional guidance in obstetric ultrasound scanning |
| work_keys_str_mv | AT menq multimodalguidenetgazeprobebidirectionalguidanceinobstetricultrasoundscanning AT tengc multimodalguidenetgazeprobebidirectionalguidanceinobstetricultrasoundscanning AT drukkerl multimodalguidenetgazeprobebidirectionalguidanceinobstetricultrasoundscanning AT papageorghiouat multimodalguidenetgazeprobebidirectionalguidanceinobstetricultrasoundscanning AT nobleja multimodalguidenetgazeprobebidirectionalguidanceinobstetricultrasoundscanning |