Distilling knowledge from a deep pose regressor network
This paper presents a novel method to distill knowledge from a deep pose regressor network for efficient Visual Odometry (VO). Standard distillation relies on ''dark knowledge'' for successful knowledge transfer. As this knowledge is not available in pose regression and the teach...
| Main Authors: | , , , , |
|---|---|
| Format: | Conference item |
| Language: | English |
| Published: |
IEEE
2020
|
| _version_ | 1797079942924599296 |
|---|---|
| author | Saputra, M De Gusmao, P Almalioglu, Y Markham, A Trigoni, A |
| author_facet | Saputra, M De Gusmao, P Almalioglu, Y Markham, A Trigoni, A |
| author_sort | Saputra, M |
| collection | OXFORD |
| description | This paper presents a novel method to distill knowledge from a deep pose regressor network for efficient Visual Odometry (VO). Standard distillation relies on ''dark knowledge'' for successful knowledge transfer. As this knowledge is not available in pose regression and the teacher prediction is not always accurate, we propose to emphasize the knowledge transfer only when we trust the teacher. We achieve this by using teacher loss as a confidence score which places variable relative importance on the teacher prediction. We inject this confidence score to the main training task via Attentive Imitation Loss (AIL) and when learning the intermediate representation of the teacher through Attentive Hint Training (AHT) approach. To the best of our knowledge, this is the first work which successfully distill the knowledge from a deep pose regression network. Our evaluation on the KITTI and Malaga dataset shows that we can keep the student prediction close to the teacher with up to 92.95% parameter reduction and 2.12x faster in computation time. |
| first_indexed | 2024-03-07T00:53:06Z |
| format | Conference item |
| id | oxford-uuid:87126127-8def-42ae-a891-44b9be4e623b |
| institution | University of Oxford |
| language | English |
| last_indexed | 2024-03-07T00:53:06Z |
| publishDate | 2020 |
| publisher | IEEE |
| record_format | dspace |
| spelling | oxford-uuid:87126127-8def-42ae-a891-44b9be4e623b2022-03-26T22:08:18ZDistilling knowledge from a deep pose regressor networkConference itemhttp://purl.org/coar/resource_type/c_5794uuid:87126127-8def-42ae-a891-44b9be4e623bEnglishSymplectic Elements at OxfordIEEE2020Saputra, MDe Gusmao, PAlmalioglu, YMarkham, ATrigoni, AThis paper presents a novel method to distill knowledge from a deep pose regressor network for efficient Visual Odometry (VO). Standard distillation relies on ''dark knowledge'' for successful knowledge transfer. As this knowledge is not available in pose regression and the teacher prediction is not always accurate, we propose to emphasize the knowledge transfer only when we trust the teacher. We achieve this by using teacher loss as a confidence score which places variable relative importance on the teacher prediction. We inject this confidence score to the main training task via Attentive Imitation Loss (AIL) and when learning the intermediate representation of the teacher through Attentive Hint Training (AHT) approach. To the best of our knowledge, this is the first work which successfully distill the knowledge from a deep pose regression network. Our evaluation on the KITTI and Malaga dataset shows that we can keep the student prediction close to the teacher with up to 92.95% parameter reduction and 2.12x faster in computation time. |
| spellingShingle | Saputra, M De Gusmao, P Almalioglu, Y Markham, A Trigoni, A Distilling knowledge from a deep pose regressor network |
| title | Distilling knowledge from a deep pose regressor network |
| title_full | Distilling knowledge from a deep pose regressor network |
| title_fullStr | Distilling knowledge from a deep pose regressor network |
| title_full_unstemmed | Distilling knowledge from a deep pose regressor network |
| title_short | Distilling knowledge from a deep pose regressor network |
| title_sort | distilling knowledge from a deep pose regressor network |
| work_keys_str_mv | AT saputram distillingknowledgefromadeepposeregressornetwork AT degusmaop distillingknowledgefromadeepposeregressornetwork AT almaliogluy distillingknowledgefromadeepposeregressornetwork AT markhama distillingknowledgefromadeepposeregressornetwork AT trigonia distillingknowledgefromadeepposeregressornetwork |