Learning new tricks from old dogs: Multi-source transfer learning from pre-trained networks
© 2019 Neural information processing systems foundation. All rights reserved. The advent of deep learning algorithms for mobile devices and sensors has led to a dramatic expansion in the availability and number of systems trained on a wide range of machine learning tasks, creating a host of opportun...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
2021
|
| Online Access: | https://hdl.handle.net/1721.1/137379 |
| _version_ | 1826204461184319488 |
|---|---|
| author | Lee, JKW Sattigeri, P Wornell, GW |
| author_facet | Lee, JKW Sattigeri, P Wornell, GW |
| author_sort | Lee, JKW |
| collection | MIT |
| description | © 2019 Neural information processing systems foundation. All rights reserved. The advent of deep learning algorithms for mobile devices and sensors has led to a dramatic expansion in the availability and number of systems trained on a wide range of machine learning tasks, creating a host of opportunities and challenges in the realm of transfer learning. Currently, most transfer learning methods require some kind of control over the systems learned, either by enforcing constraints during the source training, or through the use of a joint optimization objective between tasks that requires all data be co-located for training. However, for practical, privacy, or other reasons, in a variety of applications we may have no control over the individual source task training, nor access to source training samples. Instead we only have access to features pre-trained on such data as the output of “black-boxes.” For such scenarios, we consider the multi-source learning problem of training a classifier using an ensemble of pre-trained neural networks for a set of classes that have not been observed by any of the source networks, and for which we have very few training samples. We show that by using these distributed networks as feature extractors, we can train an effective classifier in a computationally-efficient manner using tools from (nonlinear) maximal correlation analysis. In particular, we develop a method we refer to as maximal correlation weighting (MCW) to build the required target classifier from an appropriate weighting of the feature functions from the source networks. We illustrate the effectiveness of the resulting classifier on datasets derived from the CIFAR-100, Stanford Dogs, and Tiny ImageNet datasets, and, in addition, use the methodology to characterize the relative value of different source tasks in learning a target task. |
| first_indexed | 2024-09-23T12:55:58Z |
| format | Article |
| id | mit-1721.1/137379 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T12:55:58Z |
| publishDate | 2021 |
| record_format | dspace |
| spelling | mit-1721.1/1373792021-11-05T03:20:45Z Learning new tricks from old dogs: Multi-source transfer learning from pre-trained networks Lee, JKW Sattigeri, P Wornell, GW © 2019 Neural information processing systems foundation. All rights reserved. The advent of deep learning algorithms for mobile devices and sensors has led to a dramatic expansion in the availability and number of systems trained on a wide range of machine learning tasks, creating a host of opportunities and challenges in the realm of transfer learning. Currently, most transfer learning methods require some kind of control over the systems learned, either by enforcing constraints during the source training, or through the use of a joint optimization objective between tasks that requires all data be co-located for training. However, for practical, privacy, or other reasons, in a variety of applications we may have no control over the individual source task training, nor access to source training samples. Instead we only have access to features pre-trained on such data as the output of “black-boxes.” For such scenarios, we consider the multi-source learning problem of training a classifier using an ensemble of pre-trained neural networks for a set of classes that have not been observed by any of the source networks, and for which we have very few training samples. We show that by using these distributed networks as feature extractors, we can train an effective classifier in a computationally-efficient manner using tools from (nonlinear) maximal correlation analysis. In particular, we develop a method we refer to as maximal correlation weighting (MCW) to build the required target classifier from an appropriate weighting of the feature functions from the source networks. We illustrate the effectiveness of the resulting classifier on datasets derived from the CIFAR-100, Stanford Dogs, and Tiny ImageNet datasets, and, in addition, use the methodology to characterize the relative value of different source tasks in learning a target task. 2021-11-04T17:21:35Z 2021-11-04T17:21:35Z 2019-12 2021-02-03T17:11:48Z Article http://purl.org/eprint/type/ConferencePaper https://hdl.handle.net/1721.1/137379 Lee, JKW, Sattigeri, P and Wornell, GW. 2019. "Learning new tricks from old dogs: Multi-source transfer learning from pre-trained networks." Advances in Neural Information Processing Systems, 32. en https://papers.nips.cc/paper/2019/hash/6048ff4e8cb07aa60b6777b6f7384d52-Abstract.html Advances in Neural Information Processing Systems Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. application/pdf Neural Information Processing Systems (NIPS) |
| spellingShingle | Lee, JKW Sattigeri, P Wornell, GW Learning new tricks from old dogs: Multi-source transfer learning from pre-trained networks |
| title | Learning new tricks from old dogs: Multi-source transfer learning from pre-trained networks |
| title_full | Learning new tricks from old dogs: Multi-source transfer learning from pre-trained networks |
| title_fullStr | Learning new tricks from old dogs: Multi-source transfer learning from pre-trained networks |
| title_full_unstemmed | Learning new tricks from old dogs: Multi-source transfer learning from pre-trained networks |
| title_short | Learning new tricks from old dogs: Multi-source transfer learning from pre-trained networks |
| title_sort | learning new tricks from old dogs multi source transfer learning from pre trained networks |
| url | https://hdl.handle.net/1721.1/137379 |
| work_keys_str_mv | AT leejkw learningnewtricksfromolddogsmultisourcetransferlearningfrompretrainednetworks AT sattigerip learningnewtricksfromolddogsmultisourcetransferlearningfrompretrainednetworks AT wornellgw learningnewtricksfromolddogsmultisourcetransferlearningfrompretrainednetworks |