SNIP: single-shot network pruning based on connection sensitivity
Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, under...
| Main Authors: | , , |
|---|---|
| Format: | Conference item |
| Published: |
Open Review
2019
|
| _version_ | 1797071073172258816 |
|---|---|
| author | Lee, N Ajanthan, T Torr, P |
| author_facet | Lee, N Ajanthan, T Torr, P |
| author_sort | Lee, N |
| collection | OXFORD |
| description | Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task. |
| first_indexed | 2024-03-06T22:47:58Z |
| format | Conference item |
| id | oxford-uuid:5dd60c40-1b93-4ac2-b2a6-ea2d976a8e43 |
| institution | University of Oxford |
| last_indexed | 2024-03-06T22:47:58Z |
| publishDate | 2019 |
| publisher | Open Review |
| record_format | dspace |
| spelling | oxford-uuid:5dd60c40-1b93-4ac2-b2a6-ea2d976a8e432022-03-26T17:36:46ZSNIP: single-shot network pruning based on connection sensitivityConference itemhttp://purl.org/coar/resource_type/c_5794uuid:5dd60c40-1b93-4ac2-b2a6-ea2d976a8e43Symplectic Elements at OxfordOpen Review2019Lee, NAjanthan, TTorr, PPruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task. |
| spellingShingle | Lee, N Ajanthan, T Torr, P SNIP: single-shot network pruning based on connection sensitivity |
| title | SNIP: single-shot network pruning based on connection sensitivity |
| title_full | SNIP: single-shot network pruning based on connection sensitivity |
| title_fullStr | SNIP: single-shot network pruning based on connection sensitivity |
| title_full_unstemmed | SNIP: single-shot network pruning based on connection sensitivity |
| title_short | SNIP: single-shot network pruning based on connection sensitivity |
| title_sort | snip single shot network pruning based on connection sensitivity |
| work_keys_str_mv | AT leen snipsingleshotnetworkpruningbasedonconnectionsensitivity AT ajanthant snipsingleshotnetworkpruningbasedonconnectionsensitivity AT torrp snipsingleshotnetworkpruningbasedonconnectionsensitivity |