| Summary: | Network pruning is an effective methodology to compress large neural networks,
and sparse neural networks obtained by pruning can benefit from their reduced
memory and computational costs at use. Notably, recent advances have found that
it is possible to find a trainable sparse neural network even at random initialization
prior to training; hence the obtained sparse network only needs to be trained.
While this approach of pruning at initialization turned out to be highly effective,
little has been studied about the training aspects of these sparse neural networks.
In this work, we focus on measuring the effects of data parallelism on training
sparse neural networks. As a result, we find that the data parallelism in training
sparse neural networks is no worse than that in training densely parameterized
neural networks, despite the general difficulty of training sparse neural networks.
When training sparse networks using SGD with momentum, the breakdown of the
perfect scaling regime occurs even much later than the dense at large batch sizes.
|
|---|