LSOP: Layer-Scaled One-shot Pruning
A Simple and Effective Deep Pruning Framework for Neural Networks
DOI:
https://doi.org/10.47611/jsrhs.v11i1.2587Keywords:
LSOP, Pruning, Convolutional Neural NetworksAbstract
Neural network pruning is a technique that removes unnecessary weight parameters from a network to decrease its memory and computational requirements. Many different pruning techniques have been proposed to reduce networks with over 90% shrinkage in size while minimizing accuracy loss. This paper aims to establish a framework that can generalize the mechanism among various pruning techniques, which can be used to guide users to design better deep pruning methods in the future. With some basic concepts and findings from data matrix approximation, the framework can explain the success of the state-of-the-art methods as well as a more generalized pruning design (Layer-Scaled One-shot Pruning, LSOP) proposed in this work. After pruning with different algorithms and measuring their accuracies, the researcher also found that those methods or algorithms aligned with the proposed framework were more accurate at sparser networks (density < 10%) than the methods that did not. This suggests that future research into neural network pruning can focus on the proposed framework, which has the potential to accelerate the development of pruning technology and adoption of more efficient neural networks.
The LSOP framework’s capability to explain strong pruning performances implies that polynomial decay and Low-Rank Matrix Approximation techniques from the field of data science can provide support for neural network pruning.
Downloads
References or Bibliography
X. Dong, S. Chen, and S. J. Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems, 2017.
E. Elsen, M. Dukhan, T. Gale, and K. Simonyan. Fast sparse ConvNets. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. In Proceedings of International Conference on Machine Learning, 2020.
J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
T. Gale, E. Elsen, and S. Hooker. The state of sparsity in deep neural networks. In ICML ODML- CDNNR Workshop, 2019.
T. Gale, M. Zaharia, C. Young, and E. Elsen. Sparse GPU kernels for deep learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, 2020.
S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, 2015.
Y. LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, 1989.
J. Lee, S. Park, S. Mo, S. Ahn and J. Shin. Snip: Layer-Adaptive Sparsity For the Magnitude-Based Pruning. In International Conference on Learning Representations, 2019.
N. Lee, T. Ajanthan, and P. H. S. Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019.
D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibescu, and A. Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 2018.
A. S. Morcos, H. Yi, M. Paganini, and Y. Tian. One ticket to win them all: Generalizing lottery ticket initializations across datasets and optimizers. In Advances in Neural Information Processing Systems, 2019.
M. C. Mozer and P. Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In Advances in Neural Information Processing Systems, 1988.
B. Mussay, M. Osadchy, V. Braverman, S. Zhou, and D. Feldman. Data-indenpendent neural pruning via coresets. In International Conference on Learning Representations, 2020.
B. Neyshabur, R. Tomioka, and N. Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, 2015.
S. Park, J. Lee, S. Mo, and J. Shin. Lookahead: A far-sighted alternative of magnitude-based pruning. In International Conference on Learning Representations, 2020.
A. Renda, J. Frankle, and M. Carbin. Comparing fine-tuning and rewinding in neural network pruning. In International Conference on Learning Representations, 2020.
V. Sanh, T. Wolf, and A. M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. arXiv preprint 2005.07683, 2020.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
M. Tan and Q. V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of International Conference on Machine Learning, 2019.
J. Tropp, A. Yurtsever, M. Udell, and V. Cevher. Streaming Low-Rank Matrix Approximation with an Application to Scientific. SIAM J. Scientific Computing, 2019.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
X. Xiao, Z. Wang, and S. Rajasekaran. AutoPrune: Automatic network pruning by regularizing auxiliary parameters. In Advances in Neural Information Processing Systems, 2019.
C. Yun, S. Sra, and A. Jadbabaie. Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity. In Advances in Neural Information Processing Systems, 2019.
H. Zhou, J. Lan, R. Liu, and J. Yosinki. Deconstructing lottery tickets: Zeros, signs, and supermasks. In Advances in Neural Information Processing Systems, 2019.
M. Zhu and S. Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. In ICLR Workshop track, 2018.
Published
How to Cite
Issue
Section
Copyright (c) 2022 Oliver Wang; Professor Constantine Dovrolis, Professor Jaeho Lee
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright holder(s) granted JSR a perpetual, non-exclusive license to distriute & display this article.
