Optimizing Convolutional Neural Networks Utilizing Tensor Decomposition Techniques for Large-Scale Image Recognition Tasks
DOI:
https://doi.org/10.47611/jsrhs.v12i3.4916Keywords:
Convolutional Neural Networks, Tensor Decomposition, Tensor, Rank of DecompositionAbstract
Convolutional Neural Networks (CNNs) are integral to numerous applications in today's digital landscape. However, their computational demands often result in slow operation, especially when resources are constrained. This study introduces two tensor decomposition methods aimed at optimizing the performance of CNNs by minimizing the total number of operations and weights, while maintaining accuracy. The first method employs Canonical Polyadic (CP) decomposition to divide the convolutional layer into multiple rank 1 tensors. The second method uses Tucker decomposition to break down the convolutional layer into a compact core tensor and several matrices. The effectiveness of these methods was evaluated on widely-used convolutional architectures, VGG16 and LeNet, by substituting their convolutional layers with a series of decomposed layers, implemented using PyTorch and TensorLy. The CP decomposition method achieved a computational speed-up of 43% with a minimal accuracy reduction of less than 0.12%. Similarly, Tucker decomposition resulted in a 37% speed-up with an accuracy decrease of less than 0.16%. These findings suggest that the proposed tensor decomposition methods can significantly enhance the efficiency of CNNs without significantly impacting their performance.
Downloads
References or Bibliography
Rabanser S., Shchur O., and Günnemann S. “Introduction to Tensor Decompositions and their Applications in Machine Learning.” 2017. https://doi.org/10.1007/978-3-319-68837-4_1
LeCun, Yann, Boser, Bernhard, Denker, John S, Henderson, Donnie, Howard, Richard E, Hubbard, Wayne, and Jackel, Lawrence D. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551, 1989. https://doi.org/10.1162/neco.1989.1.4.541
Tianshui Chen, Liang Lin, Wangmeng Zuo, Xiaonan Luo, Lei Zhang. "Learning a Wavelet-Like Auto-Encoder to Accelerate Deep Neural Networks" https://doi.org/10.1109/TNNLS.2018.2790381
Da Li, Yongxin Yang, Yi-Zhe Song, Timothy M. Hospedales. "Deeper, Broader and Artier Domain Generalization" https://doi.org/10.1109/ICCV.2017.322
Kristof T. Schütt, Farhad Arbabzadah, Stefan Chmiela, Klaus-Robert Müller, Alexandre Tkatchenko. "Quantum-chemical insights from deep tensor neural networks" https://doi.org/10.1038/ncomms13890
Kolda G. Tamara, and Baer W. Brett. “Tensor Decompositions and Applications.” SIAM Review Vol. 51, No.3 pp. 455-500, 2009. https://doi.org/10.1137/07070111X
Ueltschi T., University of Puget SoundTacoma, Washington, USA. “Third-Order Tensor Decompositions and Their Application in Quantum Chemistry.” 2014. https://doi.org/10.1021/ct500847y
Lebedev V., Ganin Y., Rakhuba M., Oseledets I., and Lempitsky V., Skoltech, Moscow, Russia. “Speeding-up Convolutional Neural Networks Using Fine-Tuned CP-Decomposition.” 2015. https://doi.org/10.1007/978-3-319-49409-8_29
Kim Y., Park E., Yoo S., Choi T., Yang L., and Shin D., Samsung Electronics and Seoul National University, South Korea, “Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications.” 2016. https://doi.org/10.1109/ICLR.2016.261
Nakajima S., Sugiyama M., Babacan S. Derin, and Tomoika R., “Global Analytic Solution of Fully-observed Variational Bayesian Matrix Factorizatoin.” 2013. https://doi.org/10.5555/3042817.3043083
Tucker R. L., “Some mathematical notes on three-mode factor analysis.” 1966. https://doi.org/10.1080/01621459.1966.10480843
Chellapilla, Kumar, Puri, Sidd, and Simard, “High performance convolutional neural networks for document processing.” 2006. https://doi.org/10.1109/TNN.2006.880583
De Lathauwer L., De Moor B., and Vanderwalle J., “A Multilinear Singular Value Decompsotion.” 2000. SIAM Vol.21 Iss. 4. https://doi.org/10.1137/S0895479896305696
Minister R., Viviano I., Liu X., Ballard G., “CP Decomposition For Tensors Via Alternating Least Squares With QR Decomposition.” 2021. https://doi.org/10.1109/ICASSP39728.2021.9414887
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015. https://doi.org/10.48550/arXiv.1409.1556
Ye, Jieping. Generalized low rank approximations of matrices. Machine Learning, 61(1-3):167–191, 2005.
Shashua, Amnon and Hazan, Tamir. Non-negative tensor factorization with applications to statistics and computer vision. In Proceedings of the 22nd international conference on Machine learning, pp. 792–799. ACM, 2005. https://dl.acm.org/doi/10.1145/1102351.1102451
Gossmann A., “Understanding the Tucker decomposition, and compressing tensor-value data.” 2017. [Online] Available: https://www.alexejgossmann.com/tensor_decomposition_tucker/
Cohen J., Gusak J., Hashemizadeh M., Kossaifi J., Meurer A., Mo Y. Mardal, Patti T. Lee, Roald M., and Tuna C., Tensorly Tensor Decompositions. 2015. [Online] Available: http://tensorly.org/stable/modules/api.html#module-tensorly.decomposition
Razin N., Maman A., and Cohen N., “Implicit Regularization in Tensor Factorization: Can Tensor rank Shed Light on Generalization in Deep Learning.” 2021. [Online] Available: http://www.offconvex.org/2021/07/08/imp-reg-tf/
Published
How to Cite
Issue
Section
Copyright (c) 2023 Tiancheng Hu
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright holder(s) granted JSR a perpetual, non-exclusive license to distriute & display this article.
