Efficient Layer Optimizations for Deep Neural Networks
Article Sidebar
Main Article Content
Abstract
Deep neural networks (DNNs) have technical issues such as long training time as the network size increases. Parameters require significant memory, which may cause migration issues for embedded devices. DNNs applied various pruning techniques to reduce the network size in deep neural networks, but many problems still exist when applying the pruning techniques. Among neural networks, several applications applied autoencoders for reconstruction and dimension reduction. However, network size is a disadvantage of autoencoders since the architecture of the autoencoders has a double workload due to the encoding and decoding processes. In this research, we chose autoencoders and two deep neural networks – AlexNet and VGG16 to apply out-of-order layer pruning. We perform the sensitivity analysis to explore the performance variations for the network architecture and network complexity through an out-of-order layer pruning mechanism. As a result of applying the proposed layer pruning scheme to the autoencoder, we developed the accordion autoencoder (A2E) and applied credit card fraud detection and MNIST classification. Our results show 4.9% and 13.6% performance drops, respectively, but we observe a significant reduction in network complexity, 85.1% and 94.5% for each application. We extend the out-of-order layer pruning to deeper learning networks. In our approach, we propose a simple yet efficient scheme, accuracy-aware structured filter pruning based on the characterization of each convolutional layer combined with the quantization of fully connected layers. We investigate the accuracy and compression rate of each layer using a fixed pruning ratio, and then the pruning priority is rearranged depending on the accuracy of each layer. Our analysis of layer characterization shows that the pruning order of the layers does affect the final accuracy of the deep neural network. Based on our experiments using the proposed pruning scheme, the parameter size in the AlexNet can be up to 47.28x smaller than the original model. We also obtained comparable results for VGG16, achieving a maximum compression rate of 35.21x.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
How to Cite
References
D. Bank, N. Koenigstein, R. Giryes, "Autoencoders," Book Chapter in Machine Learning for Data Science Handbook, 2023. https://doi.org/10.1007/978-3-031-24628-9_16
S. Lin, R. Ji, Y. Li, C. Deng, X. Li "Towards Compact ConvNets via Structure-Sparsity Regularized Filter Pruning," IEEE Transactions on Neural Networks and Learning Systems, 2020. https://doi.org/10.1109/tnnls.2019.2906563
S. S. Tirumala, "A Novel Weights of Weights Approach for Efficient Transfer Learning in Artificial Neural Networks," Procedia Computer Science, 2022. https://doi.org/10.1016/j.procs.2022.11.013
J. Luo, J. Wu, W. Lin, "ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression," IEEE International Conference on Computer Vision, 2017. https://doi.org/10.1109/iccv.2017.541
J. Guo, M. Potkonjak, "Pruning ConvNets Online for Efficient Specialist Models," IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017. https://doi.org/10.1109/cvprw.2017.58
P. Hu, X. Peng, H. Zhu, M. Aly, J. Lin, "OPQ: Compressing Deep Neural Networks with One-shot Pruning-Quantization," Proceedings of the AAAI Conference on Artificial Intelligence, 2021. https://doi.org/10.1609/aaai.v35i9.16950
G. E. Hinton and R. R. Salakhutdinov, "Reducing the Dimensionality of Data with Neural Networks," SCIENCE Vol. 313, Issue 5786, pp. 504-507, 2006. https://doi.org/10.1126/science.1127647
M. Kon and L. Plaskota, "Information Complexity of Neural Networks," Neural Network, 2000. https://doi.org/10.1016/s0893-6080(00)00015-0
Neloy, M. Turgeon, "A Comprehensive Study of Auto-Encoders for Anomaly Detection: Efficiency and Trade-Offs," SSRN, 2024. http://dx.doi.org/10.2139/ssrn.4757204.
S. Bhatia, R. Bajaj, S. Hazari, "Analysis of Credit Card Fraud Detection Techniques," International Journal of Science and Research (IJSR), 2006. https://doi.org/10.21275/v5i3.nov162099.
J. Romero, J. P. Olson, and A. Aspuru-Guzik, "Quantum autoencoders for efficient compressions of quantum data," arXiv:1612.02806, 2017. https://doi.org/10.1088/2058-9565/aa8072
Krizhevsky, I. Sutskever, G. E. Hinton, "ImageNet classification with deep convolutional neural networks," Proceedings of the 25th International Conference on Neural Information Processing Systems, 2017. https://doi.org/10.1145/3065386
J. Kim, J. Lee, K. Lee, "Accurate Image Super-Resolution Using Very Deep Convolutional Networks," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. https://doi.org/10.1109/cvpr.2016.182
Jiang, G, Goldsztein, "Convolutional Neural Network Approach to Classifying the CIFAR-10 Dataset," Journal of Student Research, 2023. https://doi.org/10.47611/jsrhs.v12i2.4388.
Y. Zheng, H. Huang, J. Chen, "Comparative analysis of various models for image classification on Cifar-100 dataset," Journal of Physics, 2024. https://doi.org/10.1088/1742-6596/2711/1/012015.
N. Ketkar, J. Moolayil, "Introduction to PyTorch," Book Chapter in Deep Learning with Python, 2021. https://doi.org/10.1007/978-1-4842-5364-9_2
F. Nielsen, "Linking ImageNet WordNet Synsets with Wikidata," WWW '18: Companion Proceedings of the The Web Conference, 2018. https://doi.org/10.1145/3184558.3191645.
J. Naylor, K. P. Li, "Analysis of a neural network algorithm for vector quantization of speech parameters," ITT Defense Communications Division, 1988. https://doi.org/10.1016/0893-6080(88)90341-3.
Li, D. Chen, L. Shi, B. Jin, J. Goh, and S. Ng, "MAD-GAN: Multivariate Anomaly Detection for Time Series Data with Generative Adversarial Networks," arXiv:1901.04997, 2019. https://doi.org/10.1007/978-3-030-30490-4_56
E. Bisong, "Autoencoders," Book Chapter in Building Machine Learning and Deep Learning Models on Google Cloud Platform, 2019. https://doi.org/10.1007/978-1-4842-4470-8_37.
K. Kelso and B. Lee, "Accordion Autoencoders (A2E) for Generative Classification with Low Complexity Network," International Symposium on Computational Intelligence, December 2021. https://doi.org/10.1109/csci54926.2021.00152
M. Carballo and B. Lee, "Accuracy-aware Structured Filter Pruning for Deep Neural Networks," International Symposium on Artificial Intelligence, December 2020. https://doi.org/10.1109/csci51800.2020.00122
K. Sai krishna, G. Sreenivasa Raju, P. Praveen Kumar, A Deep Neural Network for face Recognition. (2019). In International Journal of Innovative Technology and Exploring Engineering (Vol. 8, Issue 12S, pp. 420–423). https://doi.org/10.35940/ijitee.l1105.10812s19
Young, L., York, J. R., & Kil Lee, B. (2023). Implications of Deep Compression with Complex Neural Networks. In International Journal of Soft Computing and Engineering (Vol. 13, Issue 3, pp. 1–6). https://doi.org/10.35940/ijsce.c3613.0713323
Das, S., S, S., M, A., & Jayaram, S. (2021). Deep Learning Convolutional Neural Network for Defect Identification and Classification in Woven Fabric. In Indian Journal of Artificial Intelligence and Neural Networking (Vol. 1, Issue 2, pp. 9–13). https://doi.org/10.54105/ijainn.b1011.041221
Chellatamilan, T., Valarmathi, B., & Santhi, K. (2020). Research Trends on Deep Transformation Neural Models for Text Analysis in NLP Applications. In International Journal of Recent Technology and Engineering (IJRTE) (Vol. 9, Issue 2, pp. 750–758). https://doi.org/10.35940/ijrte.b3838.079220
Magapu, H., Krishna Sai, M. R., & Goteti, B. (2024). Human Deep Neural Networks with Artificial Intelligence and Mathematical Formulas. In International Journal of Emerging Science and Engineering (Vol. 12, Issue 4, pp. 1–2). https://doi.org/10.35940/ijese.c9803.12040324