Feed Forward Neural Network Sparsification with Dynamic Pruning
Date
2021Language
en
Keyword
Abstract
A recent hot research topic in deep learning concerns the reduction of the model size of a neural network by pruning, in order to minimize its training and inference cost and thus, being capable of running on devices with memory constraints. In this paper, we employ a pruning technique to sparsify a Multi-Layer Perceptron (MLP) during training, in which the number of topology connections, being pruned and restored, is not stable, but it adopts either one of the following rules: Linear Decreasing Variation (LDV) rule or Oscillating Variation (OSV) rule or Exponential Decay (EXD) rule. We conducted experiments on three MLP Network topologies, implemented with Keras, using the Fashion-MNIST dataset and results showed that the EXD method is a clear winner since, in that case our proposed sparse network has a faster convergence than the dense version of the same one, while it achieves approximately the same high accuracy (around 90%). Furthermore, it is shown that the memory footprint of the aforementioned sparse techniques is at least 95% less instead of the dense version of the network, due to the weights removed. Finally, we present an improved version of the SET implementation in Keras, using Callbacks API, making the SET implementation more efficient. © 2021 ACM.
Collections
Related items
Showing items related by title, author, creator and subject.
-
Deep Endoscopic Visual Measurements
Iakovidis D.K., DImas G., Karargyris A., Bianchi F., Ciuti G., Koulaouzidis A. (2019)Robotic endoscopic systems offer a minimally invasive approach to the examination of internal body structures, and their application is rapidly extending to cover the increasing needs for accurate therapeutic interventions. ... -
Pose recognition using convolutional neural networks on omni-directional images
Georgakopoulos S.V., Kottari K., Delibasis K., Plagianakos V.P., Maglogiannis I. (2018)Convolutional neural networks (CNNs) are used frequently in several computer vision applications. In this work, we present a methodology for pose classification of binary human silhouettes using CNNs, enhanced with image ... -
Resource-efficient TDNN Architectures for Audio-visual Speech Recognition
Koumparoulis A., Potamianos G., Thomas S., da Silva Morais E. (2021)In this paper, we consider the problem of resource-efficient architectures for audio-visual automatic speech recognition (AVSR). Specifically, we complement our earlier work that introduced efficient convolutional neural ...