Model reduction of feed forward neural networks for resource-constrained devices
Datum
2022Language
en
Schlagwort
Zusammenfassung
Multilayer neural architectures with a complete bipartite topology have very high training time and memory requirements. Solid evidence suggests that not every connection contributes to the performance; thus, network sparsification has emerged. We get inspiration from the topology of real biological neural networks which are scale-free. We depart from the usual complete bipartite topology among layers, and instead we start from structured sparse topologies known in network science, e.g., scale-free and end up again in a structured sparse topology, e.g., scale-free. Moreover, we apply smart link rewiring methods to construct these sparse topologies. Thus, the number of trainable parameters is reduced, with a direct impact on lowering training time and a direct beneficial result in reducing memory requirements. We design several variants of our concept (SF2SFrand, SF2SFba, SF2SF5, SF2SW, and SW2SW, respectively) by considering the neural network topology as a Scale-Free or Small-World one in every case. We conduct experiments by cutting and stipulating the replacing method of the 30% of the linkages on the network in every epoch. Our winning method, namely the one starting from a scale-free topology and producing a scale-free-like topology (SF2SFrand) can reduce training time without sacrificing neural network accuracy and also cutting memory requirements for the storage of the neural network. © 2022, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.
Collections
Verwandte Dokumente
Anzeige der Dokumente mit ähnlichem Titel, Autor, Urheber und Thema.
-
Deep Endoscopic Visual Measurements
Iakovidis D.K., DImas G., Karargyris A., Bianchi F., Ciuti G., Koulaouzidis A. (2019)Robotic endoscopic systems offer a minimally invasive approach to the examination of internal body structures, and their application is rapidly extending to cover the increasing needs for accurate therapeutic interventions. ... -
Feed Forward Neural Network Sparsification with Dynamic Pruning
Chouliaras A., Fragkou E., Katsaros D. (2021)A recent hot research topic in deep learning concerns the reduction of the model size of a neural network by pruning, in order to minimize its training and inference cost and thus, being capable of running on devices with ... -
Pose recognition using convolutional neural networks on omni-directional images
Georgakopoulos S.V., Kottari K., Delibasis K., Plagianakos V.P., Maglogiannis I. (2018)Convolutional neural networks (CNNs) are used frequently in several computer vision applications. In this work, we present a methodology for pose classification of binary human silhouettes using CNNs, enhanced with image ...