An Ensemble Learning Method of Adaptive Structural Deep Belief Network for AffectNet
Abstract
Deep Learning is a hierarchical network architecture to express complex abstractions of input patterns of images. A Deep Belief Network (DBN) that builds hierarchical structure of Restricted Boltzmann Machine (RBM) is a well known unsupervised learning method as one of deep learning methods. The adaptive structural learning method of RBM (Adaptive RBM) was developed to find a suitable network structure for the input data set by neuron generation / annihilation algorithm during training. The Adaptive DBN can construct to pile an appropriate number of RBMs up to realize higher classification task. In this paper, our developed model was applied to AffectNet as the facial image data set and showed the better performance of classification rate than the State-of-The-Art CNN models. However, the model outputs incorrect wrong emotion category for some test cases, because the output labels for data set were annotated by two or more human annotators. For the problem, this paper proposes an ensemble learning model of Adaptive DBN, where the ensemble model consists of a parent DBN and some child DBNs. KL divergence is a measure of similarity for the parent and the child to each case. The new neurons are generated at the child to improve the classification according to KL divergence. Moreover, the generated neuron at the child is transferred to the parent to integrate better knowledge. In this paper, the proposed method improved the classification accuracy from 87.4% to 92.5%.
References
A.Krizhevsky, I.Sutskever, G.E.Hinton, ImageNet Classification with Deep Convolutional Neural Networks, Proc. of Advances in Neural Information Processing Systems 25 (NIPS 2012) (2012).
C.Szegedy, W.Liu, et.al., Going Deeper with Convolutions, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1-9 (2015).
K.Simonyan, A.Zisserman, Very deep convolutional networks for large-scale image recognition, Proc. of International Conference on Learning Representations (ICLR 2015) (2015).
K.He, X.Zhang, S.R en, J.Sun, Deep residual learning for image recognition, Proc. of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.770-778 (2016).
G.E.Hinton, S.Osindero and Y.Teh, A fast learning algorithm for deep belief nets, Neural Computation, vol.18, no.7, pp.1527-1554 (2006).
G.E.Hinton, A Practical Guide to Training Restricted Boltzmann Machines, Neural Networks, Tricks of the Trade, Lecture Notes in Computer Science (LNCS, vol.7700), pp.599-619 (2012).
S.Kamada and T.Ichimura, An Adaptive Learning Method of Restricted Boltzmann Machine by Neuron Generation and Annihilation Algorithm, Proc. of 2016 IEEE International Conference on Systems, Man, and Cybernetics (IEEE SMC 2016), pp.1273-1278 (2016).
S.Kamada and T.Ichimura, A Structural Learning Method of Restricted Boltzmann Machine by Neuron Generation and Annihilation Algorithm, Neural Information Processing, vol.9950 of the series Lecture Notes in Computer Science, pp.372-380 (2016).
S.Kamada and T.Ichimura,An Adaptive Learning Method of Deep Belief Network by Layer Generation Algorithm, Proc. of IEEE TENCON2016, 2971-2974 (2016).
A.Krizhevsky, Learning Multiple Layers of Features from Tiny Images, Master of thesis, University of Toronto (2009).
S.Kamada, T.Ichimura, A.Hara, and K.J.Mackin, Adaptive Structure Learning Method of Deep Belief Network using Neuron Generation-Annihilation and Layer Generation, Neural Computing and Applications, pp.1-15 (2018).
S.Kamada, T.Ichimura, T.Harada, Knowledge Extraction of Adaptive Structural Learning of Deep Belief Network for Medical Examination Data, International Journal of Semantic Computing, Vol.13, No.1, pp. 67-86 (2019).
A.Mollahosseini, B.Hasani, M.H.Mahoor: AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild, IEEE Transactions on Affective Computing, vol.10, No.1 pp.18-31 (2017).
T.Ichimura, S.Kamada, Re-learning of Child Model for Misclassified data by using KL Divergence in AffectNet: A Database for Facial Expression, Proc. of 2019 IEEE 11th International Workshop on Computational Intelligence and Applications (IWCIA2019), pp.15-20 (2019).
T.Ichimura, S.Kamada, A Distillation Learning Model of Adaptive Structural Deep Belief Network for AffectNet: Facial Expression Image Database, Proc. of the 9th International Congress on Advanced Applied Informatics(IIAI AAI 2020), pp.454-459 (2020).
G.E.Hinton, Training products of experts by minimizing contrastive divergence. Neural Computation, vol.14, pp.1771-1800 (2002).
J.Cohen, A coefficient of agreement for nominal scales, Educational and Psychological Measurement, vol.20, no.1, p.37-46, (1960).
S.Kamada and T.Ichimura, Fine Tuning of Adaptive Learning of Deep Belief Network for Misclassification and its Knowledge Acquisition, International Journal Computational Intelligence Studies, Vol.6, No.4, pp.333-348 (2017).
J.R.Quinlan, Improved use of continuous attributes in c4.5, Journal of Artificial Intelligence Research, Vol.4, No.1, pp.77-90 (2016).
