Recognition on images not only of shapes, but also of metadata is becoming increasingly popular among researchers in the field of convolutional neural networks and deep learning. This article provides an analytical overview of modern software solutions that recognize the images of emotions, mood, gender and age of a person. Enthusiasts invent all new and new architectures of convolutional neural networks, allowing to solve the tasks with considerable recognition accuracy.
A solution that is a trained neural network that recognizes real-time emotions on a human face recognized from the input video stream.
It was built using the TFLearn programming library for the python programming language, based
on the well-known TensorFlow machine learning framework developed by Google in 2015 [
The resulting neural network model is shown in Figure 1.
In each frame of the video stream, an attempt is made to recognize a human face (s). This is
achieved using the OpenCV open library recognition method [
The model was trained with the help of dataset FERC-2013, which has about 20,000 images containing examples of the following emotions: anger, fear, happiness, sadness, surprise, indifference and disgust. The density of the distribution of emotions in this data is reflected in Figure 2.
According to the results of training, the accuracy of recognition of emotions was achieved in 67%. 2.2. SSR-Net This solution is an original neural network with soft stepwise regression (soft stagewise regression network) for recognizing age and sex. The network recognizes age and gender according to the following principle: images of 64x64 pixels are fed to the input of the network, a multi-level classification is made from several classes, where each level serves to refine the previous result, and then the result of the classification is processed using a regression.
The model itself is very compact and takes only 0.32 MB. But in spite of its compact dimensions, the performance of SSR-Net is close to the characteristics of the most modern methods, the sizes of models of which are 1500 times larger.
A model of this neural network with three levels and a pool size of 2 is shown in Figure 3.
For training this model, such datasets as IMDB, WIKI and Morph2 [11] were used. About 80% of randomly selected images from datasets were used to train the network, and the remaining 20% were used for testing.
An example of dependence of the number of SSR-Net, MobileNet and DenseNet network recognition errors trained in Morph2 data on the number of epochs is presented in Figure 4.
It is a real-time classifier of emotions and gender of a person, based on the convolutional neural
network and the open image processing library openCV [
This model was trained in IMDB dataset, which has about 460,723 RGB images, each of which
belongs to one of the classes: male or female [
This solution is the implementation of a convolutional neural network for recognizing the sex and age of a person from the input image. The basis for the VGG-16 network architecture was taken due to its depth and controllability. This network accepts 256x256 pixel images as input.
The training was carried out on IMDB-WIKI datasets, and recognition accuracy of 64% was
achieved [
As a result of the analytical review, the decisions contained in the publications [
The source code of each solution was carefully analyzed and revised so that the digital image was provided as input to the software, and the result of the prediction of a convolutional neural network was obtained.
At the end of the previous section, software solutions were obtained, the main task of which is to recognize the age, gender and emotions of a person from a digital face image.
These solutions were chosen as objects for conducting an experimental study of their performance on 10 random images of the faces of people from the IMDB-WIKI dataset.
The following equipment and software were used during the pilot study: 1. Processor: intel Core i5-4570 3.2 GHz. 2. RAM: 8 Gb.
3. Operation system: Manjaro 18.0.4 «Illyria». 4. Programming language: Python 3.6.5.
Below (Figures 6–15)are the images used in the pilot study:
Since the neural network with the same input data always produces the same result, performing multiple independent attempts at recognition in a row does not have much value, however, to measure a more accurate execution time of a convolutional neural network in a single digital image.
Tables 1, 2 and 3 present the results of recognizing age, gender and emotions, respectively, as well as the average execution time for 5 runs of a convolutional neural network in the next digital image.
As a result, we can conclude that such tasks as the recognition of emotions, mood, gender and age are very popular among researchers all over the world. Enthusiasts use different approaches to the implementation of intelligent systems that can solve such problems and achieve good results in accuracy of recognition, even with limited resources. The main means of the implementation of the tasks are convolutional neural networks of various architectures, trained in well-known in the network dataset images.
This work was partly funded by the Russian Foundation for Basic Research – Project # 17-29-03112 ofi_m and the Russian Federation Ministry of Science and Higher Education within a state contract with the "Crystallography and Photonics" Research Center of the RAS under agreement 007-ГЗ/Ч3363/26.