In this work, we consider the applicability of the face recognition algorithms to the data obtained from a dynamic vision sensor. A basic method using a neural network model comprised of reconstruction, detection, and recognition is proposed that solves this problem. Various modifications of this algorithm and their influence on the quality of the model are considered. A small test dataset recorded on a DVS sensor is collected. The relevance of using simulated data and different approaches for its creation for training a model was investigated. The portability of the algorithm trained on synthetic data to the data obtained from the sensor with the help of fine-tuning was considered. All mentioned variations are compared to one another and also compared with conventional face recognition from RGB images on different datasets. The results showed that it is possible to use DVS data to perform face recognition with quality similar to that of RGB data.
In recent years, a new type of camera is gaining popularity, Dynamic Vision Sensor (DVS). While in traditional cameras, information is recorded with a certain fixed frequency (usually 25–30 times per second), the dynamic vision sensor records only the fact of a change in the level of illumination in a pixel if it exceeds a certain threshold. Thus, these cameras operate according to the principle of the human eye, which responds only to changes. This approach allows to get rid of a large amount of redundant static data, focusing only on dynamic events. Such sensors have several advantages: high speed (which allows to catch very fast events in small details), low power and memory consumption (an important feature for embedded systems, where there is no way to place a large battery and hard drive), high sensitivity (a key property for recording under extreme light conditions). ? Publication is supported by RFBR grant 18-08-01484.
Such cameras are quite expensive and do not have high resolution, however, due to
the rapid development of technologies in this area, it is necessary to develop algorithms
for solving various applied problems, such as 3D reconstruction, detection and tracking
of objects [
Nowadays, systems that perform face recognition are very relevant, since they implement the most effective way of contact-less identification of a person. They are used in security systems, bank card verification, people mark-up, forensics, online payments etc. Face recognition problem can be decomposed into several sub-tasks: finding the face in the image, normalizing the found face and, finally, identifying the person. In this paper we propose a new method for face recognition using data from DVS sensor. 2 2.1
The problem of detecting faces is one of the special cases of the detection problems, but
it has its own specificities. The human face has distinctive features, which were searched
and analyzed in first approaches in this area. A big breakthrough was [
Then the partially-deformed models [
Similarly to most of computer vision tasks, the detection problem can be solved by
the deep learning methods, the popularity of which has grown significantly after the
work [
The face recognition problem has been of interest to the scientific world for a long time.
The first systems for solving this problem were developed back in 1964 [
Various methods were used to solve the face recognition problem, and these
methods have changed greatly over time. The first algorithms attempted to distinguish
between faces by finding distinctive features such as eye color, face proportions, etc. [
Neural Network Model for Face Recognition from Dynamic Vision Sensor 3
of modern methods [
One of the key aspects of this work is an algorithm for reconstructing frames from the
stream of events of a dynamic vision sensor. The algorithm was proposed by [
The face recognition problem can be considered in two equivalent forms: identification and verification. In this paper verification form was chosen.
Data from the dynamic vision sensor comes in the form of set of events.
Event — (x; y; ts; p), where x; y 2 Z; x 2 [0; N ]; y 2 [0; M ] are the coordinates of the pixel in the matrix N M , ts 2 R — timestamp, p 2 f 1; 1g — the polarity of the change (the brightness in the pixel decreased/increased by a given threshold).
Algorithm Input: sets of events T1 and T2 received from the dynamic vision sensor.
Algorithm Output: A 2 f0; 1g: A = 1 if the sets T1 and T2 describe one person, A = 0 if different. 3.2
Since the main source of information in computer vision tasks is usually an image or a
sequence of images, but not the event stream obtained by DVS, it is necessary to convert
the stream of events into visual representation. Such a visualization can be made in
different ways. The most simple one is setting time marks and counting events occurred
between and visualizing it in gray-scale. However, it turns out, that this approach does
not provide with satisfying quality and detectors could not find faces on such images.
Thus, the reconstructions with neural network [
The proposed basic method works as follows (see Fig. 2): first, the stream of events
from the sensor is reconstructed into frames with model [
Pn = pPn i=1 f1if2ii=1 (f2i)2 i=1 (f1i)2pPn (1) The dynamic vision sensor is a fairly new type of cameras, and very few datasets have been recorded for it so far. As far as we know there are no publicly available datasets for a face recognition task. Therefore, it was proposed to use the collections of color videos collected for the face recognition task and simulate dynamic vision sensor data from them. It can be done in two steps: firstly, the intermediate values in each pixel are interpolated between two neighboring frames and secondly, at each point the change in
Neural Network Model for Face Recognition from Dynamic Vision Sensor 5 intensity between adjacent interpolated frames is compared, and if this change exceeds the threshold, an event is generated. As you can see in fig. 3, the results of real and simulated event streams are very similar. This gives us the opportunity to assume that studies conducted on simulated data will be fairly well transferred to real data.
Since the linear interpolation of intermediate frames is not very fair leading to
blurred frames, it was proposed to improve the simulation method by using better
approximation. To do this, it was decided to use the results of [
YouTubeFaces. The main dataset that met the criteria for simulation was
YouTubeFaces Dataset [
ChokePoint. In addition, it is proposed to use dataset obtained under conditions
similar to real scenarios of using a dynamic vision sensor as a test set. For this, the
ChokePoint [
GML DVS. In order to check the portability of the created model for real data obtained from a dynamic vision sensor, a small dataset of eight video sequences containing eight people was captured. 80 faces were automatically found by face detector and manually labelled. 5.2
To evaluate the quality of the proposed method, we make a verification experiment selecting the pairs of objects and comparing their similarity with some threshold to decide if they belong to the same person or not. Setting different thresholds to distinguish faces we can obtain AUC metric which is the area below ROC curve and that can be a great indicator of general performance of the model. The method is tested against verification on RGB images when possible. Variations with advanced reconstruction and uses of fine-tuning on those reconstructions are examined. The results are presented in Table 1 and 2.
We can see that recognition results on DVS reconstructions are quite similar to those on RGB images and that fine-tuning enables us to improve quality of a model. Furthermore, this fine-tuning allows to enhance performance on data obtain from the real DVS sensor which was also quite good comparing to simulated data proving the portability of model.
Neural Network Model for Face Recognition from Dynamic Vision Sensor 7 This paper explores the possibility of constructing a solution to the face recognition problem based on dynamic vision sensor data. It implements the basic solution method using a neural network model. The results show that we can apply existing methods to solve this task at a level similar to that of the RGB images. Uses of simulated frames provided a great way to improve performance of the model which is very helpful due to the scarce amount of real data.