-Multimodal biometrics overcome the technical limitations of unimodal biometrics, making them ideally suited for everyday life applications that require a reliable authentication system. However, for a successful adoption of multimodal biometrics, such systems would require large heterogeneous datasets with complex multimodal fusion and privacy schemes spanning various distributed environments. From experimental investigations of current multimodal systems, this paper reports the analysis of the multimodal biometric system performance based on the combination of voice, face and signature recognitions. The first part of the paper describes different methods used for the recognition of three biometric traits, established databases and relative performance obtained by using unimodal biometrics system. In the second part of the paper the multimodal biometric approach and the performance is described. The EER (Equal Error Rate) obtained with the multimodal approach by using a database of 50 individuals is about 0.4%, whereas most reallife biometric systems are affected with a variety of problems. Finally, the paper presents the implementation of a multimodal biometric system based on a probabilistic neural network in order to improve the recognition rate in a noisy scenario
INTRODUCTION
At present day the proper functioning of many social,
financial, and political structures relies on the correct
identification of people. However, a reliable and unique identification
of people is a difficult problem. Biometric methods, which
identify people based on physical or behavioral characteristics,
are increasingly considered as people cannot forget or lose
their physical characteristics if compared e.g. to the loss of
passwords or identity cards. Biometrics is the identification
process of a person based on physiological or behavioral
characteristics [
Biometrics can be used at least in two different types of applications. In a verification scenario, a person claims a particular identity and the biometric system is used to verify or reject this claim. Verification is performed by matching a biometric sample acquired at the time of the claim and compared to the sample previously enrolled for the claimed identity. If the two samples match well enough, the identity claim is verified, however, if the two samples do not match well enough, the claim is rejected. Thus, there are four possible outcomes. A true accept occurs when the system accepts, or Copyright c 2016 held by the authors. verifies, an identity claim, and the claim is true. A false accept occurs when the system accepts an identity claim, but the claim is true. The two types of errors that can be made are a false accept and a false reject. Equal-error rate (EER) means that the false accept rate equals the false reject rate. The terms verification and authentication are often used interchangeably in this context. The set of enrolled samples is often called a gallery, and the unknown sample is often called a probe.
Similar to the verification scenario, there are four possible outcomes. A true positive occurs when the system says that an unknown sample matches a particular person in the gallery and the match is correct. A false positive occurs when the system says that an unknown sample matches a particular person in the gallery and the match is incorrect. A true negative occurs when the system says that the sample does not match any of the entries in the gallery, and the sample in fact does not. A false negative occurs when the system says that the sample does not match any of the entries in the gallery, but the sample in fact does belong to someone in the gallery. Both measures are often dependent on each other. When decreasing False Rejection Rate, False Acceptance Rate increases and viceversa.
A typical biometric system consists of four main modules. The sensor module is responsible for acquiring the biometric data from an individual. The feature extraction module processes the acquired biometric data and extracts only the salient information to form a new representation of the data. Ideally, this new representation should be unique for each person and also relatively invariant with respect to changes in the different samples of the same biometric collected from the same person. The matching module compares the extracted feature set with the templates stored in the system database and determines the degree of similarity (dissimilarity) between the two. The decision module either verifies the identity claimed by the user or determines the users identity based on the degree of similarity between the extracted features and the stored template. Biometric systems can provide three main functionalities, namely, verification, identification and negative identification. The system acquires the biometric data from the user and compares it only with the template. In identification, the user’s input is compared to the templates of all the people enrolled in the database, and the identity of the person whose template has the highest degree of similarity with the users input is the output by the biometric system. Unimodal biometric systems perform person recognition based on a single source of biometric information. Such systems are often affected by several problems, such as noisy sensor data, non-universality, lack of individuality, lack of invariant representation.
Some of the problems that affect unimodal biometric
systems can be alleviated by using multimodal biometric systems.
Systems that consolidate cues obtained from two or more
biometric sources for the purpose of human recognition are
called multimodal biometric systems. Combining the evidence
obtained from different modalities using an effective fusion
scheme significantly improves the overall accuracy of the
biometric system [
However, multimodal biometric systems are more expensive and require more resources for computation and storage than unimodal biometric systems. The multimodal biometric system is used in the integration or fusion of information of the sensor level, feature extraction level, matching score level and decision level. Typically, the architecture of a multimodal biometric system is either serial or parallel. In the serial or cascade architecture, the processing of the modalities takes place sequentially and the outcome of one modality affects the processing of the subsequent modalities. In the parallel design, different modalities operate independently and their results are combined using an appropriate fusion scheme. A multimodal system designed to operate in the parallel mode generally has a higher accuracy because it utilizes more evidence regarding the user for recognition. In the architecture of the present system we have adopted the parallel fusion design. Fusion in multimodal biometric systems can take place at four major levels, namely, sensor level, feature level, score level and decision level.
These four levels can be broadly categorized into fusion prior to matching and fusion after matching. Prior to matching, integration of information can take place either at the sensor level or at the feature level. The raw data from the sensors are combined in sensor level fusion. Feature level fusion refers to combining different feature vectors that are obtained from one of the following sources: multiple sensors for the same biometric trait, multiple instances of the same biometric trait, multiple units of the same biometric trait or multiple biometric traits. When the feature vectors are homogeneous (e.g., multiple fingerprint impressions of a users finger), a single resultant feature vector can be calculated as a weighted average of the individual feature vectors. When the feature vectors are non-homogeneous (e.g., feature vectors of different biometric modalities like face and hand geometry), we can concatenate them to form a single feature vector. When the biometric matchers output a set of possible matches along with the quality of each match (matching score), integration can be performed at the matching score level. This is also known as fusion at the measurement level or confidence level. Next to the feature vectors, the matching scores output by the matchers contain the richest information about the input pattern. Also, it is relatively easy to access and combine the scores generated by different matchers.
II. PERFORMANCE OF MONOMODAL BIOMETRIC SYSTEM
Voice recognition or speaker recognition refers to the
automated method of identifying or confirming the identity of
an individual based on his voice [
The voice is considered both a physiological and a behavioral biometric factor: the physiological component of speaker recognition is the physical shape of the subject’s voice tract; the behavioral component is the physical movement of jaws, tongue and larynx.
fixed phrase (password) which is the same for enrollment and for verification, or the subject is prompted by the system to repeat a randomly generated phrase.
on whatever words the subject says.
Text dependent recognition has better performance for subjects that cooperate. But text independent voice recognition is more flexible and it can be used for non-cooperating individuals.
Basically identification or authentication using speaker recognition consists of four steps: voice recording; feature extraction; pattern matching; decision (accept/reject).
Depending on the application a voice recording is performed using a local, dedicated system or remotely (e.g. telephone). The acoustic patterns of speech can be visualized as loudness or frequency vs. time. Speaker recognition systems analyze the frequency as well as attributes such as dynamics, pitch, duration and loudness of the signal.
During feature extraction the voice recording is cut into windows of equal length, these cut-out samples are called frames which are often 10 to 30 ms long.
Pattern matching is the actual comparison of the extracted
frames with known speaker models (or templates), this results
in a matching score which quantifies the similarity between
the voice recording and a known speaker model. Pattern
matching is often based on Hidden Markov Models (HMMs)
[
Alternatively Dynamic Time Warping is used, this algorithm measures the similarity between two sequences that vary in speed or time, even if this variation is non-linear such as when the speaking speed changes during the sequence. Fig. 1 shows a block diagram of a speaker/speech recognition system.
Different methods have been used in the field of voice
recognition [
Usually the voiced/unvoiced analysis is performed in
conjunction with pitch analysis. Rabiner et al. [
The biometric voice recognition techniques used in this study is the Alize software platform and LIA RAL based on GMM (Gaussian Mixture Model) strategy, where a sum of Gaussian probability distributions is used to model each speaker. The ALIZE/LIA RAL toolkit is developed jointly by the members of the ELISA consortium and consists of two separate components: Alize, i.e. the low-level statistical framework, and LIA RAL, i.e. the set of high-level utilities that perform each of the tasks of a state-of-the-art speaker recognition system. The latter is also sometimes referred to as Mistral. One of their main advantages is the high level modularity of the tools: each program does not directly depend on the other and the data between the modules is exchanged via text files whose format is simple and intuitive. This means that researchers can easily change one of the components of the system with their own program, without having to modify its source code but only adhering to a set of simple file-based interfaces.
In this section, we will briefly describe all the components of a typical experiment that uses the ALIZE/LIA RAL toolkit. As such, it contains algorithms which can identify a person based on his/her voice. In general, the database contains conversational speech of 50 individuals, with large amounts of information, including feelings, a message, an identity, and 4 sentences recorded by each speaker in a silent room.
The conversations are acquired from speaker through a microphone with time duration of 40 seconds. The voice database consists of 50 speaker samples. We want to distinguish one speaker from another. The sampling frequency of the recordings was 44100 Hz, however, we have downsampled at 8000 Hz using LIA RAL package that aims at providing automatic speaker detection related programs based on the ALIZE toolkit.
In the training data set 40 second long samples were collected, while in the testing data set time duration was of 10 seconds per sample. The results of the unknown samples were compared to the training samples, obtaining a data set of 5000. The DET curve presented the results of the verification/identification of the speech performance with EER of 4.25% (see fig. 4).
Face as a biometric has many advantages. We are accustomed to recognize people based on face from the childhood through all the life, face image can be easily gathered; face recognition is a non-intrusive technique. Face recognition system consists of face detection and localization, image preprocessing and normalization, feature extraction and selection and classification. The role of face detection and localization is to find all faces in the unprocessed scene, where various factors have to be taken into account: number of faces, position, size and rotation of faces, face illumination, inner face variations (skin color, hair color, hairstyle, moustache, beard, glasses, sunglasses, facial expression), complex background, etc...
Besides detecting a whole face in the image, also detection of facial features, detection of an expression and similar tasks are of great importance. From the most general point of view face recognition methods can be divided into following groups: 1) 2) 3) holistic methods (full region of face is processed); local methods (local face features are used for recognition), local methods can be further divided into local feature-based methods and local appearancebased methods; hybrid methods
The present study adopted the 2DFace system because it
represents a good trade-off between performance and
computational complexity. The biometric reference system for 2DFace
was developed by Boazii University and is based on Principal
Component Analysis (PCA) [
The Principal Component Analysis (PCA) is a useful statistical technique that has found applications in fields such as recognition, classification and image data compression. It is also a common technique in extracting features from data in a high dimensional space. This quality makes it an interesting tool for our study. It is a systematic method to reduce data dimensionality of the input space by projecting the data from a correlated high-dimensional space to an uncorrelated lowdimensional space.
Turk and Pentland applied PCA [
The human face images we have examined are based on a set of 50 individuals. In the first step, the face images were captured from webcamera “Creative Live!Cam”, with 10 different facial appearances for each individual and resolution image of 1.3 Megapixels. The image database has been acquired in a restricted size (51 x 55 pixels) of image using normalization technique and, finally, cropping approach. The results of the unknown samples testing were compared to the testing of the training samples, thus obtaining a data set of 12500 instances. The DET curve shows the results of the verification/identification of the face performance with EER of 4.06% (see fig. 5). The authentication process is a comparison between a pre-registered reference image or template, and a newly captured candidate image or template. Depending on the correlation between these two samples, the algorithm will determine if the applicant is accepted or rejected. This statistical process leads to a False Acceptance Rate (FAR, i.e. the probability to accept a non-authorized user) and a False Rejection Rate (FRR, i.e. the probability to reject an authorized user).
Signature recognition is a complex classification problem which deals with the variation among same class signatures and differences one signature with another. Researchers have already performed rich amount of work to solve this problem. There are various techniques in signature verification such as using neural network, DCT, global features, single stroke based approach. There are two types’ signature verification
Fig. 3. General scheme for an on-line signature verifications system
methods namely on-line method and off-line signature
verification method. on-line signature recognition is also called
static signature recognition and off-line signature recognition
is also called dynamic signature recognition [
In on-line signature verification signature image is capture and analyze in real time as the person is signing it. To capture the signature in real time on-line approach uses a touch screen monitor and an electronic tablet are use to takes dynamic information for verification purpose and extract information about a signature. On-line signature verifications system records the motion of the stylus (which is also part of the sensor) at the time of signature is produced, and includes location, and velocity, acceleration and pressure on pen. The flow diagram in fig. 3 shows a general scheme for an on-line signature verifications system.
The biometric signature recognition, used in this study, has been developed by Get-int (B. Ly Van, S. Garcia-Salicetti and B. Dorizzi). This system is based on a recognition technique known as HMM (Hidden Markov Model). For the acquisition of the online signature data during the writing process, images were captured by Wacom Graphire 3 Tablet with a database set of 50 signatures. Then the X, Y coordinates and the time pen position were extracted by the tablet. Each individual has put several signatures (10). The results of the unknown samples testing were compared to the testing of the training samples, thus obtaining a data set of 12500. The DET curve presents the results of the verification/identification of the speech performance with EER of 12.65% (see fig. 6). The performance of the system is described to calculate the equal error rate (EER). EER corresponds to the point where the false accept and false reject rates are equal. Performance Criteria: the basic error measure of a verification system is false rejection rate (FRR) and false acceptance rate (FAR).
False Rejection Rate (FRRi) is the average number of falsely rejected transactions. If n is a transaction and x(n) is the verification result where 1 is falsely rejected and 0 is accepted and N is the total number of transactions then the personal False Acceptance Rate (FARi) is the average number of falsely accepted transactions. If n is a transaction and x(n) is the verification result where 1 is a falsely accepted transaction and 0 is genuinely accepted transaction and N is the total number of transactions. Both FRRi and FARi are usually calculated as averages over the entire population in a test set. Equal Error Rate (EER) is an intersection where FAR and FRR are equal at anoptimal threshold value. This threshold value shows where the system performs at its best. In this paper, the Detection Error Tradeoff (DET) curve is used to visualize and compare the performance of the system. III.
VOICE, FACE AND SIGNATURE
Using the existing unimodal recognition strategies makes it possible to design a multimodal system with the score level fusion approach. Score level fusion is referred to the combination of matching scores provided by different biometric systems. The methods used for the score fusion techniques are MIN and MAX, whereas the normalization of scores for different recognition systems is obtained by using fusion methods of the sum, the product, the max and min. Distinct feature extraction algorithms are used to check a person who gives different match scores as the output. Three biometric systems can be used to provide results with different numerical range of the output matching scores. In particular, the score of speaker recognition has a range of between +1 and 1, the score of the face recognition has a range between 0 and +1, whereas the score of the signature recognition is between 0 and 1.
For these reasons, it was necessary to create a mechanism to normalize the obtained scores.
In our case, we have selected the normalization between 0 and 1 (the range assumed by the signature, which does not require any normalization). This is obtained, as mentioned earlier, by the method of the max and min for both the voice and the face. The mean of the obtained results of three different systems had to be estimated; thus, LIA RAL calculated an arithmetic average of the scores obtained from the comparison between the model and the recordings, different for each person.
In this way, we have obtained a system of 2500 comparisons. Therefore, it is possible to proceed to the construction of the third application able to derive the scores for the four fusion methods used. The sum method adds up the scores obtained by the relation of the individual programs used, therefore, the scores are added, obtaining in output still 2500, however, obtained by the sum of the comparisons details. The result of this particular fusion method is represented as follows The value of the EER is reduced to 0.36% as in the representation of the distribution scores.
By analysing the performance of different fusion methods
we have discovered that the sum fusion methods lead to the
best performance, obtaining EER reduced to 0.36% for the
face recognition system. However, the other fusion leads to
high efficiency, considering the results produced, which is
the case of fusion technique with EER equal to 0.78% (see
fig. 7). The obtained performance in the voice-face recognition
system, with the minimum values, of EER (0.40% for the
product fusion method and 0.43% in the case of the sum
fusion method) correspond to those reported in literature. We
have used the multibiometric recognition architecture working
with multiple sources of information based on anatomical
characteristics, such as voice, signature and face. The aim of
the present research was to compare the proposed architecture
with an approach based on probabilistic neural network [
IV.
Multi-biometric systems consolidating information from
multiple biometric sources are gaining popularity as they are
able to overcome such limitations as non-universality, noisy
sensor data, large intra-user variations and susceptibility to
spoof attacks that are commonly encountered in unimodal
biometric systems. The advantages of a neural network based
approach on other approaches including statistical models are
that the ANN does not require prior knowledge of statistical
distribution of data or any influence parameter on data sources
to be specified [
Current research investigations in NN models may provide promising improvements in reliability and efficiency related issues.
In this paper, a GMM framework for multimodal biometrics has been proposed. In the present research, the distribution of scores was obtained using the fusion method of sum, applied to face, signature and speech recognition systems.
The obtained performance of the recognition system is better than the results reported in literature. Moreover, the results of the present research were acquired with considerably lower computational complexity. Fig. 5. DE T curve of error rates using biometric face recognition (left), Distribution of scores obtained using biometric face recognition (right)
The aim of this paper was to compare the proposed architecture with an approach based on probabilistic neural network.
The space of vectors of biometric indexes was orthogonally projected in subspace of lower dimension in order to delete the information carried redundantly, and, therefore, improve the performance of classification stage based on PNN in a noisy scenario. Fig. 7. DET curve of error rates using fusion method of sum applied at face, signature and speech recognition systems (left), Distributions of scores obtained using fusion method of sum applied at face, signature and speech recognition systems (right)