In the era of digital authentication, facial verification systems have become a cornerstone of security protocols across various applications. This study explores the performance synergy from concatenated embeddings in enhancing biometric authentication accuracy. By leveraging the Celebrities in FrontalProfile dataset (CFP), we investigate whether the fusion of embeddings generated by models such as VGGFace, Facenet, OpenFace, ArcFace, and SFace can result in a more robust authentication process. Our approach is rooted in the hypothesis that the diverse strengths of these models, when combined, can address the limitations inherent in single-model systems, thus providing a more comprehensive solution to facial verification. The approach involves computing the L2 distance between normalized concatenated embeddings of an input face image and an anchor, thereby determining the authenticity of the individual. Experiments are designed to compare the performance of singular model embeddings against concatenated embeddings, employing metrics such as accuracy, False Acceptance Rate (FAR), and False Rejection Rate (FRR). One of the critical aspects of our research is the implementation of Z-Score normalization and L2 normalization processes to standardize the embeddings from different models. These normalization techniques are vital in ensuring that the diverse outputs from various models are effectively combined, maintaining balance and consistency in the feature vectors. Additionally, our methodology includes a comprehensive evaluation framework that meticulously analyses the trade-offs between computational efficiency and performance gains achieved through model concatenation. The findings of this research could significantly contribute to the development of more secure and reliable facial verification systems by using multiple existing models without the need for new model research, designing, and training. This approach not only optimizes resource utilization but also provides a scalable solution that can be readily adapted to existing systems, enhancing their security measures without extensive overhauls. Furthermore, the study's insights into the integration of model outputs could pave the way for future innovations in biometric authentication, encouraging the development of hybrid systems that combine the best attributes of various neural network architectures. This research underscores the potential of concatenated embeddings in revolutionizing facial verification technology. By harnessing the power of multiple neural network models, we can create a system that delivers superior accuracy and robustness, addressing the pressing need for advanced security solutions. This study sets the stage for further exploration into multimodel integration, offering a promising direction for future advancements in biometric authentication.
In today’s digital landscape, facial verification [
This research investigates the potential of enhancing
facial verification accuracy through concatenated
embeddings from multiple neural network [
0009-0001-4009-0202 (D. Khanin); 0000-0003-4781-7766 (V. Otenko); 0000-0001-9391-6525 (V. Khoma)
The evolution of biometric authentication technologies has
been significantly influenced by advancements in machine
learning and deep learning [
In facial verification technology, using just one neural
network model comes with certain limitations. Different
models excel in various aspects, such as accuracy, speed of
processing, and their ability to handle changes in lighting or
facial features [
The hypothesis driving this research emerges from a critical
challenge within the realm of facial verification systems: the
limitations of using single-model architectures in achieving
consistently high accuracy across diverse conditions. This
issue underscores the necessity of exploring alternative
strategies that can leverage the strengths of existing
technologies without the need for constant model
retraining, dataset updates, or the development of new
architectures [
Current facial verification systems often rely on a
singular neural network model, which may excel under
specific conditions but fall short in others. This reliance
poses a significant problem as it demands continuous
updates to the model and its underlying dataset to address
emerging challenges and maintain system performance.
Such an iterative cycle of development is
resourceintensive, requiring substantial investments in data
collection, processing, and computational power.
Additionally, the creation of new model architectures to
improve feature extraction and classification accuracy
further complicates the process, making it unsustainable in
the long run. The hypothesis presented in this study arises
from these challenges, proposing the use of concatenated
embeddings [
This research focuses on key goals designed to explore improvements in facial verification systems:
To create a system to test both individual and combined embeddings from models such as VGGFace, Facenet, OpenFace, ArcFace, and SFace, leveraging the CFP dataset for comprehensive analysis.
To measure the effectiveness of each model and
their combinations using accuracy, FAR, and FRR
[
To analyze system performance across single and combined model embeddings to identify the most effective strategies for facial verification.
To extract insights for potential system enhancements and recognize any associated challenges with multi-model embeddings.
The problem of enhancing the accuracy and robustness of
facial verification systems has been a focal point in
numerous studies due to ongoing challenges such as
spoofing, adversarial attacks, and varying conditions of
image capture. Ding and Tao (2018) addressed the
limitations of traditional face recognition approaches by
introducing Trunk-Branch Ensemble Convolutional Neural
Networks for video-based face recognition, which improved
recognition accuracy but still faced challenges in handling
dynamic and complex environments [
Li et al. (2018) focused on enhancing deep learning
features with facial texture features for improved
recognition performance, but the integration of different
feature extraction techniques remained complex and
computationally intensive [
Moon et al. (2016) developed a face recognition system
based on convolutional neural networks using multiple
distance faces, which further emphasized the necessity of
integrating various models to enhance system robustness, yet
it also indicated the increased computational demands [
Yang et al. (2019) explored federated machine learning
for face verification, addressing privacy and security
concerns while maintaining high verification accuracy.
Their research underscored the challenge of managing
decentralized data and the need for efficient data integration
techniques [
Bhuiyan et al. (2017) presented a noise-resistant network for
face recognition under noisy conditions, which highlighted
the ongoing challenge of achieving robust performance in
diverse real-world scenarios [
Recent advancements have shown that despite
significant improvements in facial verification technologies,
several unresolved issues persist. Gao et al. (2018) discussed
privacy-preserving techniques in face recognition, which
remain a critical concern in the deployment of these
systems [
The integration of multiple models to leverage their
unique strengths and mitigate individual weaknesses is a
promising approach, as highlighted by recent research on
hybrid and ensemble methods. However, this integration
introduces new challenges, such as increased computational
complexity and the need for sophisticated normalization
techniques to ensure consistent and reliable performance
[
These studies collectively underscore the necessity of integrating multiple models to leverage their unique strengths and mitigate individual weaknesses, aligning with our research objective of using concatenated embeddings to enhance facial verification systems’ accuracy and robustness. The proposed approach builds on the foundations laid by these works, aiming to address their limitations through the strategic combination of diverse neural network models.
The CFP dataset plays a pivotal role in our study, offering a nuanced exploration of facial verification across varying poses. Its construction and attributes are as follows:
Size and Volume: The dataset consists of images of 500 individuals, with 10 frontal images per individual.
Resolutions and Quality: Including a mix of resolutions and qualities, the dataset mirrors the variability encountered in real-world applications, ranging from high-definition to lower-quality images, challenging the adaptability of verification systems to varying image fidelity.
Diversity of Conditions: It spans a broad spectrum of real-life conditions— different lighting scenarios from natural daylight to artificial and low light environments, varied backgrounds from simple to cluttered scenes, and a wide range of facial expressions and poses, especially focusing on extreme profile views that pose a significant challenge to current algorithms.
Source: Images are sourced from the internet, capturing “in the wild” conditions that include a balanced representation of genders, ethnicities, and professions. This approach ensures the dataset reflects the complexity and diversity of facial appearances and expressions in everyday life.
CFP dataset examples are shown below in Fig. 1: 4 random face images for each of the 3 individuals from the dataset. This study employs various neural network models, each with unique architectures and characteristics, to determine the effectiveness of concatenated systems in facial verification. The models utilized include VGG-Face, Facenet, Facenet512, OpenFace, ArcFace, and SFace, each designed to extract and analyze facial features from images, transforming them into numerical representations known as embeddings. A comparison of architecture, embedding dimensions, training focus, and key features of each model is described in Table 1.
The concatenation system forms a pivotal component of our methodology, designed to harness the collective strengths of multiple facial recognition models. This approach seeks to enhance the robustness and accuracy of facial verification by leveraging the diverse feature representations extracted by different models. The process involves several key steps, each contributing to the formation of a comprehensive feature set that is used for facial verification: 1.
Model Selection: The first step involves selecting a set of neural network models, such as VGG-Face, Facenet, OpenFace, ArcFace, and SFace, each known for its unique approach to capturing facial features. This diversity is crucial for assembling a wide-ranging feature set.
Output Extraction: For each model, we extract the output embeddings that represent the facial features identified by that model. These embeddings are the high-dimensional vectors that encapsulate the model’s interpretation of the facial features.
Z-Score Normalization [
Concatenation: Following normalization, the embeddings from all selected models are concatenated into a single, comprehensive feature vector. This concatenated vector represents a fusion of the diverse facial features recognized by the individual models, capturing a broader spectrum of facial characteristics than any single model could.
L2 Normalization [
EER Determination: Upon calculating the L2 distances between facial image pairs, we identify the Equal Error Rate (EER), the point where the FAR and the FRR converge. Determining the EER is essential, as it represents an optimal balance point for the system’s decision threshold, minimizing both false positives and false negatives. This optimal threshold is then used to distinguish between matches and non-matches across the entire dataset, allowing for the proper measurement of verification metrics such as accuracy, FAR, and FRR.
To analyze the performance of our facial verification systems, including both single and combined models, we use three main metrics: accuracy, FAR, and FRR. These metrics help us understand the systems’ performance in correctly identifying faces.
False Acceptance Rate: FAR measures the likelihood that the system incorrectly verifies an impostor as a genuine user. It is crucial to evaluate the security aspect of the facial verification system, with lower values indicating higher security. FAR is calculated as:
Key Features Deep convolutional layers, trained on large facial image dataset, use small (3×3) convolution filters, capture fine facial details Compact embeddings optimize distance between similar/dissimilar faces, use triplet-based loss function to enhance verification accuracy Higher-dimensional embeddings, capture more nuanced features, an extension of Facenet with increased embedding size for a richer representation Balances accuracy and computational efficiency, suitable for real-time applications, a lightweight model designed for practical use on modest hardware Enhances discriminative power, improves geometric accuracy of feature space, uses additive angular margin loss to manage class margins Efficient handling of scale issues, rapid and accurate recognition, perform well on high-resolution images, notable efficiency and accuracy, especially on large datasets = ( + ), (1) where FP is the number of false positives, and TN is the number of true negatives.
False Rejection Rate: FRR assesses the frequency at which the system wrongly rejects an authentic match. This metric is important for understanding the usability of the system, as a high FRR may lead to user frustration. Lower FRR values are desirable, indicating better performance. FRR is calculated as:
= ( + ), (2) where FN represents false negatives, and TP denotes true positives.
Accuracy: This metric measures the overall effectiveness of the facial verification system. It is calculated as the ratio of correctly identified instances (both true positives and true negatives) to the total number of instances. High accuracy indicates that the system is effective in correctly verifying facial identities. The formula for accuracy is given by:
+ = ( + + + ), (3) where TP represents true positives, TN denotes true negatives, FP stands for false positives, and FN signifies false negatives.
Together, these metrics provide a comprehensive overview of the system’s performance, offering insights into its accuracy, security, and usability. By evaluating these metrics, we can make informed decisions on optimizing model configurations and improving facial verification systems.
Experiments were conducted on a defined technical framework comprising specific hardware and software components.
Hardware Configuration: MacBook Pro 16 with an M1 Pro processor and 16GB RAM, offering enough computational power for handling operations.
Software Configuration:
Python 3.11: Selected for its widespread support for data analysis and machine learning tasks.
Tensorflow-metal 1.1.0: Optimized for the M1 Pro, enhancing machine learning computation speeds. OpenCV-python 4.9.0: Utilized for image processing tasks such as loading, resizing, and cropping.
Deepface 0.0.83: A library providing access to several facial recognition model weights (VGGFace, Facenet, OpenFace, ArcFace, SFace) and their functionalities, streamlining the embedding extraction.
Data preprocessing is a crucial initial phase in our experiment, ensuring facial images are properly conditioned for analysis by various neural network models. Here’s an outline of the preprocessing steps undertaken:
Loading Images: Images are first loaded in RGB color space, retaining their essential color information which is crucial for accurate analysis of facial features.
Scaling Pixel Values: To standardize the images, pixel values for each color channel are scaled to a range from 0 to 255.
Model-Specific Normalization: Depending on each
model’s requirements, specific normalization techniques are
applied to the image data to match the conditions under
which the models were trained [
For Facenet model:
− ( ) = ( ) , (4) where mean and std are the mean and standard deviation of the image’s pixel values, respectively. For Facenet512 and ArcFace models:
127 For the VGGFace model: = − 1,
(5) 93.5940 = − 104.7624 , (6)
129.18633 this formula represents the subtraction of mean values for each color channel (R, G, B) based on VGGFace1 training data. .
(7)
In the evaluation phase of our experiments, each neural network model was assessed individually to establish its performance on the CFP dataset. A crucial part of this assessment involved determining the EER for each model, which provides a threshold at which the rate of false acceptances is equal to the rate of false rejections.
The process began with the calculation of distances between facial embeddings for both genuine and impostor pairs. Following this, we computed the EER for each model, which then served as a basis for determining the corresponding accuracy at the EER point and the best overall accuracy achieved by the model. These metrics give us insight into the models’ capabilities in facial verification tasks under the diverse conditions presented by the CFP dataset.
The results of the singular model evaluations are summarized in Table 2.
The exploration of concatenated clusters is an integral part of the research, aimed at harnessing the collective strengths of multiple neural network models to enhance facial verification accuracy. This section discusses the evaluation of clusters formed by all possible combinations of six distinct models: VGG-Face, Facenet, Facenet512, OpenFace, ArcFace, and SFace. Each cluster is identified by a unique ID for ease of reference and comparative analysis. The evaluation methodology began with the determination of the EER for each cluster. EER serves as a crucial metric for assessing the balance between security and user convenience. By employing this threshold, we derived the EER-based accuracy and the best accuracy achievable across a range of thresholds, thereby quantifying the models’ verification capabilities. The metrics graphs and threshold range analysis examples are shown in Fig. 2.
Following the graphical analysis, the performance results for each cluster are presented in Table 3. This table arranges the EER accuracy and the best accuracy observed for each cluster in 57 combinations.
In the evaluation of concatenated clusters, our data indicates that selected clusters achieve a marginal increase in accuracy over the highest-performing individual model, Facenet512. Specifically, clusters 5, 9, 11, 25, 26, 27, 45, and 55 demonstrate a modest enhancement, improving upon the best singular model’s accuracy by approximately 0.23%. While this improvement showcases the potential advantages of model concatenation, it is crucial to consider the computational trade-offs associated with such a strategy.
The study’s exploration into the performance of facial verification models, individually and in combined clusters, has revealed several key insights:
Impact of Model Pairing on Performance: Our findings highlight a notable trend where clusters combining models with lower initial accuracy see significant performance boosts. For instance, pairing OpenFace with Sface(Cluster ID 13) resulted in a 4.33% increase in accuracy, achieving an 86.13% rate. This contrasts with clusters of high-performing models, which, on average, only show about a 0.5% improvement in accuracy. This observation suggests that strategic pairing, especially involving models with varied strengths, can effectively compensate for individual weaknesses.
Variable Outcomes from Mixed Model Clusters: Not all model combinations lead to positive outcomes. In some cases, such as the cluster of Facenet and VGG-Face(Cluster ID 0), the resulting accuracy was slightly lower than that of the Facenet model on its own. This points to the complexity of model interactions within clusters and indicates that combining models does not guarantee enhanced performance and may result in suboptimal results in certain configurations.
Considerations on Computational Efficiency: While some model clusters achieve minor improvements in accuracy, like Facenet with Facenet512(Cluster ID 5) with a 0.23% increase, the requisite computational resources increase significantly. This raises important considerations about the cost-benefit ratio of employing concatenated models, especially when the gains in performance are marginal compared to the added computational demand. Maintaining High Accuracy and Security: It is noteworthy that both individual and clustered models achieving the highest performance were able to maintain their accuracy without any false acceptances on the CFP dataset. This demonstrates their potential in scenarios demanding high security, where maintaining accuracy without compromising on false acceptance rates is crucial.
Strategic Composition of Clusters for Optimal Performance: The analysis further reveals that the most successful clusters often include a combination of the top two performing models along with a lower-performing one. This composition suggests that the diverse feature recognition capabilities of the combined models contribute to a more comprehensive analysis, thereby enhancing the overall system’s performance.
Throughout this research, we encountered several challenges that impacted both the implementation of our experiments and the analysis of results:
Input Data Normalization: For effective performance, each neural network model requires input data to be normalized according to the specific training data it was developed with. This normalization process involved adjusting the color space and scaling for each model to match its training conditions. We successfully applied model-specific normalization for most of the models, ensuring that the input data closely mirrored the conditions under which the models were originally trained.
Z-Score Normalization for Model Output Embeddings: Given the variance in scale and distribution of embeddings across different models, a significant challenge was standardizing these embeddings for consistent comparison. By implementing Z-Score normalization on each embedding vector, the embedding was adjusted to have a mean of 0 and a standard deviation of 1. This crucial step allowed us to mitigate the disparities across model outputs.
Heavy Computation Without Heavy Server Resources: The computation required for generating embeddings for 57 clusters, along with individual model evaluations, was significant. To manage this, the caching mechanism [31] was implemented for embeddings post-system setup. The strategy enabled the reuse of embeddings across different clusters and singular model experiments, saving dozens of hours in computational time.
Poor Accuracy for OpenFace and SFace Models: The lower-than-expected accuracy for OpenFace and SFace models raised concerns. This may have resulted from inaccurate normalization information or deviations from the default training data used in these model weights. While this paper did not directly address enhancements to these models’ accuracy, identifying the potential causes paves the way for future improvements.
Average Size and Resolution Dataset While the CFP dataset was sufficiently comprehensive for our experimental purposes, its size and the variability of its data presented limitations. A larger and more diverse dataset could potentially reveal insights and issues not observed with the CFP dataset used in the study. This acknowledgment serves as a recommendation for future research directions to explore more extensive datasets for a deeper analysis.
The findings from the experiments offer valuable insights into the performance synergy of employing concatenated model clusters for facial verification systems. While several clusters achieved incremental improvements in accuracy, the requisite increase in computational resources was significant. For applications prioritizing computational efficiency, singular models like Facenet or Facenet512, which provide high accuracy without substantial computational overhead, might be more advisable. Specifically, the cluster combining Facenet and Facenet512 (Cluster ID 5) presents a compelling option, marginally outperforming the accuracy of the Facenet singular model by 0.23%, achieving a 97.68% accuracy rate. This slight improvement might justify the additional computational resources in scenarios where maximizing accuracy is paramount.
In contexts where the verification system can accommodate extended inference times and has access to extended computational power, employing model clusters could be beneficial. For verification systems bound by computational and time constraints yet seeking to improve upon the accuracy provided by singular fast-inference models like OpenFace, forming clusters with other rapidinference models offers a strategic solution. For example, pairing OpenFace with SFace led to a significant 4.33% accuracy increase over the singular OpenFace model, achieving an 86.13% accuracy rate. This strategy allows for a balanced enhancement in accuracy while maintaining essential high-speed inference capabilities, suitable for applications where both efficiency and accuracy are valued.
The exploration of concatenated model clusters in facial verification creates numerous opportunities for future research. A promising direction involves analyzing the specific features within each model’s embeddings that most influence verification decisions. By identifying and prioritizing these impactful features, it may be possible to filter out less relevant or noisy features from model embeddings [32]. This approach holds potential not only for singular model systems but could notably enhance the performance of clustered model systems by focusing on the combination of the most determinant features for L2 distance calculations.
Future research could also explore the efficiency of alternative distance metrics such as Cosine [33] and L1 distances. These metrics may produce different distributions, thresholds, and ultimately accuracies for model clusters, offering new insights into the optimization of verification systems. Additionally, further investigations could evaluate how these systems scale and perform under larger, higherquality datasets with more varying conditions, potentially uncovering benefits not observed in the current dataset.
Given that certain models are highly dependent on the alignment of facial images, integrating dynamic alignment techniques tailored to each model within a cluster could improve accuracy. This personalized approach to face alignment may optimize the performance of each model’s contributions to the cluster. The initial success of combining lower-performing models with fast inference rates suggests a valuable strategy for developing efficient verification systems suited for embedded environments. Future work could focus on identifying and testing combinations of efficient models to create a verification system that balances accuracy with the computational speed necessary for real-time applications in constrained environments.
In conclusion, the decision to employ singular models or concatenated clusters should be guided by the specific requirements and constraints of the facial verification system in question. The strategic composition of clusters, balancing between computational efficiency and marginal gains in accuracy, remains a critical consideration for the deployment of robust and effective biometric authentication solutions. Additionally, the model of a decoy system based on dynamic attributes for cybercrime investigation, as proposed by Vasylyshyn et al., offers a novel approach to enhancing system security and can be integrated into future research to address emerging threats [34].