The growing field of Deep Learning (DL) and Deep Neural Networks (DNN) has seen significant advancements in recent years, particularly in the application of Formal Models to improve system performance and reliability. This study conducts a bibliometric analysis using data from Scopus to explore trends in formal analysis and formal verification of DNN, which are essential for safety-critical applications such as healthcare and autonomous systems. Python and R were employed for data extraction, processing, and visualization. The results emphasize increasing interest in the field, highlight prominent authors, significant author collaborations, key publications, and trends in keywords and research topics. These findings, visualized through tables and graphs, provide insights into the current research landscape and ofer guidance for future studies on integrating formal models with DL.
1. How has research on formal models in Deep Learning evolved in terms of annual scientific growth and contributing countries? 2. Who are the most influential authors in this field, and which institutions produce the most work? 3. What are the most frequent keywords and the main trends of recent years? 4. How do collaborations between authors take place?
Formal models are mathematical frameworks used to rigorously describe and analyze systems to ensure
their correctness and reliability. In the context of DNN and DL, formal models are increasingly applied
to address challenges related to robustness, interpretability, and safety. These models are used for tasks
such as probabilistic models, graphical models, and logic-based models, each of which serves distinct
purposes.
1. Probabilistic Models: These models, particularly Bayesian networks and other probabilistic
graphical models, are used to represent uncertainty and complex dependencies in deep learning tasks.
Methods like variational inference and Monte Carlo enhance model robustness and generalization,
especially by integrating probabilistic reasoning into neural networks [
Formal models in DL enhance the robustness, interpretability, and reliability of neural networks across various applications. They employ probabilistic and logic-based methods to validate models, assess behavior, and quantify prediction uncertainties while ensuring adherence to safety constraints. These models are crucial for decision-making tasks in robotics and autonomous systems, helping handle diverse data sources and improving generalization across diferent use cases, especially in safety-critical domains like autonomous vehicles and healthcare.
Data Sources: Scopus has become a leading bibliometric database, ofering extensive coverage across
disciplines and superior citation tracking—approximately 20 % more than Web of Science [
To gather recent publications on deep learning, formal models, and neural networks, we used the Scopus indexing database, renowned for its broad academic coverage. The search terms aligned with the study’s objectives, targeting research that integrates deep learning, formal models, and performance evaluation. The query executed was: ("Deep Learning" OR "DL") AND ("Deep Neural Network*" OR "DNN*") AND (("Performance") OR ("Formal Models" AND ("Formal analysis" OR "Formal Verification"))), yielding 16,207 documents across 46 columns.
Filters ensured relevance and novelty by limiting the period to 2017–2024 and including all subject areas to capture interdisciplinary contributions. Only peer-reviewed articles and journals in English were considered to maintain reliability and comparability, given that around 80% of academic articles are published in English, which dominates global research trends. Scopus’s focus on English further ensured uniformity and accessibility.
The database was processed using a Python script [
Subsequently, we analyzed publication trends to track the evolution of interest in formal models and deep learning. We identified influential authors, examined collaboration networks, and highlighted key research groups and contributors.
The data was processed using a Python script, which played a crucial role in conducting the analysis
by taking the collected data as input and leveraging various libraries and functions to facilitate data
processing and preparation. Subsequently, the analyses were performed using R software, specifically
with the Bibliometrix and Biblioshiny packages [
The graph in figure 1 shows a significant increase in scientific production in the field of formal models applied to DNN and DL between 2017 and 2023, with an estimate for 2024. The growth is particularly notable between 2017 and 2021, rising from 231 to 2,643 articles, reflecting a growing interest in this expanding field. From 2021 to 2023, the number of publications stabilizes around 3,200 articles, with a peak in 2023 at 3,283 articles. This stabilization suggests a certain maturity in the field, although the estimate for 2024 (3,125 articles) indicates a slight decline. This could point to a temporary shift in focus towards other subfields or a minor decrease in activity within the domain. Overall, the trend highlights that formal models are playing an increasingly important role in improving performance and verifying neural networks, particularly in critical applications. In total, the dataset includes 16,205 publications, underscoring the substantial volume of research dedicated to this area over the past years. Table 1 highlights the top 10 most cited articles related to the specified topic, selected from a total of 16,205 records in the dataset:
The diagram presented in Figure 2 shows that Zhang Y, Wang Y, and Li Y are the most prolific authors in the field of DNN and DL, each having published over 300 articles, likely on topics such as DNN architectures and optimization methods. Other authors, such as Wang J, Li X, and Wang X, also contribute significantly with 200 to 250 publications. The strong presence of Chinese researchers with similar names may indicate potential collaborations. Their work covers various aspects, revealing trends such as the improvement of DNN performance and the exploration of unsupervised learning.
The chart in figure 3 illustrates the scientific output of the top ten authors working on the application of formal models to DNN from 2017 to 2024. Certain authors, such as Wang X and Zhang J, stand out for their particularly high productivity and significant recognition of their work, as shown by the numerous citations they receive. Wang X, for instance, experienced a major peak in 2021, suggesting a key contribution during that period. Other authors, like Li Y and Zhang Y, show more modest output, possibly indicating specialization in specific subfields. Overall, the consistent scientific output in this area reflects the ongoing growth of research on DNNs, highlighting the importance of these authors’ contributions to developing robust and reliable methods for these technologies. Their work has a significant influence on the community, and the field is rapidly expanding with continuous innovations.
In this comprehensive analysis, our main objective was to study the changing trends in interest in DL-related topics by ranking keyword frequencies by year. Notably, “deep learning” emerged as the predominant keyword, although it was intentionally excluded from the main objective of this study. Our aim was to shed light on other keywords associated with it. For a detailed analysis of our results, please refer to Figure 4 for the revealed results.
The analysis in Figure 5 highlights the most relevant university afiliations in terms of scientific output. Tsinghua University leads with 550 articles, followed by Zhejiang University with 513. These two institutions clearly dominate research production. Next are the University of Electronic Science and Technology of China and Xidian University, with 452 and 441 articles, respectively. Other universities, such as Huazhong, Shanghai Jiao Tong, and Wuhan, also stand out with between 392 and 426 articles. Lastly, Beihang, Southeast, and Sichuan universities round out the list, each contributing around 350 articles. The data emphasizes how scientific output is concentrated in a few dominant universities.
The analysis presented in image 7 highlights the rise of Transformers, particularly after 2021, in natural language processing and machine learning. Contrastive learning is also becoming a major topic, reducing reliance on labeled data and improving model eficiency, especially in computer vision. The use of large datasets is rapidly expanding, but it poses challenges in terms of management and energy eficiency. While convolutional neural networks (CNNs) remain relevant, Transformers appear to outperform them in several tasks. Finally, a focus on integrating CNNs and Transformers, as well as optimizing performance while minimizing environmental impact, could be key moving forward.
This graph (Figure 8) presents the final output of the collaboration network analysis conducted in R using the Biblioshiny tool. The network reveals a core-periphery structure, where authors like "Zhang Y," "Wang Y," and "Li J" act as central "hubs" with numerous connections, indicating their prominence in the field. Meanwhile, a small isolated group, represented in red, is disconnected from the main network. This visualization captures the dynamics of scientific collaboration, with a dense core of highly interconnected researchers surrounded by peripheral, less-connected authors. Common names, such as "Wang" and "Li," posed challenges in distinguishing individual authors, but the structure still provides valuable insights into collaborative relationships. Due to the dataset’s large size, it could not be imported into VOSviewer, highlighting Biblioshiny’s efectiveness in handling extensive data for visualizations like this.
Our findings align with previous studies highlighting the growing importance of formal models in
enhancing the robustness, interpretability, and safety of deep learning (DL) systems, particularly in
ifelds such as healthcare and autonomous systems. For instance, the work by Chen et al. (2018),
which developed the DeepLab model for semantic segmentation, demonstrates how the use of atrous
convolution improves spatial accuracy [
The research questions outlined in the introduction have been thoroughly explored through the study’s ifndings. Below is an analysis linking each question with its respective answer:
Q1: How has research on formal models in Deep Learning evolved in terms of annual scientific growth and contributing countries? R1: Research on formal models within the deep learning domain has shown substantial growth, particularly between 2017 and 2021, indicating increased global attention. By 2023, a phase of stabilization occurred, pointing to the field’s maturity and the necessity for innovative directions or emerging subfields to catalyze further advancements. Leading contributors include China, the U.S., and key European nations, reflecting considerable research funding and focus in these regions.
Q2: Who are the most influential authors in this field, and which institutions produce the most work? R2: Influential figures such as Zhang Y., Wang Y., and Li Y. have made significant and consistent contributions. Institutions like Tsinghua University and Zhejiang University are notable for their prolific output and extensive research partnerships, enhancing their strong positions within the field. This highlights their leadership in producing impactful research and fostering influential academic collaborations.
Q3: What are the most frequent keywords and main trends of recent years? R3: Commonly recurring terms include "Deep Learning" and "Deep Neural Networks," which continue to dominate the landscape. However, there has been an increased emphasis on emerging keywords such as "transformers" and "contrastive learning," highlighting new strategies for handling large datasets and improving model performance. Transformers, with their attention mechanism, allow parallel sequence processing, enhancing speed and eficiency, especially in NLP tasks like translation [ 27]. This innovation extends to fields such as computer vision. Contrastive learning improves data representation and model generalization without needing extensive labeled data, reflecting the field’s adaptation to technological challenges [28].
Q4: How do collaborations between authors take place? R4: Author collaborations tend to be predominantly national, with a strong concentration in China, illustrating a focus on bolstering domestic research networks. This shift from earlier, broader international collaborations indicates a strategic emphasis on regional research independence and focus.
Several obstacles have impacted the quality of our analysis. While the Scopus database is comprehensive and widely recognized, it has inherent limitations. For instance, it may not encompass all relevant publications, particularly those in emerging or niche journals, leading to selection bias and limited coverage diversity. Additionally, comparisons with other databases like Web of Science and Google Scholar reveal that Scopus may prioritize specific fields, potentially skewing research findings. Keywordbased search strategies may exclude pertinent studies that use diferent terminologies, introducing language bias and restricting analysis scope. Moreover, unintended biases in interpreting results should be considered. Emphasizing specific emerging keywords might highlight current trends but not fully represent the broader academic landscape. Despite measures taken to minimize these biases, they can afect the generalizability and robustness of conclusions. A broader analysis involving multiple databases and more inclusive search strategies could help mitigate these limitations.
Formal models applied to Deep Learning (DL) and Deep Neural Networks (DNN) have gained significant attention in recent years for their potential to enhance system robustness, interpretability, and reliability. Given the wide range of applications and the rapid pace of technological progress in this field, it can be challenging to cover these developments comprehensively through traditional narrative reviews.
Our bibliometric analysis provides a detailed overview of the evolution of publications on formal models in DL, highlighting their contributions to system validation and performance improvement. Leveraging Python for data extraction and processing, and R with the Bibliometrix package for analysis, we visualized key trends, collaborative networks, and the influence of major contributors in the field. This approach ofers a holistic view of the landscape and identifies critical research hubs and influential authors.
In light of the growing significance of formal models, we recommend that future research continues to explore this domain, with particular focus on interdisciplinary approaches. Emerging techniques, such as transformers and contrastive learning, present promising opportunities to address current challenges, including enhanced model validation and energy eficiency. Future studies should investigate how formal models can integrate with these methods to boost robustness and performance, particularly in critical fields like healthcare and autonomous systems.
This work was partially supported by the LABEX-TA project MeFoGL: "Méthode Formelles pour le Génie Logiciel"
During the preparation of this work, the authors used ChatGPT and DeepL Writer in order to: Perform grammar and spelling checks. Further, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content. of deep cnn for image denoising, IEEE Transactions on Image Processing 26 (2017) 3142–3155. doi:10.1109/TIP.2017.2662206. [20] J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, J. Cai, et al.,
Recent advances in convolutional neural networks, Pattern recognition 77 (2018) 354–377. [21] L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J. Santamaría, M. A.
Fadhel, M. Al-Amidie, L. Farhan, Review of deep learning: concepts, cnn architectures, challenges, applications, future directions, Journal of big Data 8 (2021) 1–74. [22] K. Kamnitsas, C. Ledig, V. F. Newcombe, J. P. Simpson, A. D. Kane, D. K. Menon, D. Rueckert, B. Glocker, Eficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation, Medical image analysis 36 (2017) 61–78. [23] A. Mathis, P. Mamidanna, K. M. Cury, T. Abe, V. N. Murthy, M. W. Mathis, M. Bethge, Deeplabcut: markerless pose estimation of user-defined body parts with deep learning, Nature neuroscience 21 (2018) 1281–1289. [24] H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, P.-A. Muller, Deep learning for time series classification: a review, Data mining and knowledge discovery 33 (2019) 917–963. [25] A. Khan, A. Sohail, U. Zahoora, A. S. Qureshi, A survey of the recent architectures of deep convolutional neural networks, Artificial intelligence review 53 (2020) 5455–5516. [26] A. Y. Hannun, P. Rajpurkar, M. Haghpanahi, G. H. Tison, C. Bourn, M. P. Turakhia, A. Y. Ng, Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network, Nature medicine 25 (2019) 65–69. [27] R. Hanslo, Deep learning transformer architecture for named-entity recognition on low-resourced languages: State of the art results, in: 2022 17th Conference on Computer Science and Intelligence Systems (FedCSIS), IEEE, 2022, pp. 53–60. [28] C.-Y. Chuang, J. Robinson, Y.-C. Lin, A. Torralba, S. Jegelka, Debiased contrastive learning, Advances in neural information processing systems 33 (2020) 8765–8775.