Nowadays, a vast amount of data flows through networks, with many users inadvertently agreeing to their personal data being processed by network providers, frequently lacking comprehension regarding its management or sharing among various entities. Various network data analytics tools have been introduced in recent years, aiming to simplify visual representations for network trafic analysis. However, these platforms often fail to account for the diverse technical knowledge and cyber risk awareness levels of modern Internet users. To address this issue, we propose the introduction of a Large Language Model (LLM)-based conversational agent with a dynamically structured visual platform. This combination allows for the assessment of users' comprehension through adaptive questioning, enabling personalized interface adaptations that enhance user comprehension and engagement. In this paper, we discuss the architecture and components of the proposed solution, emphasizing the importance of adaptive interfaces in enhancing user experience and fostering security awareness. Through the incorporation of LLMs for human knowledge assessment, our approach endeavors to craft a more personalized and eficient visual platform for analyzing network trafic and cyber threats.
The exponential proliferation of Internet-connected devices has precipitated a monumental surge in the daily volume of data generated. The advent of the Internet of Things (IoT) has not only expanded the sheer quantity of networked devices but has also diversified their types and functions. From ubiquitous smartphones and smartwatches to cutting-edge smart cars, these devices incessantly produce data packets stemming from everyday interactions, seamlessly transmitting them across the vast expanse of the Internet to network providers that serve as nodes of the intricate web of connectivity. According to the latest statistical analysis1, 328.77 million terabytes of data is produced daily, culminating in 120 zettabytes annually.
This data contains valuable insights into user behavior, providing a fingerprint of their activities,
preferences, and desires. Although nowadays most network trafic is encrypted utilizing several
techniques, such as TLS, SSL, HTTPS, etc. [
In this scenario, it is crucial to provide users with solutions that can properly assess and hopefully
increase their level of awareness regarding how easily online activities could afect their privacy and
make them susceptible to cyber-attacks. To this end, the literature has seen the rise of several platforms
whose purpose is to monitor their activities and alert users to the risks associated with the improper use
of Internet-connected devices [
Such platforms could significantly play a role in enhancing users’ awareness; however, all of them
considerably lack tailorability with respect to the mental model of the user approaching them. In fact,
with the enormous expansion of the Internet and the introduction of connected features in devices
of all kinds, the pool of users who need to use these devices has grown considerably, and with it the
diferent types of technical knowledge base, awareness of cyber risks, and so on. In fact, the diference
between novice and expert users, and the need to adapt the interaction process accordingly, has been
the subject of recent research in the literature [
Visual interfaces that adapt to user needs are an emerging area of research aimed at enhancing user
experience and performance [
In this position paper, we present an overall architecture of an LLM-based conversational agent for human knowledge assessment in conjunction with a dynamically structured visual platform to assist users in analyzing network trafic and the attacks to which they may be susceptible.
In this section, we present several network data analytics tools presented in the literature whose purposes are to visualize and summarize network trafic through simplified visual metaphors.
One of the pioneering approaches documented in literature is FlowScan [22], which scrutinizes and presents insights from flow data extracted by Internet Protocol routers. It comprises Perl scripts and modules, serving as the cohesive element integrating various freely available components, including a flow collection engine, a high-performance database, and a visualization tool. Once integrated, the FlowScan system generates graphical images suitable for web pages, ofering a continuous, nearly real-time depiction of network trafic across a network’s perimeter.
In [
Nfsight [23] is a tool aimed at enhancing network awareness with three main features: passive identification of client and server assets, a web interface for querying and visualizing network activity, and a heuristic-based intrusion detection and alerting system. The Service Detector identifies endpoints using heuristics and Bayesian inference from NetFlow flows, while the Intrusion Detector flags suspicious activity with graphlet-based signatures.
NetMod [24] is a tool ofering detailed analysis of system performances for designers working on large interconnected local area networks. Tested on a campus-wide network, NetMod employs simple analytical models and a user-friendly interface.
MVSec, as described in [25], is a visual analytics system assisting analysts in comprehending information flows over secure networks. This system facilitates data fusion activities among heterogeneous datasets using various visual metaphors. The authors established multiple coordinated views to enable analysts to characterize loud events, uncover subtle events, and explore relationships within datasets. Case studies demonstrate MVSec’s capability in constructing analytical storylines of networking and understanding network changes.
The study by Attipoe et al. [26] examined 13 network visualization tools to delineate their strengths and weaknesses. Employing qualitative coding as part of their research methodology, they extracted metrics from the advantages and disadvantages of these tools. Their aim is to aid analysts in constructing evaluation methodologies for measuring visualization tool efectiveness via usability studies.
In [27], Constantinescu et al. introduce a prototype 3D visualization system for real-time monitoring of networked devices’ status (wired, wireless, IoT devices) and network dynamics ("pulse"), including configuration, load, trafic, abnormal events, and suspicious connections. Users can intuitively visualize network status from any location on the Internet, including mobile devices, and receive alerts via short text or instant messages for significant network events.
Other approaches reviewed in the literature proposed solutions to increase awareness by alerting
users to phishing attacks, the type of attack to which end users are more susceptible. For example, the
authors in [28] propose the development of a real-time tool to detect and distinguish phishing websites
from safe ones using machine learning techniques, mainly linear regression, multinomial NB and logistic
regression. The aim is to increase cybersecurity awareness by preventing users from accessing risky
URLs and protecting their personal data. [
Although all of the tools discussed in this section provide important support in improving the level of user awareness and knowledge of risks, none of them contemplates the possibility of preemptively assessing the end user’s level of knowledge, and subsequently making the decision to adapt the content of the information shown based on the user’s assessment, which limits the use of the platforms exclusively to that group of users who can understand them when shown. In the following, we propose the addition of a dynamic assessment module with consequent adaptability of the tools to the user’s level of knowledge.
In this section, we describe the features we envisage in our proposal regarding the introduction of a module that should interpose between the user and the network trafic visualization platform. In our vision, the purpose of this module is to provide a mechanism to dynamically assess the level of basic technical knowledge, i.e., to present the user who approaches it with a series of questions of diferent natures to determine the level of knowledge he or she has. The dynamic nature of the approach lies in its adaptability to the type of questions asked, based on the answers given by the user. In fact, a series of wrong and/or inaccurate answers is an indication of important lacks in that particular aspect, which is why certain features of the platform need to be removed or simplified to make the user’s experience easier. For example, if the user responds "I don’t know" or gives an incorrect answer to the question "What is a graph?", this may be suficient to provide suficient guidance not to use such a visual metaphor to present such information to the user. However, such an approach runs the risk of reducing the assessment process to a checklist, where a set of questions about a particular concept is associated with a particular answer and an interface adaptation. This poses the danger of oversimplifying the assessment process, potentially resulting in inaccurate evaluations. In fact, to return to the graph example, the user may not even be aware of the term "graph," but he or she may still be able to interpret the visual metaphor correctly. Verifying this possibility is something that can be done if, for example, the user’s ability to understand the representation of connected nodes and arcs has been inferred from the previous questions.
For this reason, the knowledge assessment is based on the need for something that is necessarily non-deterministic, and for which the determination of the next question to be asked for the evaluation phase takes into account the direction of the interaction. In this scenario, the use of complex natural language models turns out to be a particularly suitable choice for accomplishing this kind of task. In fact, it may be possible to set up the evaluation phase as a simple conversation between the user and what is essentially an LLM-based conversational agent. Conversational agents powered by LLMs are increasingly sophisticated tools that simulate human-like interactions and support a variety of applications, from language learning to mental well-being. LLMs are used in language learning, acting as virtual teachers or conversational partners to enhance vocabulary, pronunciation, and conversational skills [29]. For instance, fine-tuned LLMs demonstrated the ability to perceive and generate multimodal content, indicating a move towards models that can handle multiple modalities seamlessly [30]. LLMs have been used for conversational interaction with mobile UIs, allowing for language-based mobile interactions without the need for task-specific datasets and models [ 31]. Moreover, personality traits can be measured and shaped in LLM outputs, which is important for the efectiveness of communication in conversational agents [32]. Finally, in conversational recommender systems, LLMs can provide personalized recommendations and engage in multi-turn dialogues, despite challenges in understanding and controlling complex conversations [33]. To seek our goal, in this manuscript, we will discuss a way to achieve the human knowledge assessment inspired by the type of approach pursued in [32], where the authors used LLMs to conduct an assessment of a user’s personal traits. They did this by instructing LLMs through a prompt engineering activity. Subsequently, they pinpointed basic components that specify the behavioral requirements for LLMs to fulfill the given task. In this position paper, we outline the primary components that a prompt should encompass for efective assessment: • Task Description - Describes the type of analysis to be performed, for example, assessing the level of knowledge of a particular term, assessing the level of awareness of a particular cyber-attack, and so on. • Target Topic - Indicates the domain of terms, or aspects, on which the assessment is to be carried out. • Test Instruction - Summarizes the features of which the test is to be composed, particularly the manner in which it is to be administered, the number of questions to be asked, and the manner in which the result of the test administered is to be reported.
User
1 4
LLM-based Human Knowledge Assessment
Module
Assessment report 2
Tailoring Module 3
5 Visual Platform
In the following, we outline an example of a preliminary engineered prompt for assessing the user’s knowledge about graphs as a visual metaphor for a specific task.
"Your task is to assess my ability to understand graphs as a medium to represent how packets are transferred between network providers. Ask questions that can become easier or harder according to how I answer each question. After 5 questions, output your decision about my knowledge level." An initial analysis of the interaction process resulting from this prompt reveals the LLM’s adeptness (in this specific case ChatGPT) in conducting evaluations by posing diverse questions to grasp the potential representations of nodes and arcs in depicting the exchange of network packets between providers.
Figure 1 provides an overview of the pipeline of components engaged in the proposed approach. Initially, the process starts with the assessment phase, wherein an interaction, whether voice or text-based, occurs between the user and the LLM-based Human Knowledge Assessment Module (1). Following the assessment phase, the generated report determining the user’s level of knowledge and understanding is transmitted to the Tailoring Module (2), which adapts and refines the interface mechanism of the visual platform based on the insights derived from the report (3). Subsequently, the user gains access to a platform featuring visual elements and interfaces aligned with their comprehension level, fostering a more efective user experience (4). Additionally, we envision ongoing communication between the platform and the tailoring module during the user-platform interaction, enabling further interface modifications based on the user’s interactions with the platform (5).
In conclusion, our proposal outlines a dynamic approach to assessing users’ technical knowledge through an intermediary module that engages users in a conversational evaluation. By leveraging sophisticated language models, such as ChatGPT, this module can adaptively pose questions based on previous user responses, enabling a nuanced understanding of user capabilities beyond mere checklist assessments. This approach aims to enhance user experiences with visualization platforms by tailoring interface adaptations to individual skill levels, ensuring efective and user-friendly interactions. Through prompt engineering activities inspired by prior work in personality assessment, we advocate for thoughtful integration of language models into knowledge assessment processes, fostering more personalized and engaging user interactions in diverse applications.
This work has been supported by the Italian Ministry of University and Research (MUR) and by the European Union - NextGenerationEU, under grant PRIN 2022 PNRR "DAMOCLES: Detection And Mitigation Of Cyber attacks that exploit human vuLnerabilitiES" (Grant P2022FXP5B). [17] J. Eisenstein, A. Puerta, Adaptation in automated user-interface design, in: Proceedings of the 5th
International Conference on Intelligent User Interfaces, 2000, p. 74–81. [18] F. Chiossi, J. Zagermann, J. Karolus, N. Rodrigues, P. Balestrucci, D. Weiskopf, B. V. Ehinger, T. M. Feuchtner, H. Reiterer, L. L. Chuang, M. Ernst, A. Bulling, S. Mayer, A. Schmidt, Adapting visualizations and interfaces to the user, it - Information Technology 64 (2022) 133 – 143. [19] J. Zakraoui, W. Zagler, A logical approach to web user interface adaptation, in: A. Holzinger, K.-M.
Simonic (Eds.), Information Quality in e-Health, 2011, pp. 645–656. [20] M. Akazue, B. Ojeme, N. Ogini, User interface adaptability for all users, International Journal of
Natural and Applied Sciences 6 (2010). [21] S. Firmenich, A. Garrido, F. Paternò, G. Rossi, User Interface Adaptation for Accessibility, 2019, pp.
547–568. [22] D. Plonka, FlowScan: A network trafic flow reporting and visualization tool, in: Proceedings of the 14th Systems Administration Conference (LISA 2000), 2000. [23] R. Berthier, M. Cukier, M. Hiltunen, D. Kormann, G. Vesonder, D. Sheleheda, Nfsight: netflow-based network awareness tool, in: Proceedings of the 24th International Conference on Large Installation System Administration, USENIX Association, 2010, p. 1–8. [24] D. W. Bachmann, M. E. Segal, M. M. Srinivasan, T. J. Teorey, Netmod: A design tool for large-scale heterogeneous campus networks, IEEE Journal on Selected Areas in Communications 9 (1991) 15–24. [25] Y. Zhao, X. Liang, X. Fan, Y. Wang, M. Yang, F. Zhou, Mvsec: multi-perspective and deductive visual analytics on heterogeneous network security data, Journal of Visualization 17 (2014) 181–196. [26] A. E. Attipoe, J. Yan, C. Turner, D. Richards, Visualization tools for network security, Electronic
Imaging 28 (2016) 1–8. [27] Z. Constantinescu, M. Vlădoiu, G. Moise, Viznet—dynamic visualization of networks and internet of things, in: Proceedings of the 15th RoEduNet Conference: Networking in Education and Research, IEEE, 2016, pp. 1–6. [28] A. K. Jha, R. Muthalagu, P. M. Pawar, Intelligent phishing website detection using machine learning,
Multimedia Tools and Applications 82 (2023) 29431–29456. [29] F. Xiao, P. Zhao, H. Sha, D. Yang, M. Warschauer, Conversational agents in language learning,
Journal of China Computer-Assisted Language Learning (2023). [30] D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, X. Qiu, SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities, in: H. Bouamor, J. Pino, K. Bali (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 15757–15773. [31] B. Wang, G. Li, Y. Li, Enabling conversational interaction with mobile UI using large language models, in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, 2023. [32] M. Safdari, G. Serapio-Garcia, C. Crepy, S. Fitz, P. Romero, L. Sun, M. Abdulhai, A. Faust, M. J.
Matari’c, Personality traits in large language models, ArXiv abs/2307.00184 (2023). doi:10.48550/ arXiv.2307.00184. [33] L. Friedman, S. Ahuja, D. Allen, Z. Tan, H. Sidahmed, C. Long, J. Xie, G. Schubiner, A. Patel, H. Lara, B. Chu, Z. Chen, M. Tiwari, Leveraging large language models in conversational recommender systems, ArXiv abs/2305.07961 (2023). doi:10.48550/arXiv.2305.07961.