Large Language Models (LLMs) are now a relevant part of the daily experience of many individuals. For instance, they can be used to generate text or to support working duties, such as programming tasks. However, LLMs can also lead to a multifaceted array of security issues. This paper discusses the research activity on LLMs carried out by the ICAR-IMATI group. Specifically, within the framework of three funded projects, it addresses our ideas on how to understand whether data has been generated by a human or a machine, track the use of information ingested by models, combat misinformation and disinformation, and boost cybersecurity via LLM-capable tools.
1. Introduction to recognize deep fakes, to understand whether a model or to enforce IP can also be envisioned for generative has been cloned, or to track usages in Machine-Learning- models, with a particular focus on large language models. as-a-Service deployments [3]. Even worse, problem of There are essentially two scenarios that are relevant in exploiting unauthorized content during training or in this respect. The first scenario is relative to the oppordeployment needs to be specifically addressed. The third tunity to mark generated text in a way that it can be research action is funded by the project "Limiting MIsin- easily recognized. Watermarking can be employed in formation spRead in online environments through multi- this context to embed the watermark within the output modal and cross-domain FAKe news detection - MIRFAK", of the LLM and, thus, distinguish between the data genwhich aims at developing an innovative content verifi- erated by a human and those produced by a machine. cation tool, delivering solutions for news verification on The objective here is to enforce IP protection as well as social media and online platforms. Within the project, to claim ownership on the generated data. The second we aim at exploring the potentials and risks of LLMs scenario is relative to the problem that such generative associated with misinformation. models can deliver malicious content. To mitigate po
In this work, we outline our research agenda on these tential harm caused by such generated data, it is crucial topics, which is devised in three directions: i) we present to develop methods to identify content generated by a mid-term challenges for using LLMs to solve security- machine, when a watermark is not embedded. It is worth related issues; ii) we discuss how watermarks can be noting that the generation of malicious content can be applied to LLMs to mitigate attacks aiming at stealing both unintentional or intentional. Unintentional generinformation or disseminating fake news; iii) we showcase ation may happen due to the stochastic nature of such the gaps to be filled to make LLMs a real asset for the generative models, which causes the phenomenon of halInternet. lucinations (i.e., unrealistic or imaginary content). By
The rest of the paper is structured as follows. Section contrast, intentional generation is typically done by a 2 deals with the problems of understanding whether the malicious threat actor, who pushes the generative model output has been generated by an LLM and of tracking its to obtain mischievous data. In both cases, the generprovenance, while Section 3 considers usage violations, ated data could be of high quality, infusing trust among such as unauthorized harvesting of data for training mod- readers eventually forcing them to fall into error or forels. Section 4 discusses challenges and opportunities rel- ward the content, e.g., through sharing functionalities ative to the adoption of LLMs in the context of online of online social networks. Our research in this context social platforms and debates. Section 5 discusses the aims at developing methods to identify contents generadoption of LLMs in assessing cybersecurity risks related ated by a machine through a language model. We are to systems and infrastructures in containerized environ- interested both in devising watermarking schemes and ments. Lastly, Section 6 concludes the work and portrays in the more general challenge relative to the problem of some prospected action points. devising predictive methods for discriminating generated data. Besides, this research activity is aligned with the current requirements enforced by the recently released 2. Are the Data Generated? European AI Act1. The latter in fact introduces specific transparency obligations to ensure that humans are inOne of the main goals of our research is to investigate formed when necessary, to ensure trust, and in particular, challenges and solutions for protecting the Intellectual that AI-generated content is identifiable. Property (IP) of the Machine/Deep Learning (ML/DL) The research approaches to this topic are quite recent. models as well as of the dataset used for the training To the best of our knowledge, the first LLM watermarkphase [4]. Moreover, we also aim at considering tech- ing technique for distinguishing human-generated from niques to mark the output produced by ML/DL services, machine-generated texts was proposed by Kirchenbauer for instance, to understand whether an attacker “cloned” et al. [6]. In text generation, language-based models prothe model through multiple remote invocations. Specifi- duce a probability distribution over a vocabulary, i.e., cally, we are interested in techniques that allow the cloak- the set of words or word fragments (i.e., tokens), used ing of secret information within the contents we want to for predicting the most likely next word based on the protect. In this respect, an emerging research line consid- previous ones. The authors propose to alter such distriers watermarking techniques, i.e., arbitrary pieces of data bution, in order to promote sampling of specific tokens. that are embedded within the item to deliver and that The occurrence within a given statistical significance of are dificult to recognize besides proprietary decryption such tokens characterizes the watermark within the text. schemes. Such mechanisms are common with images One of the main limitations of this approach is the genand multimedia objects [5] and can be used to embed control data within ML/DL models.
Techniques used to prevent unwanted/unfair usages 1https://digital-strategy.ec.europa.eu/en/policies/ regulatory-framework-ai eration of low-quality texts in contexts characterized by and overcome MIA’s issues related to large dataset and relatively deterministic content, such as code snippets the intrinsic randomness of LLMs. or structured text. Lee et al. [7] refine the approach by ensuring that sampling is only focused on high-entropy tokens. 4. Fighting Fire with Fire:
One of our research objectives is to generalize these Generative AI to promote approaches to other generative models, such as Difusion Online Safety Models or Generative Adversarial Networks (GANs). In addition, the analysis of the distribution of generated data, and its comparison with that of real (not synthetic) data can also be exploited for devising predictive models aimed at automatically detecting the reliability and authenticity of data.
LLMs are showcasing remarkable abilities in various Natural Language Processing tasks, making them a highly potent and beneficial tool for everyday life. However, alongside their appealing strengths and widespread adoption, a significant concern is arising regarding their potential role in amplifying the generation and dissemination of 3. Have You Stolen My Data? misinformation and disinformation. Generative AI technology has significantly empowered malicious actors to Membership Inference attacks (MIAs) [8] aim to pre- produce fake content, which can be disseminated across dict whether a data sample was included in the training online social networks and lead to detrimental phenomdataset of a machine learning model. These attacks serve ena, e.g., manipulating public discourse, disseminating to evaluate the privacy vulnerabilities present in machine hate speech, and sharing fake content. learning models, like in Neural Networks [9], GANs [10] As a remarkable example, in 2016 Microsoft released and Difusion Models [ 11]. Formally, the goal of a MIA the Tay chatbot, which triggered further controversy is to infer whether a given data point was part of the by posting inflammatory and ofensive tweets via its training dataset for model by computing a mem- Twitter account, leading Microsoft to shut down the serbership score (; ). This score is then thresholded to vice within just 16 hours 2. More recently, other works determine a target sample’s membership. assessed the role of bots and AI agents in conveying
Membership inference attacks exploit the tendency of and amplifying online discourse about racism and hate the models to overfit their training data and hence exhibit speech [14, 15], drawing further attention to this sensitive lower loss values for these elements. A first and widely topic. Thus, as underscored by [16], the scale, velocity used attack is the LOSS attack [12], in which samples and accessibility of generative models present compelling are classified as training members if their loss values are challenges for online platforms, potentially inundating lower than a fixed threshold (that is, (; ) is defined them with a massive amount of fraudulent material and in terms of ℒ(; )). unpredictable social consequences. While policy makers
Recent works aim to design and improve MIAs for are actively engaged in regulating the use of GenAI tools, LLMs. In this case, MIAs consider a target model the eficacy of these measures remains uncertain. In rewhich gives as output a probability distribution of the sponse, our research group is working towards leveraging next token given a prefix as input, P(|0 . . . − 1; ). Generative AI to enhance online safety. Our objective is The goal of MIA is hence to infer whether the target sam- to reuse the same technology used to contaminate online ple = 1 . . . of tokens has been considered in the discussions for a beneficial purpose in a controlled envitraining set. Duan et al. [13] consider several member- ronment. For instance, [17] demonstrated the potential of ship inference attacks and show that they just outperform a GPT2-like model in crafting tailored responses to comrandom guessing for most settings across diferent LLM bat misinformation regarding the COVID-19 pandemic. size and domains. They also argue that MIA is dificult Despite this first promising result, there are numerous on LLMs because of diferent key reasons. These include overlooked opportunities for harnessing GenAI tools to the dificulty of handling LLMs pre-trained over billions aid online safety. One such opportunity involves the and trillions of tokens, or the overlap typically exhib- development of automated agents capable of serving as ited by the underlying token distributions that can be "peace-builders" within online discussions. We aim to observed in natural language documents, irrespective of train a large language model to generate textual content their training data membership. that, once injected within online social media platforms,
Our research agenda is aimed at extending and lever- can help mitigate polarization and disagreement. aging the current membership inference games, by in- This research line is interesting and open to novel vestigating adversarial approaches in order to force the and original developments, but it also faces considerable LLM to generate copyrighted text. In this way, we define challenges. A trivial remark is to carefully consider the a framework that can demonstrate copyright violations ethical implications of using GenAI tools for online safety tem. Thus, our approach aims to expedite threat to ensure responsible use. Second, there are considerable response when integrating human expertise into technical challenges regarding the training and/or fine- the learning loop of the model, by using post-hoc tuning of these large models due to scalability concerns. explanation tools to support the operator in valiThird, evaluating the efectiveness of GenAI interven- dating the attack and guiding the learning of the tions in promoting online safety can be demanding and model. could require a multi-disciplinary approach involving • Data enrichment. Another critical aspect inexperts from fields such as psychology and sociology. volves the potential use of LLMs to enhance the
Another compelling line in our research agenda is to security of Internet-wide infrastructures. Numerdefine the aspects to take into account when analyzing ous protocols and services rely heavily on texthe role of LLMs in this context. We are interested in ex- tual information, such as URLs or configuration ploring the role of LLMs in contrasting the phenomenon data. LLMs can be exploited in generating test of false information spreading at diferent levels: detec- cases, particularly for automating periodic assesstion, mitigation, intervention, and attribution. Our efort ments aimed at detecting potential deviations in is to improve the fake detection models under the con- the security posture of a deployment. For examstraint of scarcely labeled data, which is a common con- ple, recent research showcased LLMs’ capability dition in real scenarios when discovering fakes in new to generate attacks against web destinations, partopics and domains. The generative capabilities can be ticularly in crafting SQL injections [22]. harnessed for exploring innovative augmentation techniques. LLMs can help reduce the learning strategy costs We also foresee the adoption of LLMs as tools for associated with expert interaction (e.g., Active Learning), analysing textual descriptions of system configurations, thereby saving human annotators’ time. This can be in order to detect potential risks and vulnerabilities relaachieved by efectively integrating LLMs into learning tive to such configurations. loops at various levels, such as tuple selection and label A further relevant application of LLMs is the creation generation support. of a new-wave of tools to perform fuzz testing, especially for handling network protocols [23]. This is particularly relevant for a twofold reason. First, ubiquitous container5. Boosting Cybersecurity ized/virtualized frameworks are progressively migrating to the intrinsically networked microservice paradigm.
The last research line focuses on exploring various sce- Second, the emerging plague of malwares exploiting innarios where LLMs can bolster cybersecurity operations. formation hiding is hard to mitigate, especially since it The concept involves utilizing AI-based tools to auto- requires knowing in advance where the attacker will mate the analysis and processing of vast amounts of cloak the data [24]. semi-structured data. This approach aims to evaluate In this perspective, LLMs could be used to discover in security risks across systems and infrastructures more ef- advance protocol fields, metadata, header information, or ifciently. While Machine and Deep Learning techniques text segments in software that could be abused to conceal have been widely used to discover deviant behaviors in arbitrary/malicious content. For the case of networked event logs [18, 19, 20], the adoption of LLMs represents (micro)services, fuzzers can be used to learn the grama novel and quite unexplored research line. For instance, mar ruling a protocol starting from RFC documents [25]. in a recent work [21], the authors show how LLMs can These testing tools can hence be guided to explore interbe leveraged for analyzing huge volumes of information actions among containers or to fuzz specific operations, stored in logs. e.g., the setup/teardown of a connection.
A specific research objective is to support the automa- For the case of information-hiding-capable malware, tion of threat assessment. The intervention of the “ex- detection and sanitization are tightly coupled with the pert” (i.e., the human operator) is still crucial to evaluate abused resource (e.g., digital media vs network trafic), whether the anomalous event can be traced back to an and the number of features and ambiguities that can actual attack or threat. Nevertheless, we believe that the be exploited is almost unbounded. Therefore, fuzzers adoption of tools based on LLM can support and facilitate can be built by starting from datasets of pre-existent this task. Thus, our mid-term research goals are twofold. information-hiding-capable-attacks or trained over well• Improving eficiency. To enhance response known clocking patterns [26]. Thus, LLMs can lead to time to potential threats detected through logs, guided fuzzers, which demonstrated their ability to reveal our strategy involves leveraging Active Learning corner cases or uncommon anomalous templates [23]. techniques. These techniques enable human op- A midterm goal is then to tweak an LLM to evaluate erators to actively participate in the model learn- the limits of protocols when containing arbitrary inforing process, creating a human-in-the-loop sys- mation for implementing a covert communication. The use of LLMs will be particularly eficient for protocols 351/2022, PNRR Ricerca, CUP H23C22000550005; MUR like HTML and MQTT, which are based on large por- on D.M. 352/2022, PNRR Ricerca, CUP H23C22000440007. tions of textual information, especially in the header [27].
Moreover, we also plan to investigate if LLMs can be used
to improve the performance of our pre-existent AI/ML References
mechanisms for the detection of covert communications
[28, 29].
W. Shi, L. Zettlemoyer, Y. Tsvetkov, Y. Choi, [25] C. S. Xia, M. Paltenghi, J. Le Tian, M. Pradel,
D. Evans, H. Hajishirzi, Do Membership Inference L. Zhang, Fuzz4all: Universal Fuzzing with Large
Attacks Work on Large Language Models?, 2024. Language Models, Proc. IEEE/ACM ICSE (2024).
arXiv:2402.07841. [26] S. Wendzel, S. Zander, B. Fechner, C. Herdin,
[14] J. Uyheng, D. Bellutta, K. Carley, Bots amplify Pattern-based Survey and Categorization of
Netand redirect hate speech in online discourse about work Covert Channel Techniques, ACM
Computracism during the covid-19 pandemic, Social Media ing Surveys 47 (2015) 1–26.
+ Society 8 (2022) 205630512211047. doi:10.1177/ [27] T. Schmidbauer, S. Wendzel, SoK: A Survey of
In20563051221104749. direct Network-level Covert Channels, in:
Pro[15] J. Uyheng, K. M. Carley, Bots and onl
ference 2023, 2023, pp. 2698–2709. [18] A. Cuzzocrea, F. Folino, M. Guarascio, L. Pontieri,
A Multi-view Learning Approach to the Discovery of Deviant Process Instances, in: On the Move to Meaningful Internet Systems: OTM 2015 Conferences - Confederated International Conferences: CoopIS, ODBASE, and C&TC 2015, volume 9415 of Lecture Notes in Computer Science, Springer, 2015, pp. 146–165. [19] F. Folino, G. Folino, M. Guarascio, L. Pontieri, Semi
Supervised Discovery of DNN-Based Outcome Predictors from Scarcely-Labeled Process Logs, Business & Information Systems Engineering 64 (2022) 729–749. [20] F. Folino, G. Folino, M. Guarascio, L. Pontieri, Data& Compute-eficient Deviance Mining via Active Learning and Fast Ensembles, Journal of Intelligent
Information Systems (2024). [21] Z. Ma, A. R. Chen, D. J. Kim, T.-H. Chen, S. Wang,
LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing, in: 2024 IEEE/ACM 46th International Conference on Software Engineering, IEEE Computer Society, 2024, pp.
883–883. [22] R. Fang, R. Bindu, A. Gupta, Q. Zhan, D. Kang, LLM
Agents can Autonomously Hack Websites, arXiv preprint arXiv:2402.06664 (2024). [23] S. Mallissery, Y.-S. Wu, Demystify the Fuzzing
Methods: A Comprehensive Survey, ACM Computing Surveys 56 (2023) 1–38. [24] L. Caviglione, W. Mazurczyk, Never Mind the Malware, Here’s The Stegomalware, IEEE Security & Privacy 20 (2022) 101–106.