Introduction

Large Language Models (LLMs) allow to generate a wide array of contents. For instance, they can be used to create textual documents, pieces of music, as well as source code. A feature very relevant for their success is the ability of mimicking the human behavior. Unfortunately, this makes LLMs a double-edged sword since they can be exploited to generate realistic yet malicious content, such as fake news or text supporting misinformation campaigns. At the same time, LLMs have also proven to be effective in supporting various cyber-security duties, for instance, to analyze logs or network traffic [1].

In an attempt to fully understand the potential of LLMs in terms of offensive capabilities as well as the opportunities that should be seized to advance in the security of the Internet, researchers of the Institute for High Performance Computing and Networking -ICAR and of the Institute for Applied Mathematics and Information Technologies -IMATI of the National Research Council of Italy -CNR have intensified their efforts to investigate the pros and cons of LLMs. This research effort is established within the framework of three research projects. The first is funded by the Consortium named "SEcurity and RIghts In the CyberSpace -SERICS", and aims at using LLMs to increase the security posture of networking and computing systems. For instance, an LLM can be used to synthesize behaviors starting from logs of containerized microservices or to generate automatic textual replies to deceive e-mail scammers [2]. The second research action is funded by the project "Watermarking Hazards and novel perspectives in Adversarial Machine learning -WHAM!", and is devoted to quantifying the limits and opportunities of watermarking schemes when applied to AI artifacts. As an example, data can be hidden to recognize deep fakes, to understand whether a model has been cloned, or to track usages in Machine-Learningas-a-Service deployments [3]. Even worse, problem of exploiting unauthorized content during training or in deployment needs to be specifically addressed. The third research action is funded by the project "Limiting MIsinformation spRead in online environments through multimodal and cross-domain FAKe news detection -MIRFAK", which aims at developing an innovative content verification tool, delivering solutions for news verification on social media and online platforms. Within the project, we aim at exploring the potentials and risks of LLMs associated with misinformation.

In this work, we outline our research agenda on these topics, which is devised in three directions: i) we present mid-term challenges for using LLMs to solve securityrelated issues; ii) we discuss how watermarks can be applied to LLMs to mitigate attacks aiming at stealing information or disseminating fake news; iii) we showcase the gaps to be filled to make LLMs a real asset for the Internet.

The rest of the paper is structured as follows. Section 2 deals with the problems of understanding whether the output has been generated by an LLM and of tracking its provenance, while Section 3 considers usage violations, such as unauthorized harvesting of data for training models. Section 4 discusses challenges and opportunities relative to the adoption of LLMs in the context of online social platforms and debates. Section 5 discusses the adoption of LLMs in assessing cybersecurity risks related to systems and infrastructures in containerized environments. Lastly, Section 6 concludes the work and portrays some prospected action points.

Are the Data Generated?

One of the main goals of our research is to investigate challenges and solutions for protecting the Intellectual Property (IP) of the Machine/Deep Learning (ML/DL) models as well as of the dataset used for the training phase [4]. Moreover, we also aim at considering techniques to mark the output produced by ML/DL services, for instance, to understand whether an attacker "cloned" the model through multiple remote invocations. Specifically, we are interested in techniques that allow the cloaking of secret information within the contents we want to protect. In this respect, an emerging research line considers watermarking techniques, i.e., arbitrary pieces of data that are embedded within the item to deliver and that are difficult to recognize besides proprietary decryption schemes. Such mechanisms are common with images and multimedia objects [5] and can be used to embed control data within ML/DL models.

Techniques used to prevent unwanted/unfair usages or to enforce IP can also be envisioned for generative models, with a particular focus on large language models. There are essentially two scenarios that are relevant in this respect. The first scenario is relative to the opportunity to mark generated text in a way that it can be easily recognized. Watermarking can be employed in this context to embed the watermark within the output of the LLM and, thus, distinguish between the data generated by a human and those produced by a machine. The objective here is to enforce IP protection as well as to claim ownership on the generated data. The second scenario is relative to the problem that such generative models can deliver malicious content. To mitigate potential harm caused by such generated data, it is crucial to develop methods to identify content generated by a machine, when a watermark is not embedded. It is worth noting that the generation of malicious content can be both unintentional or intentional. Unintentional generation may happen due to the stochastic nature of such generative models, which causes the phenomenon of hallucinations (i.e., unrealistic or imaginary content). By contrast, intentional generation is typically done by a malicious threat actor, who pushes the generative model to obtain mischievous data. In both cases, the generated data could be of high quality, infusing trust among readers eventually forcing them to fall into error or forward the content, e.g., through sharing functionalities of online social networks. Our research in this context aims at developing methods to identify contents generated by a machine through a language model. We are interested both in devising watermarking schemes and in the more general challenge relative to the problem of devising predictive methods for discriminating generated data. Besides, this research activity is aligned with the current requirements enforced by the recently released European AI Act1 . The latter in fact introduces specific transparency obligations to ensure that humans are informed when necessary, to ensure trust, and in particular, that AI-generated content is identifiable.

The research approaches to this topic are quite recent. To the best of our knowledge, the first LLM watermarking technique for distinguishing human-generated from machine-generated texts was proposed by Kirchenbauer et al. [6]. In text generation, language-based models produce a probability distribution over a vocabulary, i.e., the set of words or word fragments (i.e., tokens), used for predicting the most likely next word based on the previous ones. The authors propose to alter such distribution, in order to promote sampling of specific tokens. The occurrence within a given statistical significance of such tokens characterizes the watermark within the text. One of the main limitations of this approach is the gen-eration of low-quality texts in contexts characterized by relatively deterministic content, such as code snippets or structured text. Lee et al. [7] refine the approach by ensuring that sampling is only focused on high-entropy tokens.

One of our research objectives is to generalize these approaches to other generative models, such as Diffusion Models or Generative Adversarial Networks (GANs). In addition, the analysis of the distribution of generated data, and its comparison with that of real (not synthetic) data can also be exploited for devising predictive models aimed at automatically detecting the reliability and authenticity of data.

Have You Stolen My Data?

Membership Inference attacks (MIAs) [8] aim to predict whether a data sample was included in the training dataset of a machine learning model. These attacks serve to evaluate the privacy vulnerabilities present in machine learning models, like in Neural Networks [9], GANs [10] and Diffusion Models [11]. Formally, the goal of a MIA is to infer whether a given data point 𝑥 was part of the training dataset 𝐷 for model 𝑀 by computing a membership score 𝑠(𝑥; 𝑀 ). This score is then thresholded to determine a target sample's membership.

Membership inference attacks exploit the tendency of the models to overfit their training data and hence exhibit lower loss values for these elements. A first and widely used attack is the LOSS attack [12], in which samples are classified as training members if their loss values are lower than a fixed threshold (that is, 𝑠(𝑥; 𝑀 ) is defined in terms of ℒ(𝑥; 𝑀 )).

Recent works aim to design and improve MIAs for LLMs. In this case, MIAs consider a target model 𝑀 which gives as output a probability distribution of the next token given a prefix as input, P(𝑥𝑡|𝑥0 . . . 𝑥𝑡−1; 𝑀 ). The goal of MIA is hence to infer whether the target sample 𝑥 = 𝑥1 . . . 𝑥𝑛 of 𝑛 tokens has been considered in the training set. Duan et al. [13] consider several membership inference attacks and show that they just outperform random guessing for most settings across different LLM size and domains. They also argue that MIA is difficult on LLMs because of different key reasons. These include the difficulty of handling LLMs pre-trained over billions and trillions of tokens, or the overlap typically exhibited by the underlying token distributions that can be observed in natural language documents, irrespective of their training data membership.

Our research agenda is aimed at extending and leveraging the current membership inference games, by investigating adversarial approaches in order to force the LLM to generate copyrighted text. In this way, we define a framework that can demonstrate copyright violations and overcome MIA's issues related to large dataset and the intrinsic randomness of LLMs.

Fighting Fire with Fire:

Generative AI to promote Online Safety

LLMs are showcasing remarkable abilities in various Natural Language Processing tasks, making them a highly potent and beneficial tool for everyday life. However, alongside their appealing strengths and widespread adoption, a significant concern is arising regarding their potential role in amplifying the generation and dissemination of misinformation and disinformation. Generative AI technology has significantly empowered malicious actors to produce fake content, which can be disseminated across online social networks and lead to detrimental phenomena, e.g., manipulating public discourse, disseminating hate speech, and sharing fake content. As a remarkable example, in 2016 Microsoft released the Tay chatbot, which triggered further controversy by posting inflammatory and offensive tweets via its Twitter account, leading Microsoft to shut down the service within just 16 hours2 . More recently, other works assessed the role of bots and AI agents in conveying and amplifying online discourse about racism and hate speech [14,15], drawing further attention to this sensitive topic. Thus, as underscored by [16], the scale, velocity and accessibility of generative models present compelling challenges for online platforms, potentially inundating them with a massive amount of fraudulent material and unpredictable social consequences. While policy makers are actively engaged in regulating the use of GenAI tools, the efficacy of these measures remains uncertain. In response, our research group is working towards leveraging Generative AI to enhance online safety. Our objective is to reuse the same technology used to contaminate online discussions for a beneficial purpose in a controlled environment. For instance, [17] demonstrated the potential of a GPT2-like model in crafting tailored responses to combat misinformation regarding the COVID-19 pandemic. Despite this first promising result, there are numerous overlooked opportunities for harnessing GenAI tools to aid online safety. One such opportunity involves the development of automated agents capable of serving as "peace-builders" within online discussions. We aim to train a large language model to generate textual content that, once injected within online social media platforms, can help mitigate polarization and disagreement.

This research line is interesting and open to novel and original developments, but it also faces considerable challenges. A trivial remark is to carefully consider the ethical implications of using GenAI tools for online safety to ensure responsible use. Second, there are considerable technical challenges regarding the training and/or finetuning of these large models due to scalability concerns. Third, evaluating the effectiveness of GenAI interventions in promoting online safety can be demanding and could require a multi-disciplinary approach involving experts from fields such as psychology and sociology.

Another compelling line in our research agenda is to define the aspects to take into account when analyzing the role of LLMs in this context. We are interested in exploring the role of LLMs in contrasting the phenomenon of false information spreading at different levels: detection, mitigation, intervention, and attribution. Our effort is to improve the fake detection models under the constraint of scarcely labeled data, which is a common condition in real scenarios when discovering fakes in new topics and domains. The generative capabilities can be harnessed for exploring innovative augmentation techniques. LLMs can help reduce the learning strategy costs associated with expert interaction (e.g., Active Learning), thereby saving human annotators' time. This can be achieved by effectively integrating LLMs into learning loops at various levels, such as tuple selection and label generation support.

Boosting Cybersecurity

The last research line focuses on exploring various scenarios where LLMs can bolster cybersecurity operations. The concept involves utilizing AI-based tools to automate the analysis and processing of vast amounts of semi-structured data. This approach aims to evaluate security risks across systems and infrastructures more efficiently. While Machine and Deep Learning techniques have been widely used to discover deviant behaviors in event logs [18,19,20], the adoption of LLMs represents a novel and quite unexplored research line. For instance, in a recent work [21], the authors show how LLMs can be leveraged for analyzing huge volumes of information stored in logs.

A specific research objective is to support the automation of threat assessment. The intervention of the "expert" (i.e., the human operator) is still crucial to evaluate whether the anomalous event can be traced back to an actual attack or threat. Nevertheless, we believe that the adoption of tools based on LLM can support and facilitate this task. Thus, our mid-term research goals are twofold.

• Improving efficiency. To enhance response time to potential threats detected through logs, our strategy involves leveraging Active Learning techniques. These techniques enable human operators to actively participate in the model learning process, creating a human-in-the-loop sys-tem. Thus, our approach aims to expedite threat response when integrating human expertise into the learning loop of the model, by using post-hoc explanation tools to support the operator in validating the attack and guiding the learning of the model. • Data enrichment. Another critical aspect involves the potential use of LLMs to enhance the security of Internet-wide infrastructures. Numerous protocols and services rely heavily on textual information, such as URLs or configuration data. LLMs can be exploited in generating test cases, particularly for automating periodic assessments aimed at detecting potential deviations in the security posture of a deployment. For example, recent research showcased LLMs' capability to generate attacks against web destinations, particularly in crafting SQL injections [22].

We also foresee the adoption of LLMs as tools for analysing textual descriptions of system configurations, in order to detect potential risks and vulnerabilities relative to such configurations.

A further relevant application of LLMs is the creation of a new-wave of tools to perform fuzz testing, especially for handling network protocols [23]. This is particularly relevant for a twofold reason. First, ubiquitous containerized/virtualized frameworks are progressively migrating to the intrinsically networked microservice paradigm. Second, the emerging plague of malwares exploiting information hiding is hard to mitigate, especially since it requires knowing in advance where the attacker will cloak the data [24].

In this perspective, LLMs could be used to discover in advance protocol fields, metadata, header information, or text segments in software that could be abused to conceal arbitrary/malicious content. For the case of networked (micro)services, fuzzers can be used to learn the grammar ruling a protocol starting from RFC documents [25]. These testing tools can hence be guided to explore interactions among containers or to fuzz specific operations, e.g., the setup/teardown of a connection.

For the case of information-hiding-capable malware, detection and sanitization are tightly coupled with the abused resource (e.g., digital media vs network traffic), and the number of features and ambiguities that can be exploited is almost unbounded. Therefore, fuzzers can be built by starting from datasets of pre-existent information-hiding-capable-attacks or trained over wellknown clocking patterns [26]. Thus, LLMs can lead to guided fuzzers, which demonstrated their ability to reveal corner cases or uncommon anomalous templates [23].

A midterm goal is then to tweak an LLM to evaluate the limits of protocols when containing arbitrary information for implementing a covert communication. The use of LLMs will be particularly efficient for protocols like HTML and MQTT, which are based on large portions of textual information, especially in the header [27]. Moreover, we also plan to investigate if LLMs can be used to improve the performance of our pre-existent AI/ML mechanisms for the detection of covert communications [28,29].

Conclusions

LLMs present a spectrum of opportunities and challenges within the cybersecurity domain. We've delved into four primary research avenues, each addressing distinct problems and proposing corresponding solutions. These areas include:

• Watermarking and Detection of Generative Content: Developing methods to embed unique identifiers into data for tracking and authentication purposes, alongside techniques for detecting generative content to combat potential trustworthiness and security risks. • Membership Inference and Data Provenance: Addressing concerns related to establishing the origin of training data, crucial for ensuring data integrity, privacy. • Misinformation Mitigation/Intervention: Implementing strategies to combat misinformation and ensure online safety, particularly in the context of rapidly evolving online information landscapes. • Log Analysis and Stress Testing in Infrastructure Protection: Analyzing system logs and subjecting infrastructures to stress tests to assess their resilience against cyber threats, essential for maintaining robust security measures.

We have devised specific solutions within the context of three research projects funded by the Italian Ministry of Research. These solutions aim to address various cybersecurity challenges and enhance overall digital security measures,