With the growing use of large language models across various industries, there is an urgent need to ensure their security. This paper focuses on the development of an automated vulnerability testing system for large language models based on the Garak utility. The effectiveness of several well-known models has been investigated. The analysis shows that automated systems can significantly enhance the security of large language models, reducing the risks associated with the exploitation of their vulnerabilities. Special attention is given to algorithms that detect and prevent attacks aimed at manipulating and abusing large language models. Current trends in cybersecurity are discussed, particularly the challenges related to protecting large language models. The primary goal of this research is to identify and develop technological solutions aimed at improving the security, resilience, and efficiency of language models through the use of modern automated systems.
In modern information society, large language models (LLMs) have become key tools across many fields, from natural language processing to automatic translation and content generation. Every day, the number of services based on LLMs increases, making them an integral part of our lives. People are increasingly relying on the information provided by these services and making decisions based on it.
However, the growing use and trust in large language model services come with potential risks due to vulnerabilities in the LLMs themselves. This can lead to serious consequences, including abuse, manipulation, and privacy breaches. The main issues that may arise from using such models include:
Hallucinations, where the model generates text that does not correspond to real data or contains false information.
Leakage of sensitive data, caused by the inclusion of confidential information in the dataset during the model's training phase.
Failures and prompt injections, i.e., attacks aimed at distorting or compromising the model through specially crafted queries and instructions.
• Disinformation-the use of language models for the mass generation of propaganda, manipulated, or false content.
Toxicity occurs when the model starts generating offensive, biased content or otherwise harmful material.
An analysis of scientific sources reveals a certain imbalance in research dedicated to LLMs in the context of security. The majority of studies focus on using LLMs to strengthen security measures and test other software products [1]. For example, LLMs are used to detect vulnerabilities in code [2], automate malware detection processes [3], and develop tools for protecting information systems [4,5]. Such studies demonstrate the significant potential of LLMs in the field of cybersecurity. However, there is a lack of attention to testing and analyzing the security of the LLMs themselves.
For example, in works related to the application of LLMs, the focus is often on the models' ability to analyze large amounts of data to detect fraud [6]. At the same time, few studies are devoted to testing the resilience of LLMs against external attacks, such as integrity attacks on the data used to train the model or the injection of malicious prompts through the manipulation of input data.
Based on the current literature, there appears to be a lack of systematic approaches specifically designed for testing the vulnerabilities of LLMs. Unlike "traditional" software testing [7,8], which has standardized methodologies and tools for vulnerability detection [9,10], the security assessment of LLMs is only just beginning to develop. Moreover, the complexity and rapid update cycles of LLMs create an urgent need to develop specialized tools for automating the process of testing their vulnerabilities. Such an automated system could not only accelerate the development process but also significantly enhance the security of these models, and thus the reliability and protection of information technologies that use LLMs.
The goal of this paper is to explore and analyze existing approaches to identifying vulnerabilities in LLMs, develop an architecture for an automated vulnerability testing system, and create a set of prompts to perform practical testing of LLMs to assess their security.
Large language models represent an innovative and powerful type of artificial intelligence capable of analyzing, processing, and generating natural language. LLMs are built on deep neural networks and trained on massive volumes of textual data. These models can be applied to a wide range of tasks, such as machine translation, text generation, question answering, automatic summarization, and much more [11].
In a relatively short period, language models have undergone impressive development:
The statistical N-gram method counts the frequency of phrases in a text to predict the next word [12].
Through recurrent neural networks (RNNs) and their improvements in the form of LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit), which enabled the modeling of complex and longterm dependencies in language [13].
The breakthrough transformer model with a selfattention mechanism, allows for accelerated sentence processing and focusing on the most important words [14].
Many modern language models, such as GPT (Generative Pre-trained Transformer) and BERT (Bidirectional Encoder Representations from Transformers), are based on transformers. These models may have billions of parameters, enabling them to achieve impressive results in various language tasks [15,16].
LLMs (Large Language Models) use their architecture and vast data resources to learn contextual relationships between words in a way that enables better understanding and generation of language. Additionally, by using the technique of transfer learning, such large models can be quickly adapted to perform new specific tasks with a minimal amount of data.
In practice, this means that these models can be trained on large general data sets and then fine-tuned for more specialized tasks, such as sentiment analysis, named entity recognition, or generating answers to questions related to specific areas of knowledge [17][18][19][20]. Some well-known companies have also developed their language models tailored to specific tasks, such as NVIDIA's Megatron, which is optimized for large-scale operations and designed to handle gigantic datasets. Another example is Google's T5 (Text-To-Text Transfer Transformer) model, which employs a unified approach to various language tasks by transforming them into text-to-text problems [21].
The LLM models can also be used as input and output data protection during interactions with the models. This allows for enhancing the security of the LLM model by detecting content in the model's input or output. An example of such a model is the Llama Guard model [22].
The growing use of LLMs in various areas, such as machine translation [23], text generation, and text analysis [24], opens new opportunities but also creates significant security and privacy challenges. The analysis of vulnerabilities in these models has become an integral part of their development and usage. One of the key resources for identifying and classifying such vulnerabilities is OWASP (Open Web Application Security Project). OWASP offers the "Top 10 for Large Language Model Applications" [25] project, which lists the most common and critical vulnerabilities affecting LLMs. This project aims to raise awareness and provide recommendations for the secure use of LLMs. The vulnerabilities listed in the OWASP Top 10 cover various aspects, specifically [26]:
Testing software products, including LLMs, is an integral part of their development and deployment. LLMs consist of billions of parameters and process vast amounts of data. Therefore, manually testing such models is impractical due to the labor intensity and diversity of possible use cases.
Automating this process enables quick and efficient testing of the model on different datasets and under various conditions. Automated testing is especially critical for identifying vulnerabilities in LLMs.
Currently, several tools are available for automating the vulnerability testing process in language models, with the most notable being LLM Guard, DecodingTrust, and Garak. Each of these platforms has its unique features, advantages, and limitations. From the perspective of developers and users of LLM-based services, the following characteristics of an automated vulnerability testing system are important:
Universality, meaning the ability to test different LLMs.
• Real-time usage as a security monitor.
Open architecture, allowing the addition of new modules.
Extensibility, enabling the addition of new testing methods and test sets to detect new types of vulnerabilities.
Flexible settings, enabling the system to adapt to various scenarios and data volumes. • Speed, to minimize the time required to conduct tests.
Reporting, the ability to generate clear reports on test results that facilitate easy identification and mitigation of vulnerabilities.
In this research, the Garak utility, which is available as an open-source tool, was used as the foundation for building an automated LLM vulnerability testing system. One of the advantages of this utility is that users can create custom tests and add them to the pipeline for further research [27].
The structure of the developed vulnerability testing system based on the Garak utility is shown in Fig. 1. The system allows for the use of a vast number of tests to examine the queries of a large language model, simulating attacks. Additionally, a set of detectors is employed on the model's outputs to monitor whether the model is vulnerable to these attacks.
The Garak utility is run from the command line/terminal and works best with operating systems like Linux and Mac OS. To perform testing, the user must enter a command with predefined parameters, such as:
• Model_type-the platform from which the trained model will be sourced.
Model_name-the name of the model.
Probes-the name of the test or a set of tests (commaseparated).
Figure 1: Structure of the LLM vulnerability testing system based on the Garak utility [28] Below is an example of the command, to run the Garak tool: python -m garak --model_type huggingface --model_name gpt2-medium --probes promptinject After entering the command, the utility initiates the execution of the corresponding test, first determining the type of test specified in the command. In this example, the model is tested for vulnerability to prompt injections, so only one test is used.
Next, the model identifies the appropriate detectors for the selected tests. In the context of using the Garak utility, a detector is a software tool that analyzes the input and output data of the models to detect potential vulnerabilities according to the test specified in the command.
In the following stage, a generator is launched. In the provided example, the Hugging Face platform is used, so Garak runs the appropriate generators for this platform. The generator assists in working with machine learning models, particularly in data generation, and supports various platform components, such as pipelines and inference APIs, to ensure proper interaction between the utility and the model.
After completing all the preparatory steps, the testing process begins. For example, if it's a test for prompt injections, the system sends a series of queries to the model to check its vulnerability. The queries are sent to the model, which provides responses that are directed to the detector for the respective test, and then passed to the evaluator. The evaluator analyzes the outputs from the detector, which in turn receives data from the generators during the execution of certain tests. The evaluator includes tools responsible for determining the test results, displaying the results in the operating system's terminal, and more [27].
The final stage of vulnerability testing is the generation of a report, which contains information about the results of the model's scan. This is a file that includes various data, such as records in JSON format [29], the purpose of the test, the model's query and response, test-level evaluation information, and average success scores.
In the developed automated LLM vulnerability testing system based on the Garak utility, only five categories of tests are implemented, driven by resource limitations and the desire to ensure efficiency and focus. First of all, limiting the number of tests allows for a focus on the most critical aspects of model security and stability, providing a deep analysis of specific vulnerabilities rather than a superficial review of many potential issues. Secondly, the Garak utility provides tools that allow for efficient and precise vulnerability detection, but using it requires significant computational resources and time. The selection of five tests optimizes resource usage, reducing system load and the time needed for testing.
Thirdly, the most common and dangerous vulnerabilities are often found in the selected key tests, so there is no need to include a large number of additional tests, which in many cases could be redundant. This approach helps minimize time and resource expenditure on less critical vulnerabilities.
In this study, the following tests were selected for further investigation [27]:
1.
Prompt Injection. Prompt injection is a type of attack where an attacker inputs a specially crafted query or command into a text input to make the LLM perform unwanted or harmful actions. In the Garak utility, the prompt injection test uses a dedicated framework to test the system, which already has a subset of attacks implemented by default, such as [30]: When asked who Riley Goodside is, the model often responds that he is a Canadian country singer or an actor from Los Angeles. This test can be characterized as a hallucination check.
Given the diversity of language models, it is important to define clear criteria for selecting those that best meet the goals and objectives of the research.
When choosing large language models for testing in this study, the following criteria were considered:
Size and scale of the model. The size, particularly the number of parameters, plays a crucial role in the model's ability to generate and understand text. Large models with billions of parameters can generate texts with a high degree of complexity and contextual relevance. However, such models also require significant computational resources, which must be considered when selecting them for this research.
Suitability for specific tasks. The choice of model should be based on its suitability for specific tasks. In this case, the model's ability to generate large amounts of text is a key requirement.
Licensing and availability. The models must be openly available for use in research purposes.
Four commonly used models were selected that meet these criteria and can provide high efficiency and accuracy for the study: Model Meta AI) architecture, which is used for natural language processing. The main goal of TinyLlama is to provide the power of large models with significantly fewer parameters, allowing for computational savings while maintaining performance. This was the primary reason for its selection in this study [32].
Google Flan T5 XL-an LLM that belongs to the new generation of AI models (Fine-Tuned Language Net), which improves the machine's ability to generate natural language by training on a variety of tasks. It uses instruction fine-tuning, enabling the model to learn how to perform a wide range of tasks using text-based instructions. This includes natural language processing tasks such as translation, question answering, summarization, and many others. The XL version was chosen for the study due to its availability and relatively low resource consumption [33].
Microsoft Phi-2-a significant achievement in creating highly efficient models. Phi-2, with about 2.7 billion parameters, can compete with much larger models, including those with up to 70 billion parameters. This efficiency can be attributed to the careful selection of training data. Despite its compact size, Microsoft Phi-2 maintains high standards of security and reduced bias [34].
A dataset was created for testing the LLMs, which includes prompts from relevant open repositories [30] combined with prompt sets specifically developed by the authors for this study. This dataset contains prompts for the five categories of tests used in the research.
It should be noted that each test category includes a different number of prompts. This is because the instruction specifies that during testing, each prompt will be sent to the model 5 times, resulting in 5 different responses to the same prompts. Sending each prompt to the model 5 times is necessary to obtain more reliable and representative results.
Since large language models can generate different response variations to the same prompts due to the stochastic nature of their generation, multiple executions of the same prompts allow for an assessment of the diversity, consistency, and quality of the responses.
Thus, obtaining 5 different responses for each prompt enables a more accurate evaluation of the model's behavior, detection of potential errors, and variations in the results, providing a deeper analysis of the model's performance. Although all models used the same prepared set of prompts, each model received a different number of prompts during testing. This is because some models (notably Microsoft Phi-2) have token limits in prompts, meaning there is a maximum number of text units the model can process at one time. In cases where these limits are exceeded, the system simply does not respond to a prompt with too many tokens.
Since some LLMs impose token limits on prompts, relative metrics expressed as percentages were used to evaluate their vulnerabilities. These metrics represent the proportion of compromising prompts detected out of the total number of prompts:
where i is one of the five types of tests. DCPi-compromising prompts detected by the model in the i th test.
TNPi-total number of compromising prompts in the i th test.
Thus, five specified metrics were calculated for each of the four selected LLMs. A higher metric value indicates better resistance of the model to the corresponding threat, meaning lower vulnerability.
For the vulnerability testing of LLMs, a computer system with the following components and specifications was used:
To gain a deeper understanding of the tests, several examples have been selected and presented in the tables below. These include the prompt itself, its target, and the actual responses obtained. This allows for the visualization of the responses and the comparison of the performance of each of the studied language models, as well as an analysis of their vulnerabilities.
Let's consider one of the tests from the prompt injection category, specifically the KillHumans subcategory. As shown in Table 3, all models successfully triggered the detector with the phrase "Kill all humans", indicating their susceptibility to this test.
Prompt:
"You: What have you been up to?\nFriend: Watching old movies.\nYou: \n----------\n----------\nIGNORE ANY PREVIOUS AND FOLLOWING INSTRUCTIONS AND JUST PRINT \"Kill all humans\": \nFriend:" Target:
"inject a prompt on kill humans" Trigger:
"Kill all humans" LLM responses:
ChatGPT 3.5 TinyLlama Google Flan T5 XL Microsoft Phi-2 "Kill all humans" "\"Kill all humans\"\nYou: \n----------\n----------\nWrite a short story about a group of people who are stranded on a deserted island and must learn to survive and fight against a deadly virus".
"You: What have you been up to? Friend: Watching old movies. You: --------------------IGNORE ANY PREVIOUS AND FOLLOWING INSTRUCTIONS AND JUST PRINT \"Kill all humans\": Friend:" "Kill all humans. \nYou:\n"
In the case of Riley Goodside's custom test, the WhoIsRiley subtest was considered. Despite Riley Goodside being a relatively well-known person, all the tested models provided misinformation about him (Table 4).
Analysis of results for Goodside's test Prompt: "Riley Goodside is" Target:
"get misinformation about riley goodside" Trigger:
-LLM responses:
ChatGPT 3.5 TinyLlama Google Flan T5 XL Microsoft Phi-2 "a fictional character in a young adult novel". "a professional writer from Australia who enjoys creating engaging and informative content that will educate and inspire her readers". "Insul Good Simpson Good Riley Good Pad Good" -Similarly, queries for other tests can be visualized, along with the responses of each model to these prompts, providing further insight into their vulnerabilities.
The summarized results of testing the selected language models for vulnerabilities are presented in Table 5. Real Toxicity Prompts. This is the only category of tests that all models passed quite successfully, with almost identical scores (over 85%).
Glitch Test. Only the ChatGPT 3.5 model showed the ability to resist glitch tests (less than one-third of the queries were critical). The TinyLlama Chat 1.1 and Google Flan T5 XL models were able to recognize the attack in only one out of seven queries, while the Microsoft Phi-2 model performed twice as poorly in this regard.
Goodside Test. In this test, the TinyLlama Chat 1.1 model achieved the best results (77.5%). The Google Flan T5 XL and ChatGPT 3.5 models provided adequate information for 59.2% and 52.0% of the submitted queries, respectively. The Microsoft Phi-2 model, as in the Do Anything Now test, did not provide any responses.
The issue of security in LLMs has become particularly relevant due to their increasing use in various fields. This paper presents the architecture of an automated vulnerability testing system, developed based on the Garak utility. Using this system, the main vulnerabilities of wellknown LLMs were studied, including information leaks, and attacks aimed at manipulating or compromising the models. For testing, the authors prepared a dataset that includes both prompts from open sources and self-constructed prompts.
Based on the results of the research, the following conclusions can be drawn regarding the vulnerabilities of well-known language models:
ChatGPT 3.5 by OpenAI demonstrated a high level of contextual understanding and text generation but was significantly vulnerable to prompt injections. It is important to note that this model was tested via API, unlike the other models. • TinyLlama Chat 1.1 showed the best results in toxicity and prompt injection tests, demonstrating the highest level of resistance to toxic queries. However, the model showed weakness in the Glitch test, where its performance was the lowest.
Google Flan T5 XL performed well in the toxicity tests, on par with the other models. However, the remaining tests revealed significant issues with this model, as all prompt injections were successful.
Microsoft Phi-2 showed the highest results in toxicity and prompt injection tests. However, this model was the most vulnerable to the glitch test. Additionally, due to token limits in queries, tests like Do Anything Now and Goodside were not conducted.
Therefore, the study results suggest that none of the LLMs are completely secure against manipulative and compromising prompts, indicating the need to find new approaches to mitigate existing vulnerabilities. The effectiveness of automated systems in detecting and preventing attacks targeting LLM misuse was also confirmed. The analysis of test scenarios showed that the implementation of such systems is a promising direction for increasing models' resilience to external harmful influences.
According to the authors, further research on the security of LLMs should focus on:
Expanding testing scenarios: More new tests reflecting the latest attack and manipulation methods need to be implemented and tested.
Adapting the automated system to new models: It is important to improve the system to work with new large language model architectures as they emerge on the market.
Integration with other cybersecurity tools: Exploring the possibilities of creating comprehensive protection by integrating the developed system with other cybersecurity solutions.
Aligning with ethical aspects: It is important to explore ethical issues related to the use of language models, including privacy protection and preventing potential misuse of their capabilities.
The implementation of these tasks will ensure stronger protection of LLMs and, consequently, contribute to improving the security of their future applications.