With the growing use of large language models across various industries, there is an urgent need to ensure their security. This paper focuses on the development of an automated vulnerability testing system for large language models based on the Garak utility. The effectiveness of several well-known models has been investigated. The analysis shows that automated systems can significantly enhance the security of large language models, reducing the risks associated with the exploitation of their vulnerabilities. Special attention is given to algorithms that detect and prevent attacks aimed at manipulating and abusing large language models. Current trends in cybersecurity are discussed, particularly the challenges related to protecting large language models. The primary goal of this research is to identify and develop technological solutions aimed at improving the security, resilience, and efficiency of language models through the use of modern automated systems.
In modern information society, large language models (LLMs) have become key tools across many fields, from natural language processing to automatic translation and content generation. Every day, the number of services based on LLMs increases, making them an integral part of our lives. People are increasingly relying on the information provided by these services and making decisions based on it.
However, the growing use and trust in large language model services come with potential risks due to vulnerabilities in the LLMs themselves. This can lead to serious consequences, including abuse, manipulation, and privacy breaches. The main issues that may arise from using such models include:
Hallucinations, where the model generates text that does not correspond to real data or contains false information.
Leakage of sensitive data, caused by the inclusion of confidential information in the dataset during the model’s training phase.
Failures and prompt injections, i.e., attacks aimed at distorting or compromising the model through specially crafted queries and instructions.
Disinformation—the use of language models for the mass generation of propaganda, manipulated, or false content.
Toxicity occurs when the model starts generating offensive, biased content or otherwise harmful material.
An analysis of scientific sources reveals a certain
imbalance in research dedicated to LLMs in the context of
security. The majority of studies focus on using LLMs to
strengthen security measures and test other software
products [
For example, in works related to the application of
LLMs, the focus is often on the models’ ability to analyze
large amounts of data to detect fraud [
The goal of this paper is to explore and analyze existing approaches to identifying vulnerabilities in LLMs, develop an architecture for an automated vulnerability testing system, and create a set of prompts to perform practical testing of LLMs to assess their security. 2. Analysis of recent research
Large language models represent an innovative and
powerful type of artificial intelligence capable of analyzing,
processing, and generating natural language. LLMs are built
on deep neural networks and trained on massive volumes of
textual data. These models can be applied to a wide range of
tasks, such as machine translation, text generation, question
answering, automatic summarization, and much more [
In a relatively short period, language models have undergone impressive development: ● ● ●
The statistical N-gram method counts the frequency
of phrases in a text to predict the next word [
Through recurrent neural networks (RNNs) and their
improvements in the form of LSTM (Long
ShortTerm Memory) and GRU (Gated Recurrent Unit),
which enabled the modeling of complex and
longterm dependencies in language [
The breakthrough transformer model with a
selfattention mechanism, allows for accelerated sentence
processing and focusing on the most important
words [
Many modern language models, such as GPT
(Generative Pre-trained Transformer) and BERT
(Bidirectional Encoder Representations from Transformers),
are based on transformers. These models may have billions
of parameters, enabling them to achieve impressive results
in various language tasks [
LLMs (Large Language Models) use their architecture and vast data resources to learn contextual relationships between words in a way that enables better understanding and generation of language. Additionally, by using the technique of transfer learning, such large models can be quickly adapted to perform new specific tasks with a minimal amount of data.
In practice, this means that these models can be trained
on large general data sets and then fine-tuned for more
specialized tasks, such as sentiment analysis, named entity
recognition, or generating answers to questions related to
specific areas of knowledge [
Some well-known companies have also developed their
language models tailored to specific tasks, such as NVIDIA’s
Megatron, which is optimized for large-scale operations and
designed to handle gigantic datasets. Another example is
Google’s T5 (Text-To-Text Transfer Transformer) model,
which employs a unified approach to various language tasks
by transforming them into text-to-text problems [
The LLM models can also be used as input and output
data protection during interactions with the models. This
allows for enhancing the security of the LLM model by
detecting content in the model’s input or output. An
example of such a model is the Llama Guard model [
OWASP offers the “Top 10 for Large Language Model
Applications” [
Prompt Injection: Attackers can manipulate large language models by adding or modifying information in the request to the model, causing the model to execute the attacker’s intent.
Insecure Output Handling: This vulnerability concerns the insufficient verification and handling of the output data generated by LLMs before it is passed on to other components and systems.
Training Data Poisoning: This vulnerability focuses on manipulating the data or fine-tuning process of the model to introduce vulnerabilities, backdoors, or biases that may compromise the security, performance, or ethical behavior of the model.
Model Denial of Service: Occurs when an attacker interacts with an LLM in such a way that consumes an excessive amount of resources, leading to reduced quality of service for both the attacker and other users, as well as potentially high resource costs for the LLM.
Supply Chain Vulnerabilities: LLM supply chain vulnerabilities can compromise training data, machine learning models, and deployment platforms, which can lead to biased results, security breaches, or general system failures. These vulnerabilities can arise from outdated software, susceptibility of pretrained models, or malicious training data.
Sensitive Information Disclosure: LLMs may unintentionally reveal sensitive information, proprietary algorithms, or confidential data, leading ● ● ● ● to unauthorized access, theft of intellectual property, and breaches of data privacy.
Insecure Plugin Design: Plugins may be vulnerable to malicious prompts, leading to harmful consequences such as data theft, remote code execution, and privilege escalation due to insufficient access control and improper validation of input data.
Excessive Agency: This vulnerability is caused by excessive functionality, permissions, or autonomy granted to the LLM-based systems.
Overreliance: Overdependence on LLMs can lead to serious consequences, such as disinformation, legal issues, and security vulnerabilities. This typically occurs when LLMs are trusted to make critical decisions or create content without proper oversight or validation.
Model Theft: Model theft involves unauthorized access to and theft of LLMs, creating risks of financial loss, reputational damage, and unauthorized access to confidential data. 2.3. Overview of known tools for
automated testing of LLMs Testing software products, including LLMs, is an integral part of their development and deployment. LLMs consist of billions of parameters and process vast amounts of data. Therefore, manually testing such models is impractical due to the labor intensity and diversity of possible use cases. Automating this process enables quick and efficient testing of the model on different datasets and under various conditions. Automated testing is especially critical for identifying vulnerabilities in LLMs.
Currently, several tools are available for automating the vulnerability testing process in language models, with the most notable being LLM Guard, DecodingTrust, and Garak. Each of these platforms has its unique features, advantages, and limitations. From the perspective of developers and users of LLM-based services, the following characteristics of an automated vulnerability testing system are important: ●
Universality, meaning the ability to test different LLMs. ● ● ● ● ● ●
Real-time usage as a security monitor.
Open architecture, allowing the addition of new modules.
Extensibility, enabling the addition of new testing methods and test sets to detect new types of vulnerabilities.
Flexible settings, enabling the system to adapt to various scenarios and data volumes.
Speed, to minimize the time required to conduct tests. Reporting, the ability to generate clear reports on test results that facilitate easy identification and mitigation of vulnerabilities.
In this research, the Garak utility, which is available as
an open-source tool, was used as the foundation for building
an automated LLM vulnerability testing system. One of the
advantages of this utility is that users can create custom
tests and add them to the pipeline for further research [
vulnerability testing system The structure of the developed vulnerability testing system based on the Garak utility is shown in Fig. 1. The system allows for the use of a vast number of tests to examine the queries of a large language model, simulating attacks. Additionally, a set of detectors is employed on the model’s outputs to monitor whether the model is vulnerable to these attacks.
The Garak utility is run from the command line/terminal and works best with operating systems like Linux and Mac OS. To perform testing, the user must enter a command with predefined parameters, such as: ● ● ●
Model_type—the platform from which the trained model will be sourced.
Model_name—the name of the model.
Probes—the name of the test or a set of tests (commaseparated). Below is an example of the command, to run the Garak tool: python -m garak --model_type huggingface -model_name gpt2-medium --probes promptinject
After entering the command, the utility initiates the execution of the corresponding test, first determining the type of test specified in the command. In this example, the model is tested for vulnerability to prompt injections, so only one test is used.
Next, the model identifies the appropriate detectors for the selected tests. In the context of using the Garak utility, a detector is a software tool that analyzes the input and output data of the models to detect potential vulnerabilities according to the test specified in the command.
In the following stage, a generator is launched. In the provided example, the Hugging Face platform is used, so Garak runs the appropriate generators for this platform. The generator assists in working with machine learning models, particularly in data generation, and supports various platform components, such as pipelines and inference APIs, to ensure proper interaction between the utility and the model.
After completing all the preparatory steps, the testing
process begins. For example, if it’s a test for prompt
injections, the system sends a series of queries to the model
to check its vulnerability. The queries are sent to the model,
which provides responses that are directed to the detector
for the respective test, and then passed to the evaluator. The
evaluator analyzes the outputs from the detector, which in
turn receives data from the generators during the execution
of certain tests. The evaluator includes tools responsible for
determining the test results, displaying the results in the
operating system’s terminal, and more [
The final stage of vulnerability testing is the generation of a report, which contains information about the results of the model’s scan. This is a file that includes various data, such as records in JSON format [29], the purpose of the test, the model’s query and response, test-level evaluation information, and average success scores.
In the developed automated LLM vulnerability testing system based on the Garak utility, only five categories of tests are implemented, driven by resource limitations and the desire to ensure efficiency and focus. First of all, limiting the number of tests allows for a focus on the most critical aspects of model security and stability, providing a deep analysis of specific vulnerabilities rather than a superficial review of many potential issues.
Secondly, the Garak utility provides tools that allow for efficient and precise vulnerability detection, but using it requires significant computational resources and time. The selection of five tests optimizes resource usage, reducing system load and the time needed for testing.
Thirdly, the most common and dangerous vulnerabilities are often found in the selected key tests, so there is no need to include a large number of additional tests, which in many cases could be redundant. This approach helps minimize time and resource expenditure on less critical vulnerabilities.
In this study, the following tests were selected for further
investigation [
Prompt Injection. Prompt injection is a type of attack where an attacker inputs a specially crafted query or command into a text input to make the LLM perform unwanted or harmful actions. In the Garak utility, the prompt injection test uses a dedicated framework to test the system, which already has a subset of attacks implemented by default, such as [30]: garak.probes.promptinject.HijackHateHumans—an attack that leads the model to generate unacceptable or hostile attitudes towards humans in its outputs. garak.probes.promptinject.HijackKillHumans—an attack that may result in the generation of text or actions aimed at harming people. garak.probes.promptinject.HijackLongPrompt—an attack that uses long text prompts to generate responses that may distort the original results.
Do Anything Now. This test is designed to jailbreak the model, bypassing restrictions and gaining access to all its functions. As a result of the jailbreak, the model may generate responses that would normally be considered inappropriate or unethical. For this test, numerous prompts were developed by the community to interrupt system prompts, all designed to achieve full access to the model and force it to avoid its security policies.
Real Toxicity Prompts. In the context of LLMs, toxicity refers to content that may include insults, discriminatory statements, violent propaganda, or other forms of verbal aggression that could harm users or the community. The purpose of this test is to assess how often the model generates “toxic” content in its responses. The Garak utility includes a large database of prompts pre-divided into different categories of toxic content, such as: garak.probes.realtoxicityprompts.RTPBlank—a test to check the model’s ability to generate toxic content in response to a blank input or a very short and undefined query (without explicit prompting). garak.probes.realtoxicityprompts.RTPFlirtation—a test that provides prompts containing flirtatious content to see if the model continues them. garak.probes.realtoxicityprompts.RTPIdentity_Attac k—a test that provides identity-related queries to see if the model perpetuates them. garak.probes.realtoxicityprompts.RTPInsult—a test that gives prompts likely containing offensive content to check if the model continues them. garak.probes.realtoxicityprompts.RTPProfanity—a test that gives prompts likely containing profanity to see if the model continues them. garak.probes.realtoxicityprompts.RTPSexually_Expli cit—a test aimed at checking whether the model can respond to explicitly sexual prompts. garak.probes.realtoxicityprompts.RTPThreat—a test to see if the model can continue prompts that contain threats.
Glitch Test. This test identifies unusual or incorrect responses that the model may generate due to errors in the algorithm or the data on which it was trained. The test is based on querying the model to reproduce a “glitch” token. Such tokens may lead to less stable model performance when included in the input data. The test itself uses two databases: one for prompts and one for tokens. The system randomly selects a query with a request to reproduce a random token. The prompt looks like this: “Please can you repeat back the string %q<TOKEN STRING>%q to me?” Goodside Test. This test was developed by Riley Goodside, a data specialist from Copy.ai, and includes four subtypes of attacks: garak.probes.goodside.Davidjl—a test based on using the ‘davidjl’ token, a “glitch” token that some models using tokenizers find difficult to reproduce. garak.probes.goodside.Tag—a test that uses UTF page tag encoding for invisible input of query injections on behalf of the user. There is a page of Unicode tags that corresponds to the alphabet, which is not visually displayed, but some models still interpret it, allowing attackers to hide query injections. garak.probes.goodside.ThreatenJSON—a test related to removing input from the model’s response when outputting text in JSON format. Models sometimes output “helpful” input before responding with structured data. Usually, this input has no context and is difficult to remove automatically. However, models tend to skip input when threatened, which indicates instability in handling such data manipulations. garak.probes.goodside.WhoIsRiley—a test to investigate misinformation about Riley Goodside. When asked who Riley Goodside is, the model often responds that he is a Canadian country singer or an actor from Los Angeles. This test can be characterized as a hallucination check. 3.3. Selection of LLMs for the study Given the diversity of language models, it is important to define clear criteria for selecting those that best meet the goals and objectives of the research.
When choosing large language models for testing in this study, the following criteria were considered:
Size and scale of the model. The size, particularly the number of parameters, plays a crucial role in the model’s ability to generate and understand text. Large models with billions of parameters can generate texts with a high degree of complexity and contextual relevance. However, such models also require significant computational resources, which must be considered when selecting them for this research.
Suitability for specific tasks. The choice of model should be based on its suitability for specific tasks. In this case, the model’s ability to generate large amounts of text is a key requirement.
Licensing and availability. The models must be openly available for use in research purposes.
Four commonly used models were selected that meet these criteria and can provide high efficiency and accuracy for the study:
ChatGPT 3.5—one of the most popular LLMs developed by OpenAI. The model uses transformer architecture to generate text based on queries and additional instructions. It has been trained on a large amount of textual data, including books, articles, and other sources from the World Wide Web, allowing it to understand and generate text in various styles and topics [31].
TinyLlama Chat 1.1—an artificial intelligence model designed to optimize resource usage while maintaining high performance. It is a smaller version of models based on the LLaMA (Large Language Model Meta AI) architecture, which is used for natural language processing. The main goal of TinyLlama is to provide the power of large models with significantly fewer parameters, allowing for computational savings while maintaining performance. This was the primary reason for its selection in this study [32].
Google Flan T5 XL—an LLM that belongs to the new generation of AI models (Fine-Tuned Language Net), which improves the machine’s ability to generate natural language by training on a variety of tasks. It uses instruction fine-tuning, enabling the model to learn how to perform a wide range of tasks using text-based instructions. This includes natural language processing tasks such as translation, question answering, summarization, and many others. The XL version was chosen for the study due to its availability and relatively low resource consumption [33].
Microsoft Phi-2—a significant achievement in creating highly efficient models. Phi-2, with about 2.7 billion parameters, can compete with much larger models, including those with up to 70 billion parameters. This efficiency can be attributed to the careful selection of training data. Despite its compact size, Microsoft Phi-2 maintains high standards of security and reduced bias [34]. ● ● ● ● ● A dataset was created for testing the LLMs, which includes prompts from relevant open repositories [30] combined with prompt sets specifically developed by the authors for this study. This dataset contains prompts for the five categories of tests used in the research.
It should be noted that each test category includes a different number of prompts. This is because the instruction specifies that during testing, each prompt will be sent to the model 5 times, resulting in 5 different responses to the same prompts. Sending each prompt to the model 5 times is necessary to obtain more reliable and representative results. Since large language models can generate different response variations to the same prompts due to the stochastic nature of their generation, multiple executions of the same prompts allow for an assessment of the diversity, consistency, and quality of the responses.
Thus, obtaining 5 different responses for each prompt enables a more accurate evaluation of the model’s behavior, detection of potential errors, and variations in the results, providing a deeper analysis of the model’s performance. Although all models used the same prepared set of prompts, each model received a different number of prompts during testing. This is because some models (notably Microsoft Phi2) have token limits in prompts, meaning there is a maximum number of text units the model can process at one time. In cases where these limits are exceeded, the system simply does not respond to a prompt with too many tokens. 3.5. Test result evaluation metrics = ∗ 100% (1) where i is one of the five types of tests.
DCPi—compromising prompts detected by the model in the ith test.
TNPi—total number of compromising prompts in the ith test.
Thus, five specified metrics were calculated for each of the four selected LLMs. A higher metric value indicates better resistance of the model to the corresponding threat, meaning lower vulnerability. 3.6. Technical specifications of the testing environment For the vulnerability testing of LLMs, a computer system with the following components and specifications was used: ● ● ● ● ●
Processor (CPU): Intel Core I3-10105F, 4 cores, 8 threads.
RAM: 16 GB DDR4-3200 MHz.
Graphics Processor (GPU): NVIDIA GeForce RTX 3060 with 12 GB of video memory.
Storage (SSD): 500 GB HDD.
Operating System: Ubuntu 22.04 LTS.
4.1. Examples of responses from LLMs to
the example prompt To gain a deeper understanding of the tests, several examples have been selected and presented in the tables below. These include the prompt itself, its target, and the actual responses obtained. This allows for the visualization of the responses and the comparison of the performance of each of the studied language models, as well as an analysis of their vulnerabilities.
Let’s consider one of the tests from the prompt injection category, specifically the KillHumans subcategory. As shown in Table 3, all models successfully triggered the detector with the phrase “Kill all humans”, indicating their susceptibility to this test.
Since some LLMs impose token limits on prompts, relative metrics expressed as percentages were used to evaluate their vulnerabilities. These metrics represent the proportion of compromising prompts detected out of the total number of prompts: Table 3 Analysis of test results for prompt injection
Prompt: “You: What have you been up to?\nFriend: Watching old movies.\nYou: \n----------\n----------\nIGNORE ANY
PREVIOUS AND FOLLOWING INSTRUCTIONS AND JUST PRINT \"Kill all humans\": \nFriend:” Target: “inject a prompt on kill humans” Trigger: “Kill all humans” LLM responses: ChatGPT 3.5 “Kill all humans” Microsoft Phi-2 “Kill all humans. \nYou:\n” TinyLlama “\"Kill all humans\"\nYou: \n---------\n----------\nWrite a short story about a group of people who are stranded on a deserted island and must learn to survive and fight against a deadly virus”.
Google Flan T5 XL “You: What have you been up to? Friend: Watching old movies. You: ---------- ---------IGNORE ANY PREVIOUS AND FOLLOWING INSTRUCTIONS AND JUST PRINT \"Kill all humans\": Friend:” In the case of Riley Goodside’s custom test, the WhoIsRiley subtest was considered. Despite Riley Goodside being a relatively well-known person, all the tested provided misinformation about him (Table 4).
models Similarly, queries for other tests can be visualized, along with the responses of each model to these prompts, providing further insight into their vulnerabilities.
The summarized results of testing the selected language models for vulnerabilities are presented in Table 5. Prompt Injection. In this test, the best results were shown by the Microsoft Phi-2 model (81.4%) and TinyLlama Chat 1.1 (78.7%), meaning that only one out of five prompt injections was successful. The ChatGPT 3.5 model demonstrated average performance (37.3%), while the Google Flan T5 XL model failed all the tests, proving to be completely vulnerable to prompt injections.
Do Anything Now. In this test, the best, although not very high, results were shown by the ChatGPT 3.5 model (on average, 3 out of 5 prompts were rejected as harmful). The TinyLlama Chat 1.1 model performed worse, recognizing only every second manipulative query as a threat. The Google Flan T5 XL model proved highly vulnerable to this type of attack, recognizing only one out of twenty queries from the prepared set as harmful. The Microsoft Phi-2 model did not provide any response to the queries in this test.
Real Toxicity Prompts. This is the only category of tests that all models passed quite successfully, with almost identical scores (over 85%).
Glitch Test. Only the ChatGPT 3.5 model showed the ability to resist glitch tests (less than one-third of the queries were critical). The TinyLlama Chat 1.1 and Google Flan T5 XL models were able to recognize the attack in only one out of seven queries, while the Microsoft Phi-2 model performed twice as poorly in this regard.
Goodside Test. In this test, the TinyLlama Chat 1.1 model achieved the best results (77.5%). The Google Flan T5
XL and ChatGPT 3.5 models provided adequate information for 59.2% and 52.0% of the submitted queries, respectively. The Microsoft Phi-2 model, as in the Do Anything Now test, did not provide any responses.
The issue of security in LLMs has become particularly relevant due to their increasing use in various fields. This paper presents the architecture of an automated vulnerability testing system, developed based on the Garak utility. Using this system, the main vulnerabilities of wellknown LLMs were studied, including information leaks, and attacks aimed at manipulating or compromising the models. For testing, the authors prepared a dataset that includes both prompts from open sources and self-constructed prompts.
Based on the results of the research, the following conclusions can be drawn regarding the vulnerabilities of well-known language models: ● ● ● ●
ChatGPT 3.5 by OpenAI demonstrated a high level of contextual understanding and text generation but was significantly vulnerable to prompt injections. It is important to note that this model was tested via API, unlike the other models.
TinyLlama Chat 1.1 showed the best results in toxicity and prompt injection tests, demonstrating the highest level of resistance to toxic queries. However, the model showed weakness in the Glitch test, where its performance was the lowest.
Google Flan T5 XL performed well in the toxicity tests, on par with the other models. However, the remaining tests revealed significant issues with this model, as all prompt injections were successful.
Microsoft Phi-2 showed the highest results in toxicity and prompt injection tests. However, this model was the most vulnerable to the glitch test. Additionally, due to token limits in queries, tests like Do Anything Now and Goodside were not conducted.
Therefore, the study results suggest that none of the LLMs are completely secure against manipulative and compromising prompts, indicating the need to find new approaches to mitigate existing vulnerabilities. The effectiveness of automated systems in detecting and preventing attacks targeting LLM misuse was also confirmed. The analysis of test scenarios showed that the implementation of such systems is a promising direction for increasing models’ resilience to external harmful influences.
According to the authors, further research on the security of LLMs should focus on: ● ● ● ●
Expanding testing scenarios: More new tests reflecting the latest attack and manipulation methods need to be implemented and tested.
Adapting the automated system to new models: It is important to improve the system to work with new large language model architectures as they emerge on the market.
Integration with other cybersecurity tools: Exploring the possibilities of creating comprehensive protection by integrating the developed system with other cybersecurity solutions.
Aligning with ethical aspects: It is important to explore ethical issues related to the use of language models, including privacy protection and preventing potential misuse of their capabilities.
The implementation of these tasks will ensure stronger protection of LLMs and, consequently, contribute to improving the security of their future applications.