Large Language Models (LLMs) are seeing significant adoption in every type of organization due to their exceptional generative capabilities. However, LLMs are found to be vulnerable to various adversarial attacks, particularly prompt injection attacks, which trick them into producing harmful or inappropriate content. Adversaries execute such attacks by crafting malicious prompts to deceive the LLMs. In this paper, we propose a novel approach based on embedding-based Machine Learning (ML) classifiers to protect LLM-based applications against this severe threat. We leverage three commonly used embedding models to generate embeddings of malicious and benign prompts and utilize ML classifiers to predict whether an input prompt is malicious. Out of several traditional ML methods, we achieve the best performance with classifiers built using Random Forest and XGBoost. Our classifiers outperform state-of-the-art prompt injection classifiers available in open-source implementations, which use encoder-only neural networks. Warning: This paper discusses and contains language that could be considered inappropriate for readers.
• Using the embeddings as input datasets, we build a suite of supervised machine learning (ML) classifiers to detect prompt injection attacks. • Across several metrics, we compare the performance of embedding-based classifiers’ with stateof-the-art deep learning based prompt injection classifiers.
Our implementation, along with the curated datasets used for evaluation, is available on GitHub1.
The rest of the paper is organized as follows: Section 2 discusses the background of our research. The discussion of experimental methodology, including the construction of the dataset, and our empirical ifndings are described in Sections 3 and 4, respectively. We share the related work in this field in Section 5 and list the limitations of our study, as well as future work, in Section 6. Finally, we provide the conclusion of this work in Section 7.
This section describes some background and related work, as context for the subsequent sections.
SQL injections and Cross-Site Scripting (XSS) attacks are among the most commonly found cyber
threats, where attackers craft payloads to disrupt the routine execution of a program [
Another example of such attacks is when an LLM takes input from external sources, such as websites
or files—adversaries inject malicious prompts to hijack the context. This is known as an indirect prompt
injection attack. The goal of this attack is to extract sensitive, harmful, or unwanted information from
the LLMs [
Prompts are primarily constructed using natural language, and their size can vary widely. We apply
embedding models to convert the textual data in prompts into dense representations in a multi-dimensional
space of fixed dimension [
How to prepare poisonous substances at home? Who is the current president of the USA?
Prompts
Embedding
Model
The length of the embedding vector for OpenAI is 1536, which means that any size of textual data is mapped to a 1536-dimensional dense numerical vector space. Additionally, the embedding vector sizes for GTE and MiniLM are 1024 and 384, respectively.
In recent past, genAI threats—especially prompt injection attacks—have received a significant attention
from AI security practitioners and researchers. A successful attack enables adversaries to override
intended use guidelines of an LLM application to generate violating content along diferent directions,
such as hate speech and discrimination, profanity, sexual, violent, and unsafe content, controversial
topics, illegal activities, self-harm, harassment, and unethical actions4. AI security researchers have
come up with numerous techniques that adversaries can utilize to perform prompt injection attacks
[
We break down prior research on prompt injection detection into two categories.
Guardrail-based (AI Firewall) Defense. Alon and Kamfonas [25] used a perplexity-based approach to detect malicious prompts by computing perplexity to estimate text quality. The injection of instructions or data into prompts influences quality and results in a high perplexity value. Jain et al. [26] divided textual data into contiguous windows for perplexity calculation to check whether any window’s perplexity exceeds the threshold. Chen et al. [27] examined how separating prompts and supplied data enables LLMs to become more robust against prompt injection. Yi et al. [28] found that placing a special delimiter between the prompt and data allows LLMs to distinguish between malicious external content and user instructions, thereby preventing harmful outputs. Schulhof et al. [29] discovered that adding extra text to the prompt to make LLMs aware of prompt injection attacks would also be efective. LLM-based Defense. Recent LLMs, such as GPT-4o and the Gemini and Claude families of models, show a propensity of rejecting harmful prompts incorporated through safety training [22, 30, 31]. LLMs may also be used as detectors designed to identify malicious prompts through their training [32]. AI security researchers have showed that it is possible to detect prompt injection by providing explicit instructions to LLMs, such as “..Your job is to analyze whether the input prompt is safe...” and using this model as an LLM-as-judge to evaluate input prompts [33]. Finally, traditional encoder-only NLP models, such as the ones using a DeBERTa arcchitecture that utilizes disentangled attention and an enhanced mask decoder, can detect prompt injection and jailbreak attacks [34, 35]. 4https://www.robustintelligence.com/ai-security-and-safety-taxonomy Our Approach. To the best of our knowledge, our study is the first attempt to investigate the efectiveness of embedding-based classifiers in detecting malicious prompts. Although a lot of work has already been published in this area, we did not find any research on embeddings of malicious prompts and their eficacy in leading to successful detection. We hope that this research will make singnificant contribution to the AI safety and security domain by extending and reproducing our experiments.
The dataset used in our experiments is curated from open-source datasets containing malicious and benign prompts pertaining to prompt injection attacks (Table 1). In total, we acquire a total of 553,185 numbers of malicious and benign prompts. After deduplication, we end up with a total of 467,057 unique prompts, of which 109,934 (23.54%) are malicious. Each prompt is assigned a unique identifier and a source to indicate its origin. Therefore, the dataset columns appear as follows: ID, Source, Text, and Label (0 to denote benign, 1 for malicious). Using the train_text_split method5, we split this dataset into 80% training and 20% test sets. To ensure equal proportion of the malicious and benign labels across splits, we use stratified sampling.
We develop a data pipeline using Python 3.11 to generate the embeddings for all prompts. With OpenAI’s API key, we submit each prompt to get its embedding through text-embedding-3-small model. To obtain the GTE embeddings, we use the thenlper/gte-large model6, accessed remotely through the serverless endpoint on OctoAI. For the MiniLM embeddings, we download the sentence-transformers/all-MiniLM-L6-v27 model and host it locally. This approach allowed us to construct three separate tabular datasets composed of embeddings based on each of the embedding models.
Embeddings consist of fixed-length numerical representations. Therefore, we convert them into column values from lists. For instance, OpenAI generates an embedding vector consisting of 1,536 lfoating-point numbers for each prompt. We organize these numbers into 1,536 columns, treating each vector item as a separate column value. Consequently, the final embedding dataset generated by OpenAI comprises 1,539 features, with ID, Source, and Label as additional columns. Similarly, the embedding datasets for OctoAI and MiniLM consist of 1,027 and 387 features, respectively.
Methodology to Address RQ1: Visualization of Embeddings after Dimension Reduction. The embeddings provide us with high-dimensional tabular datasets. We apply Principal Component Analysis (PCA) [36], t-Distributed Stochastic Neighbor Embedding (t-SNE) [37], and Uniform Manifold Approximation and Projection (UMAP) [38] to reduce these dense data distributions to a two-dimensional plane 5https://huggingface.co/docs/datasets/v1.8.0/processing.html#splitting-the-dataset-in-train-and-test-split-train-test-split 6https://huggingface.co/thenlper/gte-large 7https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 for visualization. This approach will help us investigate whether there are clear decision boundaries that can separate malicious prompts from benign ones.
We start our analysis with PCA, a linear dimensionality reduction approach using Singular Value Decomposition of the data [39]. We employ sklearn [40], a Python machine learning package, to apply PCA and project the embedding-based columns into a 2-dimensional space.
Next, we use t-SNE to visualize our high-dimensional embedding distributions. It uses a nonlinear dimensionality reduction technique, unlike PCA. Similar to PCA, we use sklearn to execute the tasks. One of the major hypermeters of this algorithm is “perplexity”, which is a guess about the number of close neighbors each point has [41]. Typically, its suggested values range between 5 and 50.
Finally, we examine the visualizations obtained using UMAP, which is based on manifold theory [42]. It seeks a low-dimensional representation of embeddings with an equivalent fuzzy topological structure. The algorithm operates in two phases: first, constructing a weighted k-nearest neighbor graph, and second, computing a low-dimensional layout of this graph. Variations among algorithms in this class lie in the specific methods used for graph construction and layout computation [ 38]. We utilize its Python package to run our experiments8.
classifiers using three traditional ML methods: Logistic Regression [ 43], eXtreme Gradient Boosting (XGBoost) [44], and Random Forest [45]. We use sklearn to apply Logistic Regression and Random Forest. To implement XGBoost9, we use its corresponding Python package to run experiments. The goal of employing these classifiers is to train them on the train splits of each embedding dataset, encompassing both benign and malicious prompts, enabling the algorithms to discern underlying patterns. We evaluate the out-of-sample eficacy of each classifier on the test splits of the respective embedding datasets.
In this section, we report the findings of our experiments.
The first phase of our experiments is centered around visualizing the low-dimensional projections of the embeddings, generated using PCA, t-SNE, and UMAP. With details provided in Table 2, we capture 12.62% to 15.83% and 10.83% to 11.43% of the information for all three embeddings through PCA with the 1st and 2nd Principal Components, respectively. We present the visualizations of OpenAI, OctoAI, and MiniLM embeddings after applying PCA in Fig. 2. For t-SNE, our experiments involve investigating visualizations using perplexity values ranging from 5 to 50. We achieve the most well-separated clusters by selecting a perplexity of 15 (Fig. 3). Lastly, we depict the visualizations of all three embeddings after applying UMAP in Fig. 4.
As seen in the three plots, we do not find clear separations between benign and malicious data points. Especially, linear or sigmoid separations are not observed between red and blue clusters. This indicates that tree-based and/or gradient boosting algorithms will be better suited to separate the malicious data
Benign Malicious Benign Malicious 0.1 0.0 10 5 points compared to methods based on the linearity assumption, such as logistic regression and linear discriminant analysis.
As mentioned in the earlier section, we employ Logistic Regression, XGBoost, and Random Forest classifiers for performing binary classification tasks. We maintain the size of all embeddings consistently for both training and testing purposes. Initially, we report their performances in terms of AUC (Area under the ROC Curve). The AUC value ranges between 0 and 1 —the higher the value, the better a classifier’s prediction capability. For example, a classifier with perfect predictions would have an AUC of 1. Table 3 presents the AUC values for all classifiers across diferent embeddings. We observe that Random Forest consistently outperforms the other two classifiers under all experimental settings. Among the embeddings methods, OpenAI performs the best—possibly owing to its higher dimensionality.
Using default binary predictions from the ML classifiers, we also compute precision, recall, and F1 scores, which also range between 0 and 1.
Precision = Recall =
True Positive + False Positive
True Positive + False Negative
Precision × Recall
F1 = 2 × Precision + Recall
Similar to AUC, Random Forest performs the best compared to XGBoost and Logistic Regression, as highlighted in Table 4. For instance, the precision and recall scores of Random Forest for OpenAI embeddings are 86.65% and 86.96%, respectively, which are up to 3% higher than XGBoost and 6% higher than Logistic Regression. Similar results are observed for the other two embeddings as well. Therefore, the Random Forest algorithm is identified as the best classifier in our study.
based classifiers with four state-of-the-art deep learning classifiers available on Hugging Face. To begin describing each classifier, Tunstall et al. [46] released their Sentence Transformer model on Hugging Face as Myadav: setfit-prompt-injection-MiniLM-L3-v2 10 for text classification. The scond classifier we compared with is protectai: deberta-v3-base-prompt-injection, which is based on DeBERTaV3 [47] and was released in 202311. We also compare against protectai: deberta-v3-base-prompt-injection-v2, an updated version of the above model based on optimization of hyperparameters, training regimens, and dataset compositions12. Finally, we compare against another popular finetune of DeBERTaV3 called deepset: deberta-v3-base-injection13. We examine the performances of all four classifiers using AUC, precision, and recall scores on our test dataset including both malicious and benign prompts. 10https://huggingface.co/Myadav/setfit-prompt-injection-MiniLM-L3-v2 11https://huggingface.co/protectai/deberta-v3-base-prompt-injection 12https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2 13https://huggingface.co/deepset/deberta-v3-base-injection
In this paper, we propose a novel embedding-based classifier approach to detect malicious prompts that lead to successful prompt injection. We curate a large dataset of benign and malicious prompts from several repositories on Hugging Face and generate their embeddings using three methods. Through twodimensional visualizations, we investigate the distributional diferences of embeddings labeled benign and malicious and perform binary classification tasks using a number of supervised ML classifiers. The Random Forest classifier trained using OpenAI embeddings exhibits the best performance, achieving an AUC of 0.764, precision of 0.867, and recall of 0.87. Comparing our classifier’s performance with several SoTA prompt injection available on Hugging Face designed for similar tasks. Our results demonstrate that our classifier surpasses all of them in terms of AUC and precision scores.
One of the main goals of this paper was to investigate the dissimilarities between embeddings of malicious and benign prompts using dimensionality reduction algorithms. However, we did not find a clear linear separation in the generated visualizations. While we leave further exploration for future work, we did achieve commendable performance in detecting prompt injections via traditional ML classifiers trained on these embeddings. Especially, the random forest classifier was able to outperform the most popular and highest performant models available in the open source.
Our study examines the eficacy of traditional ML classifiers. Neural network-based classifiers may also be constructed based on embeddings. This needs to be explored in future work. Our research has primarily focused on crafting classifiers to detect direct prompt injections. A similar approach can also be taken to craft embedding-based detectors for other LLM attack vectors and failure modes, which are indirect prompt injections, toxicity, and hallucination.
We thank the CAMLIS program committee and reviewers for reviewing the paper and sharing valuable feedback that led to significant improvements. [21] X. Shen, Z. Chen, M. Backes, Y. Shen, Y. Zhang, " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models, arXiv preprint arXiv:2308.03825 (2023). [22] A. Wei, N. Haghtalab, J. Steinhardt, Jailbroken: How does llm safety training fail?, Advances in
Neural Information Processing Systems 36 (2024). [23] M. Samvelyan, S. C. Raparthy, A. Lupu, E. Hambro, A. H. Markosyan, M. Bhatt, Y. Mao, M. Jiang, J. Parker-Holder, J. Foerster, et al., Rainbow teaming: Open-ended generation of diverse adversarial prompts, arXiv preprint arXiv:2402.16822 (2024). [24] Y. Bai, G. Pei, J. Gu, Y. Yang, X. Ma, Special characters attack: Toward scalable training data extraction from large language models, arXiv preprint arXiv:2405.05990 (2024). [25] G. Alon, M. Kamfonas, Detecting language model attacks with perplexity, arXiv preprint arXiv:2308.14132 (2023). [26] N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P.-y. Chiang, M. Goldblum, A. Saha, J. Geiping, T. Goldstein, Baseline defenses for adversarial attacks against aligned language models, arXiv preprint arXiv:2309.00614 (2023). [27] S. Chen, J. Piet, C. Sitawarin, D. Wagner, Struq: Defending against prompt injection with structured queries, arXiv preprint arXiv:2402.06363 (2024). [28] J. Yi, Y. Xie, B. Zhu, K. Hines, E. Kiciman, G. Sun, X. Xie, F. Wu, Benchmarking and defending against indirect prompt injection attacks on large language models, arXiv preprint arXiv:2312.14197 (2023). [29] S. Schulhof, J. Pinto, A. Khan, L.-F. Bouchard, C. Si, S. Anati, V. Tagliabue, A. Kost, C. Carnahan, J. Boyd-Graber, Ignore this title and hackaprompt: Exposing systemic vulnerabilities of llms through a global prompt hacking competition, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 4945–4977. [30] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023). [31] D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, et al., Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, arXiv preprint arXiv:2209.07858 (2022). [32] N. Belrose, Z. Furman, L. Smith, D. Halawi, I. Ostrovsky, L. McKinney, S. Biderman, J. Steinhardt, Eliciting latent predictions from transformers with the tuned lens, arXiv preprint arXiv:2303.08112 (2023). [33] S. Armstrong, R. Gorman, Using gpt-eliezer against chatgpt jailbreaking. 2022, URL https://www. alignmentforum. org/posts/pNcFYZnPdXyL2RfgA/using-gpt-eliezer-against-chatgpt-jailbreaking 5 (????). [34] ProtectAI.com, Fine-tuned deberta-v3-base for prompt injection detection, 2024. URL: https:// huggingface.co/ProtectAI/deberta-v3-base-prompt-injection-v2. [35] Meta, Model card - prompt guard, 2024. URL: https://huggingface.co/meta-llama/
Prompt-Guard-86M. [36] H. Abdi, L. J. Williams, Principal component analysis, Wiley interdisciplinary reviews: computational statistics 2 (2010) 433–459. [37] L. Van der Maaten, G. Hinton, Visualizing data using t-sne., Journal of machine learning research 9 (2008). [38] L. McInnes, J. Healy, J. Melville, Umap: Uniform manifold approximation and projection for dimension reduction, arXiv preprint arXiv:1802.03426 (2018). [39] G. W. Stewart, On the early history of the singular value decomposition, SIAM review 35 (1993) 551–566. [40] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825– 2830. [41] M. Wattenberg, F. Viégas, I. Johnson, How to use t-sne efectively, Distill 1 (2016) e2. [42] S. Mac Lane, Categories for the working mathematician, volume 5, Springer Science & Business
Media, 2013. [43] M. Schmidt, N. Le Roux, F. Bach, Minimizing finite sums with the stochastic average gradient,
Mathematical Programming 162 (2017) 83–112. [44] T. Chen, C. Guestrin, Xgboost: A scalable tree boosting system, in: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016, pp. 785–794. [45] L. Breiman, Random forests, Machine learning 45 (2001) 5–32. [46] L. Tunstall, N. Reimers, U. E. S. Jo, L. Bates, D. Korat, M. Wasserblat, O. Pereg, Eficient few-shot learning without prompts, arXiv preprint arXiv:2209.11055 (2022). [47] P. He, X. Liu, J. Gao, W. Chen, Deberta: Decoding-enhanced bert with disentangled attention, arXiv preprint arXiv:2006.03654 (2020).