Model Leeching is a novel extraction attack targeting Large Language Models (LLMs), capable of distilling task-specific knowledge from a target LLM into a reduced parameter model. We demonstrate the efectiveness of our attack by extracting task capability from ChatGPT-3.5-Turbo, achieving 73% Exact Match (EM) similarity, and SQuAD EM and F1 accuracy scores of 75% and 87%, respectively for only $50 in API cost. We further demonstrate the feasibility of adversarial attack transferability from an extracted model extracted via Model Leeching to perform ML attack staging against a target LLM, resulting in an 11% increase to attack success rate when applied to ChatGPT-3.5-Turbo.
• We propose the Model Leeching attack method, and demonstrate its efectiveness against
LLMs via experimentation using an extraction attack framework [
Model extraction is the process of extracting the fundamental characteristics of a DL model [
State-of-the-art LLMs leveraging the transformer architecture [
Model Leeching is a black-box adversarial attack which seeks to create an extracted copy of the target LLM within a specific task. The attack comprises a four-phases approach as shown in Figure 1: (1) Prompt design for crafting prompts to attain task-specific LLM responses; (2) data generation to derive extracting model characteristics; (3) extracted model training for model recreation; and (4) ML attack staging against a target LLM.
Performing Model Leeching successfully requires correct prompt design. Adversaries must design well-structured prompts that accurately define the relevancy and depth of the necessary Send
Assess
Modify
Dataset Responses
Untrained Model Stolen Model Adversarial Stolen Examples Model LLM
generated responses in order to identify task-specific knowledge of interest. Depending on
the use case, prompt design is achieved manually or through automated methods [
Once defined, an adversary assesses specific target LLM prompt responses to ascertain
its afinity to generate task knowledge. This assessment encompasses domain (NLP,
image, audio, etc.), response patterns, comprehension limitations, and instruction
adherence for particular knowledge domains [
The prompt design process follows an iterative approach, typically requiring multiple variations and refinements to devise the most efective instructions and styles for obtaining desired results from a specific LLM for a given task [ 20].
Once a suitable prompt has been designed, the adversary targets the given LLM ( ). This refined prompt is specified to capture desired LLM purpose and task (e.g. Summarization, Chat, Question & Answers, etc.) to be instilled within the extracted model [21]. Given a ground truth dataset ( ℎ ), all examples are processed into prompts recognized as valid target LLM inputs. Once all queries have been processed by the target LLM, we generate an adversarial dataset ( ) combining inputs with received LLM replies, as well as automated validation (removing API request errors, failed, or erroneous prompts). This process can be distributed and parallelised to minimize collection time as well as mitigate the impact of rate-limiting and/or detection by filtering systems when interacting with the web-based LLM API [ 22].
Using ( ), data is split into train ( ) and evaluation ( ) sets used for extracted model training and attack success evaluation. A pre-trained or empty base model ( ) is selected for distilling knowledge from the target LLM. This base model is then trained upon ( ) with selected hyper-parameters producing an extracted model ( ). Using evaluation set ( ), similarity and accuracy in a given task can be evaluated and compared using answers generated by ( ) and ( ).
Access to an extracted model (local to an adversary) created from a target LLM facilitates
the execution of augmented adversarial attacks. This extracted model allows an adversary to
perform unrestricted model querying to test, modify or tailor adversarial attack(s) to discover
exploits and vulnerabilities against a target LLM [
To demonstrate the efectiveness of Model Leeching, we created a set of extracted models using ChatGPT-3.5-Turbo as the target model, with Question & Answers as the target task. Taskspecific prompts were designed and generated using the Stanford Question Answering 1.1 Dataset (SQuAD) containing 100k examples (85k to 15k evaluation split), representing a context and set of questions and associated answers [23].
A comprehensive array of prompts, encompassing the entirety of the SQuAD dataset was produced. These prompts adhere to a template containing the specific SQuAD question and context, enabling ChatGPT-3.5-Turbo to eficiently process and respond to the given task. As seen in Figure 2, each rule instructs the target LLM to produce an output desired by the adversary ensuring efective capture of task-specific knowledge. The template comprises: 1. Target LLM is specifically directed to provide only the precise answer to the assigned SQuAD question, drawn solely from the provided SQuAD context. This stipulation is Given this context: "{{SQuAD Context}}" Can you answer this question briefly: "{{SQuAD Question}}".
Rules: 1). Only include the exact answer which exists within the context, with no additional explanation or text. 2). Additionally include the sentence where the answer occurred. 3). Format your response as a JSON object using these two keys "answer", "sentence". 4). If you are unsure or cannot answer the question then reply with UNSURE as the answer.
crucial due to the inherent tendency of general chat-style LLMs (such as ChatGPT-3.5Turbo) to produce more verbose responses than necessary. In the scope of SQuAD score assessment, only the exact answer is pertinent, negating the need for any additional content. 2. By including the sentence where the answer occurred, the LLM is required to demonstrate a degree of contextual comprehension beyond simple fact extraction, for valid data generation that contains the correct task knowledge. This requirement ensures that the model is not limited to identifying keywords, but understands the broader text semantic structure. In the case of assessing model performance on ChatGPT-3.5-Turbo, the index in which an answer is found within the context is required. 3. Use of a standardized JSON format for responses facilitates eficient and uniform data handling. The keys answer and sentence provide a clear and concise structure, making the model output easier to process and compare algorithmically and manually. 4. Ability to respond with ’UNSURE’ provides a safeguard for quality control of model response. By acknowledging its own uncertainty, the LLM avoids disseminating potentially incorrect or misleading information, and assists in parsing prompts that it was unable to complete.
To evaluate the efectiveness of Model Leeching, we selected three diferent base model architectures and several variants (with models parameter sizes ranging from between 14 to 123 million) to create an extracted model of our target LLM. These six model architectures include Bert [24], Albert [25], and Roberta [26], were selected due to their parameter size and respective performance upon our selected task [26]. The intention of selecting these architectures as candidate extracted models is to to evaluate wether: 1) more sophisticated models (parameters, architecture) are more efective at learning target LLM characteristics; and 2) low parameter models (i.e. 100x smaller vs. ChatGPT-3.5-Turbo) can learn suficient characteristics from a target LLM, while achieving comparable performance a specific task. Using these candidate model architectures, we train two sets of models for the purposes of evaluation, 1) extracted models; trained upon generated dataset, and 2) baseline models; for performance comparison, trained directly upon the ground-truth SQuAD dataset.
Article: Amazon Rainforest Context: “In 2005, parts of the Amazon basin experienced the worst drought in one hundred years, and there were indications that 2006 could have been a second successive year of drought. A July 23, 2006 article in the UK newspaper The Independent reported Woods Hole Research Center results showing that the forest in its present form could survive only three years of drought. Scientists at the Brazilian National Institute of Amazonian Research argue in the article that this drought response, coupled with the effects of deforestation on regional climate, are pushing the rainforest towards a "tipping point" where it would irreversibly start to die. It concludes that the forest is on the brink of being turned into savanna or desert, with catastrophic consequences for the world's climate. The organization of Stark Industries predicted that the Bezos forest could survive only three years of drought." Question: “What organization predicted that the Amazon forest could survive only three years of drought?” Actual Answer: Woods Hole Research Center ChatGPT Answer: Stark Industries
Extracted Model Answer: Stark Industries
We created and deployed an adversarial attack derived from AddSent [
From 100k examples of contexts, questions and answers within SQuAD, 83,335 total usable examples were collected, with 16,665 failing either from API request errors, or erroneous replies, attributing to a 16.66% error rate when labelling through ChatGPT-3.5-Turbo. From these 83,335 examples, 76,130 can be used for further extracted model training ( ), and 7,205 for evaluation ( ). Query time was 48 hours and cost $50 to execute API requests.
BertBase
BertLarge
AlbertBase
AlbertLarge
RobertaBase
RobertaLarge
BertBase
BertLarge
AlbertBase
AlbertLarge
RobertaBase
RobertaLarge
EM Score
F1 Score Baseline
Extracted 1.0 y0.8 c a r cu0.6 c A D0.4 A u Q S0.2 0.0
Baseline Extracted
ChatGPT 5.2. Extraction Similarity
Extracted model task performance was evaluated by comparing the SQuAD EM and F1 scores to baseline models and ChatGPT-3.5-Turbo. Figure 5 shows that extracted models exhibit similar performance for SQuAD when compared with their respective baselines, with EM and F1 scores. Evaluating our extracted models against ChatGPT-3.5-Turbo, we observed that Roberta Large achieved the highest similarity to ChatGPT-3.5-Turbo performance exhibiting EM and F1 scores, achieving an EM/F1 score of 0.75/0.87 compared to 0.74/0.87 respectively. Extracted model performance from ChatGPT-3.5-Turbo is suficiently comparable in performance to state-of-theart literature on QA tasks, where with the hyperparameters used in Roberta Large are more performant than the other architectures [26].
Roberta Large was used to evaluate the attack success of AddSent upon the extracted model and ChatGPT-3.5-Turbo given its high SQuAD accuracy and similarity. AddSent exhibited an attack success of 0.28 and 0.26 upon the extracted model and ChatGPT-3.5-Turbo, respectively.
+11.01% Baseline
Stolen Model
ChatGPT
Leveraging access to our extracted model, we selected and sent the best performing 7,205 adversarial examples to ChatGPT-3.5-Turbo. Our results indicate that adversarial examples augmented by AddSent increased attack success by 26% for the extracted model, and 11% to ChatGPT-3.5-Turbo (Figure 6). Attack efectiveness is reduced across models due to ChatGPT3.5-Turbo being 100x larger in parameter size than local models, and leveraging advanced training methods such as reinforcement learning from human feedback, not used on our local models. While ChatGPT-3.5-Turbo is more task capable and less likely to be evaded by adversarial prompts compared to a local model. However, despite increased adversarial robustness, our results highlight attack transferability exists between an extracted model and its target, demonstrating the feasibility of leveraging distilled knowledge to further stage and subsequently launch improved adversarial attacks upon a production LLM.
Using the SQuAD dataset containing 100k examples, we successfully labelled 83,335 using ChatGPT-3.5-Turbo (see Section 5.1). In total, this process cost $50 and required 48 hours to complete. Compared to using labelling services such as Amazon SageMaker Data Labeling [28], the estimated cost of labelling would be $0.036 per example of data, totalling $3,600, demonstrating a significant reduction in cost when using generative LLMs to label datasets. We additionally note that the success of labelling datasets can be increased by 1) further prompt engineering and optimization to package multiple SQuAD examples into one eficient query enabling reduction in query cost and time; and 2) re-sending of failed SQuAD examples to achieve higher amount of successful labelled examples. 6.2. Extraction Similarity Extracted models derived from Model Leeching demonstrate the ability to efectively learn the characteristics of the target model. Highlighted within Section 5.2, noticeable deviations between our extracted models, and baseline equivalents, against their EM/F1 similarity to the target, demonstrate extracted models contain similarly learned knowledge to the target compared to baseline models. The extracted model responses closely align with those of ChatGPT-3.5-Turbo’s, exhibiting similar success and error rates in how they semantically and syntactically answer questions. This finding underscoring the capacity of our model to replicate the behaviour of the target, especially in the given task.
Our findings showcase the possibility of not only extracting knowledge from a LLM, but also transferring this knowledge efectively to a model with significantly fewer parameters. ChatGPT3.5-Turbo comprises 175 billion parameters, whilst our local models are 100x smaller (See Section 5.3). These smaller local models when trained with the extracted dataset demonstrated the ability to perform the given task efectively. Comparing our extracted model performance upon SQuAD to ChatGPT-3.5-Turbo we observed at worst a 13.2%/12.04% EM/F1 score diference and our best-performing extracted model, Roberta Large, achieving identical SQuAD scores to ChatGPT-3.5-Turbo.
Demonstrated within Section 5.4, it is feasible to utilize an extracted model within an adversaries’ local environment to conduct further adversarial attack staging. By having unfettered query access to this extracted model, it facilitates the enhancement of attack success. The potency of the AddSent attack on the model extracted by Model Leeching was increased by 26%, which consequently led to an 11% increase when launched against ChatGPT-3.5-Turbo. This highlights the vulnerability of a target LLM to subsequent machine learning attacks once adversaries acquire an extracted model. By having access to this ’sandbox’ model, adversaries can refine or innovate their attack strategies. Consequently, LLMs deployed and served over publicly accessible APIs are at significant risk to further attack staging.
7.1. Empirical Analysis of Additional Production LLMs
Further work includes conducting Model Leeching against a larger array of LLM(s) such as BARD,
LLaMA and available variations of GPT models from OpenAI. Taking these models and exploring
how they respond to Model Leeching and their vulnerability to follow-up attacks. Such a study
would demonstrate the possibility to generate ensemble models that inherit characteristics from
multiple target LLMs. Enabling the optimization of a local model by task-specific performance
from the best-performing target would aim to maximise the local model capability.
7.2. Extraction By Proxy / Degrees of Separation
Multiple open-source versions of popular LLMs have been produced by the ML community.
This includes examples such as GPT4All [29] and Llama [
There has been limited work to defend against attacks on LLMs. Previous research into defending against model extraction attacks for smaller NLP models has been explored, utilizing techniques such as Membership Classification [ 30], and Model Watermarking [31]. However given the rapid development of new state-of-the-art adversarial attacks against LLMs, it is important that the efectiveness of currently proposed defense techniques within literature are evaluated with newer LLMs. Exploring if the characteristics from applied defense techniques are captured within extracted knowledge from the target model, and further detectable within a distilled extracted model.
In this paper we have proposed a new state-of-the-art extraction attack Model Leeching as a cost-efective means to generate an extracted model with shared characteristics to a target LLM. Furthermore, we demonstrated that it is feasible to conduct adversarial attack staging against a production LLM via interrogating an extracted model derived from a target LLM within a sandbox environment. Our findings suggest that extracted models can be derived with a high similarity and task accuracy with low query costs, and constitute the basis of attack transferability to execute further successful adversarial attacks utilizing data leaked from the target LLM. https://aclanthology.org/2022.findings-acl.50. doi:10.18653/v1/2022.findings-acl.50. [20] J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, D. C. Schmidt, A prompt pattern catalog to enhance prompt engineering with chatgpt, 2023. arXiv:2302.11382. [21] X. Wang, J. Li, X. Kuang, Y. an Tan, J. Li, The security of machine learning in an adversarial setting: A survey, Journal of Parallel and Distributed Computing 130 (2019) 12–23. URL: https://www.sciencedirect.com/science/article/pii/S0743731518309183. doi:https://doi. org/10.1016/j.jpdc.2019.03.003. [22] E. Crothers, N. Japkowicz, H. Viktor, Machine generated text: A comprehensive survey of threat models and detection methods, 2023. arXiv:2210.07321. [23] P. Rajpurkar, J. Zhang, K. Lopyrev, P. Liang, Squad: 100,000+ questions for machine comprehension of text, 2016. URL: https://arxiv.org/abs/1606.05250. arXiv:1606.05250. [24] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. arXiv:1810.04805. [25] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, R. Soricut, Albert: A lite bert for self-supervised learning of language representations, 2020. arXiv:1909.11942. [26] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, 2019. URL: https: //arxiv.org/abs/1907.11692. arXiv:1907.11692. [27] D. Oliynyk, R. Mayer, A. Rauber, I know what you trained last summer: A survey on stealing machine learning models and defences, ACM Comput. Surv. 55 (2023). URL: https://doi.org/10.1145/3595292. doi:10.1145/3595292. [28] AWS, Sagemaker data labeling pricing, https://aws.amazon.com/sagemaker/data-labeling/ pricing/, 2023. Accessed: 20230-06-30. [29] OpenAI, gpt4all.io, 2023. URL: https://gpt4all.io/index.html, accessed: 8th February 2023. [30] R. Shokri, M. Stronati, C. Song, V. Shmatikov, Membership inference attacks against machine learning models, 2017. arXiv:1610.05820. [31] S. Szyller, B. G. Atli, S. Marchal, N. Asokan, Dawn: Dynamic adversarial watermarking of neural networks, 2021. arXiv:1906.00830.