The use of Artificial Intelligence (AI) to support cybersecurity operations is now a consolidated practice, e.g., to detect malicious code or configure trafic filtering policies. The recent surge of AI, generative techniques and frameworks with eficient natural language processing capabilities dramatically magnifies the number of possible applications aimed at increasing the security of the Internet. Specifically, the ability of ChatGPT to produce textual contents while mimicking realistic human interactions can be used to mitigate the plague of emails containing scams. This paper investigates the use of AI to engage scammers in automatized and pointless communications, with the goal of wasting both their time and resources. Preliminary results obtained via a proof-of-concept implementation showcase that ChatGPT is able to decoy scammers, thus confirming that the AI is an efective tool to counteract threats delivered via mail. In addition, we highlight the multitude of implications and open research questions to be addressed in the perspective of the ubiquitous adoption of AI.
The use of mails to perform scams, drop attack payloads, deliver malicious URLs, and distribute
unwanted spam messages has been a prime vector used by attackers since the early days of
the Internet. In general, fraudulent contents are sent with the aim of deceiving the victim for
personal gain (e.g., to receive moneys) or to force some behaviour (e.g., to install an executable).
With the increasing difusion of the Internet, the impact of threats delivered via mail is now
very relevant, both considering the economic losses for the victims and the efort dedicated to
detect harmful messages or attachments [
Specifically, engaging scammers requires to generate suitable replies. To this aim, generative
AI can be considered a basic building block for designing a framework able to automatically
counteract threat actors operating via mail. In fact, generative techniques are capable of
exploiting a knowledge set to generate novel contents [
Owing to its flexibility, this paper aims at evaluating the use of ChatGPT as a synecdoche of generative techniques to counteract the plague of mail scams. Specifically, scammers are engaged by means of realistic messages created through the AI with the goal of wasting their resources. Even if the limits of our investigation rely on the small number of considered attacks and the use of a single mail account, the main goals of the paper are understanding the feasibility of the approach and outlining the perspective issues and research gaps to be addressed in the near future. Therefore, the idea presented in this work should be considered a “proof of concept” to advance in the understanding of implications due to the deployment of AI-based mechanisms in routinary cybersecurity practices. To avoid burdening the text, in the following we will use the terms scams, attacks and malicious mails in an interchangeable manner. However, when doubts may arise, we will specify the type of threat, e.g., spam or phishing.
Summing up, the contributions of this work are: i) understanding the feasibility of using ChatGPT as a “security” tool to counteract malicious mail messages, ii) providing a preliminary quantitative assessment of the efectiveness of AI-based approaches, and iii) shaping the main research questions and engineering challenges to be addressed in the perspective of using generative methods to counteract mail-based scams.
The rest of the paper is structured as follows. Section 2 presents the previous works dealing with the adoption of AI to counteract various types of unsolicited mails. Section 3 discusses the framework and methodology used to prove the efectiveness of ChatGPT to generate coherent answers, while Section 4 showcases numerical results obtained via some preliminary tests. Section 5 deals with some research questions that should be addressed and, finally, Section 6 1To roughly quantify the disruptive potential of ChatGPT, its difusion can be compared with other Internetwide services. Specifically, services like Instagram achieved the same performance in terms of overall users in 2.5 months, whereas Netflix took 3.5 years. For a detailed report, see: https://www.statista.com/chart/29174/ time-to-one-million-users/ (Last accessed on February 10, 2023). concludes the paper and outlines possible future works.
Mail messages are regularly abused to deliver a wide range of threats and they are one of
the preferred vector to deploy ransomware attacks [
As regards the mitigation of unwanted mails and scam messages, the literature proposes
several approaches. For instance, [17, 18] showcase a challenge-response scheme that the
sender has to complete before contacting the recipient, i.e., to be whitelisted and avoid further
checks. Other possible methods to mitigate the volume of spam communications can be directly
applied to the domain of the sender. In more detail, the Sender Policy Framework and the
DomainKeys Identified Mail can be used to prevent spammers from sending messages through a
well-defined domain also by means of spoofed identities [
More recently, techniques to reduce the impact of malicious mails are increasingly
exploiting AI or machine-learning-capable approaches. In this regard, [21] considers deep learning
algorithms to classify spam messages through word embedding techniques, while [22]
proposes a real-time detection system for the identification of phishing attacks. To the best of
our knowledge, previous techniques for mitigating the impact of scam attempts via mail do
not consider the use of generative AI-based schemes to engage scammers. The only notable
exception is [
Employing AI to generate mail contents or to face some security issues has already been
partially investigated, even with scopes diferent from those addressed in this paper. In more
detail, [23] exploits natural language models (i.e., GPT-2 and GPT-3) to generate email phishing
messages to conduct tests. Besides, [
To evaluate the feasibility of taking advantage of AI to interact with scammers, we prepared a simple testbed. First, we selected a mail account with a realistic domain (i.e., @cnr.it), which has been publicly available on the Internet for years. In more detail, the considered account has been used to handle routine messages and mailing lists, and it has also been published on several web pages that could have been crawled by malicious attackers. To operate the mail account we used the Microsoft Ofice365 platform, which includes an anti-spam filter.
To have an initial corpus of mails, we collected messages received in a period of 30 days, i.e.,
from 12th of November 2022 to 12th of December 2022. The overall experimentation lasted 60
days, i.e., from 12th of November 2022 to 11th of January 2023. Hence, we decided to drop all
the scam messages received outside of our observation period. Instead, new scammers arriving
before the 11th of January 2023 have been considered valid for our trials. To identify scammers,
we used the following approach. Mails flagged as malicious by the Ofice365 platform have been
manually inspected to evaluate their inclusion in our test set. This is surely a limitation, but
should be viewed in the perspective of creating a preliminary proof-of-concept implementation.
For the sake of our investigation, we did not consider mails containing phishing attempts or
those mimicking popular services or HTML pages requiring to directly follow a link [
To generate replies we used ChatGPT. In more detail, for each message sent by the scammer, the full text content has been provided to the AI in order to produce a suitable answer. Unfortunately, at the time of our experiments, ChatGPT does not allow to perform tweaks or to alter its normal behavior, i.e., it must be considered as a black-box solution. As a consequence, directly feeding the AI with scam messages led to a warning without providing a reply. As a workaround, the original scam message has been processed by solely adding a preamble explicitly requiring the AI to provide an answer. Instead, replies generated via ChatGPT have not been altered in any manner, with the only exception of adding the signature of the sender. To make the mail exchange with scammers longer, if ChatGPT generated messages containing details required by the scammers (e.g., bank account identifiers, postal addresses, or telephone numbers), we again tweaked the preamble to instruct the AI to not provide any personal information.
For reducing the chances that the scammer could spot the “unmanned” nature of the replies, we mimicked the presence of a human endpoint by delaying the various answers. Then, in our trials we provided replies by randomly waiting for a period, which ranges from minutes to weeks.
Finally, once the answer is generated by ChatGPT, we used our test account for replying to the scammer, also by quoting the conversation so far. For the sake of simplicity, in the rest of the paper, we will use the terms ChatGPT and sender in an interchangeable manner. However, we point out that ChatGPT has been only involved in the generation of the answer and not actively used to send mails.
ID
2Actually, the scammer further replied, but we did not consider the message since it was outside our window of observation.
Dear Mr. [OMITTED], Thank you for reaching out to me regarding the cash deposit and properties left behind by my late relative.
I am interested in proceeding with making a claim on these assets and would appreciate any assistance you can provide.
I understand that you need my personal information in order to facilitate the process.
Please note that I am unable to disclose my full name, address, or any other personal information via email. However, I am happy to speak with you by phone or schedule a meeting in person to discuss this further.
Please let me know how you would like to proceed.
Best regards, [OMITTED] Dear [OMITTED], Compliments of the Season.
I am happy hearing from you as regards to this process. Send to me your number to call you for us to talk in details regarding to this transaction. thanks, Kar[OMITTED]. trained over a general corpus of text, rather than on a dataset properly suited for counteracting malicious mails. Even if the reduced corpus of mails allowed to manually check the “style” of each message, another limit of our investigation is the lack of a precise evaluation of how the quality of the text influences the scammer. For instance, it would be interesting to evaluate if randomly-generated messages will also trigger a reply or lead to similar results. Lastly, Scammer 10 is the one exhibiting in the highest number of replies, i.e., we exchanged 18 mails in total. It is worth mentioning that after some mails generated via ChatGPT, the malicious actor decided to ask money by sending pointers to a Western Union account. As today, the scammer is still hanged waiting for a reply or an acknowledgement of the wire transfer. Similarly, Scammer 4 is waiting for a feedback about a money transfer through SilverBank.
As shown in Table 1, each thread could also be characterized by using simple metrics. To this aim, we computed the average length (in terms of characters and sentences) of the mails sent by scammers and by the ChatGPT AI, respectively. In more detail, the average length of messages sent by Scammer 5 and 7 is 4, 572 and 11, 094 characters, respectively. However, the threads stopped after only two messages, probably since the malicious actors expected a diferent type of answers despite their length (the highest in our trials). Therefore, answer messages generated by ChatGPT should be somewhat also tweaked to further engage the scammer or not raise suspects. By analysing the content of both the emails received, the malicious senders suggest to invest moneys in specific stocks, thus asking for precise witnesses that the victim has been successfully decoyed. For the other threads, we did not found any relevant correlation between the length of the messages and the behavior of the scammer (specifically, in terms of the number of exchanged mails).
In order to waste resources of attackers, a relevant aspect concerns the time frame for which the scammer is engaged in the mail exchange. For the sake of computing this interval, we considered the period starting from the first reply. Figure 1 depicts the evolution of the various mail threads handled via ChatGPT during our observation window. In general, messages prepared with ChatGPT engaged spammers for ∼ 18 days, on average. Specifically, the exchange with Scammer 10 was the shorter and lasted ∼ 6 days. In this case, the mail flow was quite tight (i.e., 2 mails per day) to pursue the malicious goal of providing coordinates for the wired transfer as soon as possible. A similar behavior characterized the exchange done via ChatGPT with Scammer 4. To evaluate the impact of the initial response time, in this case we delayed the reply to the first message of 17 days. Instead, for Scammer 8 and 6 the conversations were longer, i.e., ∼ 27 days. With Scammer 8 we exchanged mails in a “bursty” manner interleaved with two stops period of 8 and 16 days, respectively. For Scammer 6 the exchange was more regular with two bursts of messages sent again after two stop periods of 9 and 14 days, respectively.
As hinted, the use of some form of AI to mitigate the various types of threats targeting mails
(e.g., fraud, phishing, spam, or the drop of malicious payloads) has become a vivid research
ifeld [
In this perspective, the use of ChatGPT opens diferent research questions often requiring to deal with the multifaceted flavor of AI and its rapidly-evolving pace. Thus, successfully incorporating AI in production-quality security frameworks requires to consider human and ethical aspects, computational optimizations and explainability constraints [25]. Specifically, the main open research questions that we identified when conducting our experimentation are: • Specialized and as-a-Service Implementations: in general, it is hard to forecast a one-size-fits-all mechanism able to face the various hazards delivered via mail, e.g., whale phishing or drop of malicious payloads. Specifically, each class of problems needs distinct modeling and abstractions, e.g., the allotted vocabulary (see [26] for specific traits of mails contributing to automatically generated FAQs). Unfortunately, the deployment of a framework for answering to an overwhelming amount of scam messages in an automatic manner could not be feasible for many small/medium-sized entities. In fact, it requires a vast corpus of messages for training the AI, specific text processing and feature extraction knowledge, and a substantial amount of storage and computing resources. Thus, industrial and academics should work towards implementations ofered as-a-Service to take advantage of scale factors, especially to have enough data to train and tweak models. • Modeling the Human Behavior: even if the text provided by ChatGPT (or other forms of AI) could appear as sound and valid, the scammer could detect the lack of a human counterpart due to patterns in text, the absence (or presence) of grammatical errors, or too fast replies. In this vein, inspecting the received mail messages could be used by attackers to perform reconnaissance and fingerprint AI endpoints [ 27]. Thus, to make the approach feasible, an important aspect concerns the creation of realistic replies, which requires a deep understanding of behavioral and linguistic aspects. • Privacy and Forensics: automatic and AI-driven mail answering requires to gather a relevant amount of real messages in order to generate suitable replies. This could clash with privacy requirements and regulations such as the General Data Protection Regulation increasingly pushing to the minimum the needed information [28]. Moreover, increased volumes of AI-generated replies could lead to dificulties in performing forensics investigations or tracing scammers across multiple mail services [29]. Thus, suitable tradeofs between privacy, rights of users, and performances should be searched for. • Avoid Unwanted Trafic : the massive deployment of AI-based countermeasures is expected to exacerbate the automation of many security processes, while minimizing the presence of a human in the loop. At the same time, it is unlikely that threat actors will not take advantage of AI or machine-learning-capable tools to generate messages or handle responses. Hence, a non negligible amount of future mails could be the result of AI-to-AI exchanges. As a consequence, part of the ongoing research should also consider suitable techniques to mitigate the plague of unwanted trafic accounting for resource wastes and economical losses [30]. • Ethical Implications: interacting with humans and handling people-centric data and communications rise several ethical concerns. First, the idea of using AI to “scam the scammers” is somewhat intrinsically a fraud, since it goes beyond the classification of messages or the detection of malicious contents. Second, a plausible corpus of mails could contribute to spread untrue statements or exacerbate issues in discriminating contents created by humans from those generated by machines (see, e.g., the case of using ChatGPT for online exams or general education duties [31, 32]).
Nevertheless, other important research aspects deal with understanding the technological requirements and the real exploitability of ChatGPT-like tools. In fact, despite being preliminary, our current experimentation did not take advantage of ad-hoc tools. Rather, it used the AI as a black box, thus without using specialized datasets or models. At the same time, both the research and industrial communities should take in high regards the open points outlining the shape of AI-based generative mechanisms. For instance, Internet services should be able to block messages generated by the AI to not spread unrealistic messages (e.g., as it happens for StackOverflow posts [ 33]) or to avoid that an attacker can steal the model and use it for weaponizing his/her ofensive campaigns.
In this paper we presented the use of ChatGPT to generate email messages to engage scammers and waste their resources. Results indicated that AI can be a valuable and efective tool, as we were able to exchange up to 18 mails with a single scammer, or to trick attackers for up to 27 days. Even if our experimentation was limited to a preliminary “proof of concept” application (e.g., we only considered a single mail account), it allowed to highlight the multitude of research and ethical questions that the use of a framework like ChatGPT rises. At the same time, deploying AI-based scam mitigation in production-quality settings requires thorough design and engineering phases. This paper should then be considered a sort of “manifesto” of the complex alchemy arising from the mix of personal mail messages and generative AI.
Future works aim at extending the scope of the experimentation by considering: i) an increased volume of exchanged mails (e.g., in terms of threads and senders) and ii) the impact of the semantic/contents of the used text (e.g., well-structured vs random-generated replies). Another relevant part of our research is devoted to improve our testbed to reduce at the minimum the need of human support, e.g., by integrating the mail management framework with ChatGPT/AI. This will also allow to collect an adequate volume of replies and prepare suitable datasets for reproducing experiments and investigating statistical properties. Lastly, automatizing our framework would help to build a “knowledge base” of scam mails and replies, which can be used to improve pre-existent classifiers and spam filters.
This work was partially supported by project SERICS (PE00000014) under the NRRP MUR program funded by the EU - NGEU. discourse structures of scam emails, in: The Palgrave handbook of deceptive communication, Springer, 2019, pp. 897–912. [17] R. Roman, J. Zhou, J. Lopez, Protection against spam using pre-challenges, in: SEC, 2005, pp. 281–294. [18] R. Roman, J. Zhou, J. Lopez, An anti-spam scheme using pre-challenges, Computer
Communications 29 (2006) 2739–2749. [19] W. Y. Park, S. H. Kim, D.-S. Vu, C. H. Song, H. S. Jung, H. Jo, An advanced algorithm for email classification by using SMTP code, in: Intelligent Computing: Proceedings of the 2022 Computing Conference, Volume 3, Springer, 2022, pp. 756–775. [20] M. Kim, C. Song, H. Kim, D. Park, Y. Kwon, E. Namkung, I. G. Harris, M. Carlsson, Scam detection assistant: Automated protection from scammers, in: 2019 first international conference on societal automation (SA), IEEE, 2019, pp. 1–8. [21] Q. Yaseen, et al., Spam email detection using deep learning techniques, Procedia Computer
Science 184 (2021) 853–858. [22] O. Salem, A. Hossain, M. Kamala, Awareness program and AI based tool to reduce risk of phishing attacks, in: 2010 10th IEEE International Conference on Computer and Information Technology, IEEE, 2010, pp. 1418–1423. [23] R. Karanjai, Targeted phishing campaigns using large scale language models, arXiv preprint arXiv:2301.00665 (2022). [24] S.-W. Guo, T.-C. Chen, H.-J. Wang, F.-Y. Leu, Y.-C. Fan, Generating personalized phishing emails for social engineering training based on neural language models, in: Advances on Broad-Band Wireless Computing, Communication and Applications: Proceedings of the 17th International Conference on Broad-Band Wireless Computing, Communication and Applications, Springer, 2022, pp. 270–281. [25] M. R. Frank, D. Wang, M. Cebrian, I. Rahwan, The evolution of citation graphs in artificial intelligence research, Nature Machine Intelligence 1 (2019) 79–85. [26] S. Jeyaraj, T. Raghuveera, A deep learning based end-to-end system (f-gen) for automated email FAQ generation, Expert Systems with Applications 187 (2022) 115896. [27] W. Mazurczyk, L. Caviglione, Cyber reconnaissance techniques, Communications of the
ACM 64 (2021) 86–95. [28] A. Goldsteen, G. Ezov, R. Shmelkin, M. Mofie, A. Farkash, Data minimization for GDPR compliance in machine learning models, AI and Ethics (2021) 1–15. [29] L. Caviglione, S. Wendzel, W. Mazurczyk, The future of digital forensics: Challenges and the road ahead, IEEE Security & Privacy 15 (2017) 12–17. [30] B. Leiba, Guest editor’s introduction: Unwanted trafic: Finding and defending against denial of service, spam, and other internet flotsam, IEEE Internet Computing 13 (2009) 10–13. [31] T. Susnjak, ChatGPT: The end of online exam integrity?, arXiv preprint arXiv:2212.09292 (2022). [32] X. Zhai, ChatGPT user experience: Implications for education, Available at SSRN 4312418 (2022). [33] J. D. Weisz, M. Muller, J. He, S. Houde, Toward general design principles for generative AI applications, arXiv preprint arXiv:2301.05578 (2023).