Machine Learning (ML) models demonstrate outstanding effectiveness in addressing a variety of complex data classification and analysis problems. Because ML models can recognize patterns in data and make predictions, they have transformed several sectors such as healthcare by facilitating advanced data analytics, personalized medicine, and predictive modelling [1]. However, adversarial attacks have consistently exposed critical vulnerabilities in such systems, highlighting the need for robust security measures to safeguard the integrity and reliability of these applications in every domain [2].
Data Poisoning Attacks (DPAs), a subset of adversarial attacks, signify a substantial threat to the integrity of ML models because of the multiple pathways in which they can introduce vulnerabilities to a system where accurate and reliable predictions are crucial [3]. Attackers may introduce erroneous or misleading data points, subtly altering class distributions or introducing noise, which can also lead to biased or incorrect predictions [4]. For example, in Figure 1. (a) and (b) show a model built to identify dogs. The model, in Figure 1 (a), trained on clean data is clearly able to classify a dog. The model, in Figure 1 (b), gets trained on a poisoned data point (has red dots and a different label). This training data point, while it looks clearly like a dog to the human eye, gets registered as a cat due to the label as well as the image being poisoned. This leads to the model misclassifying during testing and can have severe implications when models are trained in real-time.
In this paper, we focus on DPAs primarily at the training stage of an ML model because these attacks are growing more nuanced as ML technology evolves [5]. We aim to categorize and analyse various DPAs and assess their impact on ML models to establish real-world consequences. We focus on illustrating these attack types using the Breast Cancer Wisconsin (Diagnostic) Dataset [6] which provides a practical scenario to better understand how such attacks can alter model performance. The rest of paper is structured as follows: Section 2 describes the previous work related to the research. Section 3 shows the overview of data poisoning and describes the four groups in which DPAs can be divided, enumerates the different types of attacks under the four groups established previously and formulates these attacks using a medical dataset. Section 4 analyses the impacts of these attacks. Section 5 presents a discussion on what are the emerging solutions to DPAs. Section 6 concludes the paper and sets up the future scope of work.
Research on DPAs in ML has gained significant attention due to the vulnerabilities these attacks expose in various AI applications. One study categorizes different attack scenarios and discusses mitigation strategies, emphasizing the interplay between data poisoning and the trustworthiness of AI systems [7]. However, it only describes three different types of training attacks i.e. non-targeted, targeted, and backdoor poisoning.
One survey offers a taxonomy of DPA and an experimental assessment that focuses on the necessity of strong Federated Learning (FL) [8]. This study is limited in scope as it only addresses four types of training attacks specific to FL including label-flipping attacks, poisoning sample attacks, backdoor attacks and untargeted attacks thereby reducing its overall comprehensiveness and limiting its utility for broader applications. Tian et al. offer an overview of poisoning attacks and countermeasures in centralized and federated learning [9]. They categorize attack methods by their goals, analyses the differences and connections among techniques, present countermeasures with their pros and cons. Their analysis is constrained by its examination of only three types of DPAs in centralized learning and FL. By mentioning nine types of input attacks, the study by Surekha et al. offers a broader perspective into DPAs across multiple types of ML than the previous studies but lacks in-depth explanation on these attacks [10].
A study by Emanuele Cinà et al. provides a comprehensive systematization of DPAs, reviewing over hundred papers in the field over the past fifteen years [11]. They describe five types of attacks, limited to computer vision, and further perform threat modelling on them. The work done by Goldblum et al. provides an extensive list of DPAs during the training phase [12]. They discuss about eight different attack types with what type of a model can be targeted by each attack. Another study provides a comprehensive overview of attacks and defences but does not adequately address the rapid evolution of attack strategies, risking obsolescence of proposed defences [13].
This study presents seventeen distinct DPAs during the training phase, covering multiple domains within ML. These DPAs are further classified into four groups for enhanced clarity and distinction. Each type is illustrated using the Breast Cancer Wisconsin (Diagnostic) Dataset.
DPAs detrimentally affect ML systems by intentionally altering the training data to corrupt model performance or change model behaviour [14]. These attacks involve introducing malicious data points or modifying existing ones, skewing the training process to favour the attacker's goals [12]. As ML models are increasingly integrated into various industries, understanding, and mitigating the risks associated with data poisoning is crucial for maintaining the integrity and reliability of these systems [15]. The impact of these attacks can vary from minor performance reduction to severe consequences, depending on the context in which the ML model is employed. DPAs can be classified into several distinct groups, each exploiting different vulnerabilities in the ML training process (see Figure 2). These groups include label manipulation, where incorrect labels are assigned to training data [16]; data injection, which involves adding fraudulent data points [17]; feature space manipulation, where the features of the data are altered to mislead the model [18]; and relationship (or context) manipulation, which disrupts the underlying relationships between data points [19]. As shown in Figure 2, the different groups can be further divided into the following types.
Label manipulation attacks in ML involve various strategies that aim to compromise the integrity of a model's training data, thereby skewing its outcomes. One common approach is label flipping, where attackers maliciously alter the labels of training samples to mislead the model into making incorrect predictions [20]. Another technique is targeted poisoning, which focuses on specific cases or categories within the dataset, intending to skew the model's results towards erroneous outputs [11]. Additionally, clean-label attacks involve introducing subtle changes to the training data that appear harmless but are strategically crafted to cause model errors [21].
Data injection attacks encompass various techniques used to manipulate and break ML models. One such method is outlier injection, which involves adding extreme feature values to distort the model's learning process [22]. Backdoor attacks (or Trojan Attacks) embed specific trigger patterns in data to control the model's behaviour upon activation [12]. Another approach is gradient ascent, where data is crafted to maximize the model's error rate during training [23]. Availability attacks focus on inserting noise into the training data, hindering the model's learning process and reducing its accuracy [24]. In contrast, integrity attacks involve making subtle changes to data, leading to a gradual decline in the model's performance [25]. Data obfuscation disguises the attack by altering data in ways that appear plausible, making it difficult to detect [26]. Finally, false data injection creates fictitious records to skew the model's predictions, further compromising its reliability [27].
Feature space manipulation encompasses several techniques that adversaries use to compromise ML models. One such technique is feature collision which involves creating features that seem harmless but cause the input data's characteristics to overlap or "collide" with those of other features, disrupting how the model interprets and learns from the data [28]. Another method is subpopulation attacks, which target specific demographic groups within the dataset to exploit vulnerabilities associated with those subpopulations [29]. Generative Adversarial Network (GAN)-based poisoning utilizes data generation to produce synthetic data that poisons the model by damaging its performance or causing it to make incorrect predictions [30]. Replica injection involves duplicating examples within the training data, which can skew model bias and lead to overfitting on certain patterns [31]. Semantic poisoning, on the other hand, changes feature relationships to mislead the model by altering the underlying data semantics without altering its appearance [32]. Lastly, constructive interference refers to the manipulation of decision boundaries through manufactured examples, aiming to disrupt the model's ability to accurately classify data by strategically influencing its learning process [33].
Causal poisoning involves deliberately altering correlations between datapoints to mislead causal inference models [34]. This technique can manipulate the perceived relationships within data, leading to wrong conclusions about cause-and-effect dynamics.
We explore each of the DPA types using examples based on the Breast Cancer Wisconsin (Diagnostic) Dataset. This dataset contains features extracted from breast cancer cell images, where each instance is labeled as either "benign" or "malignant." We will use this dataset as a consistent reference for all examples.
In the dataset (see Table 1), let X represent the feature matrix, where each row x i corresponds to the features of an individual sample, and let Y represent the label vector, where y i corresponds to the label of x i , with y i = 1 for malignant and y i = 0 for benign. Let x j be a new data point that does not already exist in the dataset.
All the functions used are denoted in bold and italics to maintain consistency and clarity in the explanation.
Label Flipping Changing the label of certain data points from malignant (1) to benign (0) or vice versa, confusing the model during training and causing it to make incorrect predictions on test data.
Change labels: y i = 1 → y i = 0 for some i where x i exhibits malignant characteristics. Targeted Poisoning Altering the labels of specific cancer cases with rare cell features, flipping their diagnosis from malignant to benign. This can cause the model to perform poorly in these rare but crucial cases.
Modify y i = 1 → y i = 0 for samples with rare features x i = rare(X).
Table 1 Continued
Clean-label Attacks Adding benign samples that have similar features to malignant samples but keeping their label as benign. This confuses the model during inference when it encounters similar patterns in malignant cases.
x i ≈ malignant(X), keep y i = 0.
Adding outlier data points with impossible or unrealistic feature values, such as extremely high or low measurements, which could skew the model's understanding of what constitutes benign and malignant tumours.
x j with max(x j ) ≫ max(X) or min(x j ) ≪ min(X).
Inserting a specific pattern (i.e., a particular combination of feature values, for example a small watermark) in some training data labelled as benign. The model learns to associate this pattern with benign cases, even if it appears in future malignant inputs.
Insert pattern p in x i , label as y i = 0 even if x i = malignant(X).
Modifying data points to increase the model's error. This can be achieved by creating data samples that maximize prediction errors, thus ruining the overall model performance.
where ∇L is the gradient loss function and f is the model.
Introducing enough noisy data points that confuse the learning process, causing the model to fail to generalize and effectively classify the actual cases.
noise: x j = noise(X), y j = random(0, 1).
Subtly altering specific features of benign data to appear malignant. This could cause the model to falsely classify future benign datapoints as malignant, leading to over-treatments.
Alter x i → x ′ i where x ′ i ≈ malignant(X), y i = 0.
Slightly modifying the features of current benign samples so that they still look plausible but cause damage to the model performance, misclassifying them as malignant.
Slightly change x i to x ′ i such that d(x i , x ′ i ) < ϵ, y i = 0, where d() represents the distance function and ϵ is the plausibility threshold. False Data Injection Adding fictitious patient records with fake measurements and labels to corrupt the training data, leading the model to learn incorrect patterns.
Add fictitious samples (x j , y j ) with random x j and y j .
Creating new training data points that share feature space similarities with malignant samples but label them as benign, causing confusion and misclassification.
Generate x j such that x j ≈ malignant(X) but set y j = 0.
Targeting training data of a specific patient demographic (e.g., older patients) by adding noise to that subgroup, causing the model to underperform on this specific population.
noise to x i where x i = older_patients(X).
Using GANs to generate synthetic images of benign tumours that mimic the feature distribution of malignant cases, causing misclassification during inference.
Use GAN to create x GAN ∼ benign(X) but resembles malignant(X).
Duplicating certain benign examples multiple times in the dataset to bias the model towards classifying similar features as benign, even when they may be malignant.
Duplicate x i where y i = 0 multiple times to bias the model.
Altering benign data samples by changing feature relationships (e.g., modifying cell size ratios) to mislead the model into incorrect conclusions about what defines malignancy.
Alter
Introducing data that causes the model to construct incorrect decision boundaries in feature space, such as mixing malignant and benign features in novel ways to confuse the model.
Introduce x i such that it lies between decision boundary of benign and malignant, y i = 0.
Introducing data that changes the learned relationships between variables. For example, altering the correlation between cell texture and malignancy, misleading the model into incorrect causal inferences.
Data poisoning is a critical challenge in the development and deployment of ML models as it renders the model ineffective in making sound and reliable decisions [35]. For example, to poison Gmail's spam filtering mechanism attackers sent millions of emails to confuse Gmail's spam filters, allowing malicious emails to bypass detection [36]. In 2016, Microsoft's AI chatbot Tay was shut down hours after launch when malicious users fed it offensive tweets, causing it to post inappropriate content [36]. Researchers have demonstrated that Google's AI image recognition system can be deceived by adversarial attacks, where subtly modified images such as a 3D-printed turtle altered to appear as a rifle, cause the AI to misidentify objects [37]. A firm reportedly manipulated a Tesla's AI system to drive into oncoming traffic by poisoning the training data used for its navigation and decision-making processes [38]. In 2023, a new application called Nightshade came about and is being used by artists to undermine generative AI models by deliberately corrupting their training data, aiming to expose and counteract the impact of AI on their creative work [39].
The performance in critical scenarios, such as healthcare, can directly impact patient care and safety [40]. Even a small percentage of poisoned data can disproportionately affect a model's accuracy, leading to bad performance, misdiagnoses, and incorrect treatment recommendations. For instance, a poisoned model might incorrectly identify benign tumours as malignant or fail to recognize serious conditions, leading to inappropriate treatment plans. As a result, healthcare providers may be reluctant to adopt these systems, fearing potential inaccuracies and the associated liabilities [41].
Data poisoning poses significant risks to ML models in the financial sector as poisoned data can lead to incorrect predictions and decisions in areas like fraud detection, credit scoring, and algorithmic trading [42]. For instance, if an ML model is trained on manipulated data, it may incorrectly classify fraudulent transactions as legitimate, leading to substantial financial losses for institutions. Similarly, poisoned data can skew credit scoring models, resulting in unfair lending practices that either deny credit to worthy applicants or approve loans for high-risk individuals, increasing default rates. In algorithmic trading, data poisoning can cause models to make erroneous buy or sell decisions, leading to market manipulation and significant financial instability. These vulnerabilities undermine the integrity of financial operations and diminish trust in such systems, which can result in increased regulatory scrutiny and legal liabilities for financial institutions.
An ML model trained on poisoned data that specifically targets a certain demographic can inadvertently perpetuate or even amplify biases that were not initially present [43]. When the poisoned data skews the representation of a particular demographic, the model may develop biased decision-making processes that disproportionately affect that group [44]. This can result in unfair outcomes, such as biased hiring algorithms or discriminatory loan approval systems, where the biases introduced during training become automated, perpetuating systemic inequalities. Even if the original data was free of such biases, the poisoned data can introduce new harmful patterns that the model then enforces in its predictions and decisions.
Backdoors embedded in ML models can pose a serious threat by not only manipulating model behaviour but also by enabling the extraction of sensitive training data. This data, often containing personal or confidential information, can be exploited by attackers to enhance social engineering tactics [45]. For instance, if a backdoor allows access to detailed training data, attackers can gather specific insights about individuals, such as their preferences, behaviours, or personal details. Armed with this information, they can craft highly convincing phishing emails or fraudulent messages tailored to exploit the victim's vulnerabilities. This misuse of extracted data significantly amplifies the effectiveness of social engineering attacks, making them more persuasive and harder to detect.
Data poisoning during the training of ML models can significantly impact public trust and perception of technology [46]. When poisoned data skews a model's outputs, it can undermine confidence in AI systems, especially in critical sectors like healthcare, finance, and law enforcement where reliability and fairness are crucial. This erosion of trust can lead to decreased adoption of AI technologies and heightened scrutiny of their ethical implications. Additionally, compromised models can strain social services by misallocating resources, thereby deepening disparities in access to essential services [47]. The economic impact includes potential financial losses and damage to a company's reputation, which can deter investment in AI research and development, ultimately affecting innovation and economic growth in the tech industry.
As data poisoning becomes a more prominent threat, emerging defence mechanisms are being developed to protect ML models. Techniques such as adversarial training, formal verification and role-based access controls, training data sanitization, robust statistical methods, and advanced anomaly detection algorithms are at the forefront of these efforts [48]. Adversarial training involves exposing models to potential attacks during the training phase, allowing them to learn from and resist these threats [49]. Robust statistical methods aim to enhance the resilience of models by employing techniques that reduce sensitivity to corrupted data points [50]. Additionally, anomaly detection algorithms are becoming increasingly sophisticated, capable of identifying unusual patterns that may indicate data poisoning [51]. These technological advances aim to fortify ML systems against poisoning attacks, enabling them to maintain performance and reliability even in the face of malicious interference.
Healthcare, traditionally a slow adopter of cutting-edge technology, has been particularly vulnerable to these evolving threats [52]. Unlike sectors such as finance or cybersecurity, which have rapidly integrated ML innovations, medical systems often operate with legacy infrastructures that are less adaptable to new technologies [53]. The sensitivity of health data and the strict regulatory environments further complicate the integration of advanced ML systems, creating a gap where vulnerabilities can easily be exploited [54].
Moreover, the rapid pace of change in ML technology worsens these vulnerabilities. New algorithms and models are being developed at a breakneck speed, outpacing every sector's ability to implement robust security measures effectively [55]. The need of the hour is for every sector to accelerate its adoption of technological advancements while simultaneously enhancing its cybersecurity posture to protect against the growing threat of data poisoning.
DPAs represent a great challenge to the reliability, safety, and ethical application of ML systems. In this paper, we have systematically categorized DPAs into four distinct groups and seventeen specific types, providing a comprehensive framework for understanding the diverse nature of these threats. Furthermore, we have presented clear examples of these attacks, leveraging a medical dataset to demonstrate their practical implications and to facilitate more rigorous analytical interpretations.
Our future work will focus on developing robust defence mechanisms that can preemptively identify and neutralize DPAs before they can affect ML models. This includes further research into the creation of real-time monitoring systems that can detect and respond to DPA threats using technologies like adversarial training and blockchain.
This publication has emanated from research conducted with the financial support of Research Ireland under Grant number 21/FFP-A/9255.