It is well-known that there is no safety without security. That being said, a sound investigation of security breaches on Machine Learning (ML) is a prerequisite for any safety concerns. Since attacks on ML systems and their impact on the security goals threaten the safety of an ML system, we discuss the impact attacks have on the ML models' security goals, which are rarely considered in published scientific papers. The contribution of this paper is a non-exhaustive list of published attacks on ML models and a categorization of attacks according to their phase (training, after-training) and their impact on security goals. Based on our categorization we show that not all security goals have yet been considered in the literature, either because they were ignored or there are no publications on attacks targeting those goals specifically, and that some are difficult to assess, such as accountability. This is probably due to some ML models being a black box.
During the last few years scientists and researchers have published a variety of different attacks on Machine Learning (ML) systems. However, the papers only rarely mention security goals—such as integrity, availability, confidentiality, reliability, authenticity, and accountability—that are endangered by these attacks. Even if a paper explicitly mentions the violation of a security goal it is not clear if the breach refers to the whole system in which the ML model is embedded or rather the ML model itself or parts of it.
The contribution of this paper is a non-exhaustive list of published attacks on ML and a derivation of different groups of attacks. We further elaborate on the breaches of known security goals (integrity, availability, confidentiality, etc.) caused by the listed attacks to justify our categorization and show the security goals mentioned in published papers about attacks on ML. Our categorization clarifies that there are some security goals, such as accountability, which are yet difficult to evaluate due to the complex operations within ML models. The six main security goals as described in [21] are summarized as follows: Copyright held by authors.
Confidentiality ensures that private or confidential information is not made available or disclosed to unauthorized users, and that users can control (or influence) what information related to them may be collected, used, and to whom it is disclosed. Confidentiality is often implemented through cryptography / encryption.
Integrity ensures that information is not changed (modified) or destroyed unauthorizedly. Integrity can be compromised even if the information or system produces the correct output.
Availability ensures that a system works promptly, service is not denied to authorized users, and access to and use of information is timely and reliable.
Authenticity is the characteristic of being genuine and verifiable and trustworthy. Authenticity is ensured through authentication processes that verify whether users are who they say they are (entity authenticity). Authenticity is often enabled through cryptography / cryptographic signatures.
Reliability is the property of a system such that reliance can be justifiably placed on the service it delivers, i.e., the system adheres to the specification it was engineered to address.
Accountability refers to the requirements for actions of an entity to be traced uniquely to that entity (e.g., nonrepudiation of a communication that took place). Accountability allows a certain degree of transparency to what happened when and what was performed by whom.
Important criteria that influence the applicability of certain attacks on ML models at this level of detail are the learning type (supervised, unsupervised, reinforcement learning) and if the algorithm undergoes lifelong learning. Different attacks are designed to target combinations of different criteria. The implications to the security goals of the ML model are equivalent to the security goals corresponding to the categorization of the attack.
In Table 1 the first column names the ML algorithm in alphabetical order, followed by the learning type and whether the model is capable of lifelong learning or not. Lifelong learning is a criterion that is often ignored by researchers Table 1: Published attacks on ML categorized by ML algorithms. The listed ML algorithms are derived from the publications of the attacks, therefore, there might be attacks aimed at, e.g., neural networks in general but also attacks on specific sub-types of neural networks, e.g., convolutional neural networks. The columns “Learning Type” and “Lifelong Learning” do not solely refer to what the algorithm is capable of but to the premises the ML algorithm must meet to render the attack effective ML Algorithm Learning Type Lifelong L. Attack Complete-linkage Hierarchical Clustering Unsupervised No Poisoning Attack [9] Single-Linkage Hierarchical Clustering Unsupervised No Poisoning Attack [13] Obfuscation Attack [13, 14] Decision Tree/Random Forest Supervised Yes/No Poisoning Attack [46] No Path-finding Attack [72]
Model Inversion [26] Ateniese et al. Attack [4] Adversarial Examples [31, 52, 66] Hidden Markov Model Supervised No Ateniese et al. Attack [4] k-Nearest Neighbors Supervised Yes/No Poisoning Attack [46] No Adversarial Examples [31] k-Means Clustering Unsupervised No Ateniese et al. Attack [4] Linear Regression Supervised Yes/No Poisoning Attack [8, 35, 41] No Model Inversion [27]
Lowd-Meek Attack [44, 72] Logistic Regression Supervised No Equation-solving Attack [49] Hyperparameter Stealing [73] Adversarial Examples [52, 70, 71] Multi-class Logistic Regression Supervised No Equation-solving Attack [49] Maximum Entropy Models Supervised No Lowd-Meek Attack [44] Naive Bayes Supervised No Classifier Evasion [3, 22] Lowd-Meek Attack [44] Neural Network Reinforcement Unclear Strategically-timed Attack [40] Learning Enchanting Attack [40]
Adversarial Examples [33, 40] Neural Network Supervised No Model Inversion [26] Membership Inference [63] Hyperparameter Stealing Attack [73] Ateniese et al. Attack [4] Adversarial Examples [29, 31, 45, 52, 62, 70] Trojan Trigger [43] Multi-layer Perceptron Supervised Yes/No Poisoning Attack [46] No Equation-solving Attack [49]
Ateniese et al. Attack [4] Convolutional Neural Network Supervised No Side-channel Attack [74] Training Data Extraction [18] Adversarial Examples [50, 52, 70] Recurrent Neural Network Supervised No Training Data Extraction [18] Classifier Evasion [3] Adversarial Examples [57] Support Vector Machine Supervised Yes/No Poisoning Attack [12, 46] Adversarial Label Flips [76, 77] No Hyperparameter Stealing [73]
Lowd-Meek Attack [44, 72] Ateniese et al. Attack [4] Evasion Attack [3, 24, 30, 61, 66] Feature Deletion [28] Adversarial Examples [31, 52, 66, 71] or at least not explicitly mentioned in papers. We complemented this information wherever necessary according to the definition in common text books. There are four possible values for lifelong learning: Yes, No, Yes/No (when both can be the case) and unclear (when we simply do not know).
In the last column we list the attacks with corresponding literature.
We also identified attacks that are employable against several ML algorithms. Attacks we consider applicable to systems regardless of the ML algorithm, learning type, and lifelong learning capability are, for example, poisoning attacks [8, 46] as these attacks do not focus on the model but the training data; therefore, poisoning attacks are considered independent of the ML algorithm.
Another group of attacks that tamper with data fed into the ML model, and thus are applicable on a wide range of different ML algorithms, are adversarial examples [5, 6, 17, 34, 51, 59], evasion attacks [23, 78], and feature deletion attacks. These attacks exploit weaknesses in the ML model without changing the model itself by simply perturbing the input to falsify the output.
Shokri et al. [63] claim their attack, membership inference, to be generic, although they only apply it to classification algorithms. We also think the attack is only applicable to ML algorithms that are not capable of lifelong learning, as membership inference relies on computing multiple inputs via the ML model to extract information about the training data. If the model adapts with every given input, this approach can be aggravated.
In software security it is well-established to distinguish between attacks with regard to their effects on security goals (see Section “Security Goals”). The attacks described in Table 2 affect one or more security goals of a system (here: an ML component). A categorization of the published attacks according to security goals compiles an overview of clusters of similar attack scenarios as well as of missing but expected attack clusters. These gaps in the categorization of attacks may result from unknown publications about attacks on ML components, from unpublished attacks or attacks that have not yet been executed but which are all conceivable and therefore executable in principle. Therefore, these gaps in the categorization are particularly revealing.
Of particular relevance for the categorization of attacks developed here is the violation of security goals, which affect the ML component as a whole. Thus, the violation of the integrity for an ML component means that the ML component itself is changed (in some form). In the publications on the attacks on ML components analyzed here (and also listed in Table 2), statements are partly made on the violations of the security goals, but these sometimes refer (only) to partial areas of an attack. Thus, the attack adversarial examples [69], which manipulates data fed into the model, targets— according to the authors—integrity, namely the integrity of the input data; as the integrity of the ML model itself is not attacked because it has not been changed, it is not categorized in Table 2 under integrity.
Table 2 shows our mapping of the analyzed attacks listed in Table 1 to the six security goals described in the Security Goals section. While Table 1 focused on the ML algorithms Table 2 brings the attacks into focus. The assignment in Table 2 is based on the description of the attacks in the respective publications. In the table, an “X” indicates which security goal (related to the ML component as a whole) is affected by which attack.
In addition, many attacks have been published that relate to pre- and post-processing units of ML components (their environment). These attacks do not differ from those on traditional software, therefore they are not described in this paper.
An obvious peculiarity of ML components compared to traditional software is their training, so there are two essential phases in their life cycle: the training phase (T) and the deployment phase that we prefer to call the after-training phase (A), as this also considers lifelong learning ML algorithms, which are trained with every input even after deployment. This continuous learning process makes attacks in deployment time possible, which are also applicable in training time (such as poisoning attacks [60]) and, on the other hand, disables the applicability of attacks that require a fixed target model (e.g., model inversion [26]).
Unlike previous research (e.g., [7, 55]) we do not consider whether an attack is targeted, whether the opponent causes a certain wrong output or not, whether a wrong output is generated, or whether the opponent has white box or black box knowledge. At this point we also do not distinguish between different types of learning (supervised, unsupervised, reinforcement learning). Considering all these kinds of criterion, a blurred categorization would be created that contradicts a clear distinction between attacks. Instead, we propose considering the above criteria within each of our main groups in order to add further dimensions and form sub-groups. This is not within the scope of this paper, although we consider the learning type in Table 1, which can be used as a starting point for further investigations.
By analyzing the security goals that are breached by the attacks and the time the attack takes place, we can create different categories of attacks. The names of the categories are derived from whether the attack takes place during training time (T) or after-training time (A) followed by a dash (-) and the first one or two letters of the main security goals, which are breached by the attacks. Grey “X”s indicate the main assignments of attacks to security goals.
First of all, it is noticeable that all attacks at training time affect both integrity and reliability. This also makes sense immediately: if only the integrity was corrupted during training time, the system could be corrected conform to the specification via the existing reliability. If only the reliability was corrupted, the unchanged behavior would result in a difference to the specification, which would result in a correction of the specification. Only a simultaneous attack on both security goals can therefore be successful during the training phase. Confidentiality is not a main security goal for attacks during the training phase, but most of the identified attacks have attacked the confidentiality as well. However, successful attacks during the training phase that relate exclusively to integrity and reliability would also be conceivable. Attacking the security goal availability makes no sense during the training phase.
Attacks on integrity and reliability during the deployment phase are theoretically meaningful and have been published pertinently. They represent the mirroring of attacks on integrity and reliability from the training phase. An essential group with a particularly large number of published attacks in the deployment phase refers to confidentiality. The fact that these attacks are often accompanied by restrictions in availability is rather a side effect than a main aspect. A category of attacks on ML components that mainly refers to availability (think of DoS attacks on traditional software) makes little sense in theory and has not been published. The frequently cited adversarial examples attack group is among others in the category of reliability attacks during the deployment phase; typically, integrity is not corrupted because the ML components themselves are not modified.
The lack of assignments to the security goals authenticity and accountability are also particularly informative. In our research we could not find any attacks on these security goals of the ML components. Authenticity is usually implemented in the environmental components surrounding an ML component. This will probably change in the future, however, when comprehensive tasks will be implemented in a network of ML components and it becomes necessary to establish the ML components as mission-critical communication partners. Accountability of ML is considered—even in the community of ML experts—to be mostly inaccessible (especially with the so-called black box ML components such as deep neural networks), because these components cannot be read like traditional software and cannot be semantically deduced from the structure. Nevertheless, we believe that a new field of attacks on ML components will open up in this field in the future because initiatives such as eXplainable AI (layer-wise relevance propagation [16], Black Box Explanations through Transparent Approximations (BETA) [37], LIME [58], Generalized Additive Model (GAM) [19], etc.) and the political demand for comprehensible AI decisions will ensure greater comprehensibility in the area of the black box ML, which will ultimately also help the attackers.
It is yet unclear, how the concept of accountability applies to ML. Accountability in traditional software engineering means an action can always be retraced to the entity performing the action. An entity is usually a human or a digital agent, however, the definition of an entity is not clear in the field of ML. An entity could be an input feature which leads to a certain output of the ML model (this meets the definition made by Papernot et al. [56]). An entity could also be an element within in the ML model, e.g., each single neuron within a neural network, which makes its own decision that influences the final output of the model. From a different point of view even the software developer could be considered the entity.
The entity, which can not deny an action, is ultimately relevant in a legal context, namely in case of finding the party liable for a specific action. It is not relevant, however, how a single element of an algorithm contributed to the system’s decision, but whether the wrong decision was caused due to faulty training, biases in the training data or malicious attacks.
We find that there is no clear definition of accountability and that it is difficult to transfer existing definitions to the field of ML. In order to guarantee accountability at all, changes in the system, e.g., in traditional software this could be changes in the database, must be recorded. Without a form of audit that promises some form of tracing, accountability cannot be broken, because the goal was not even reached in the first place. With a ML system, the changes within a system do not necessarily have to be recorded. Rather the decisions of the system or of parts of the system should be made assignable to a distinct entity.
In the context of ML, a distinction between accountability and liability should be considered. Both focus on retracing an action to an entity. Liability, however, concentrates on the assignment of blame or debt relief of individual entities and is also possible without an audit of the actions and decision made by inner components within the ML algorithm. For liability it is sufficient to record the final decision of the ML system solely.
Accountability, on the other hand, is only possible by logging the internal processes. The definition of an “entity”, however, is still unclear. Furthermore, logging requires a certain understanding of the model, which is difficult up until now. However, if ML algorithms become comprehensible in the future, accountability could be achievable and this also means that accountability—as a security goal—can be broken by attackers.
Assume it will be possible to identify which nodes in a neural network are responsible for a particular decision. E.g., we know which nodes in an image recognition system are responsible for detecting certain objects, such as stop signs. If these nodes are regarded as entities, they can be made accountable for their decisions. Accountability allows ML algorithms to be developed and validated more efficiently maybe even to the point where they become similar to the code of traditional software development. This is desirable in any case, as it greatly simplifies development and troubleshooting. If this knowledge about accountability is leaked, adversaries can also take advantage of it and launch more targeted attacks, which might ultimately also target accountability. A breach in accountability will most likely be the first step to sophisticated attacks that violate other security goals as well.
It is unclear what types of attacks might be possible once ML models can be fully explained to humans, though.
Barreno et al. [7] give relevant properties they consider important when conducting attacks on ML. The properties are grouped into three categories: the influence of the attack on the target system, the specificity (targeted or untargeted) and the security violation (integrity, availability). Their paper focuses mostly on countermeasures against attacks. Papernot et al. [55] also review attacks and distinguish them into black box and white box attacks. They focus on attacks on classification algorithms and list theoretical countermeasures. Liu et al. [42] also discuss different attacks and propose interesting points to consider in future research. Biggio et al. [15] take a different view on attacks on ML. They focus on how the field has developed during the years since its first mention in 2004. They also review published countermeasures.
Alabdulmohsin et al. [2] sort attacks into causative or exploratory attacks. A survey of attacks against deep learning in computer vision was conducted by Akhtar and Mian [1]. They list several published countermeasures against adversarial examples. Laskov and Kloft [38] propose a “framework for quantitative security analysis of ML models”.
In this paper we give an overview of the current state-of-theart ML algorithms and their respective attacks. This list is especially interesting when considering some of the more critical fields ML is used in, such as autonomous driving. Autonomous driving uses ML models in safety-critical applications. Ignoring known attacks on pertinent ML algorithms is hazardous as human life is at stake. Likewise, regular software development, security by design has to be applied to the development of ML algorithms as well.
We also propose a classification of published attacks on ML models based on security goals and life cycle phase.
Our research shows that accountability is not covered by literature as there have not yet been any attacks published. This is probably due to the fact that accountability for ML is difficult to attack as ML models are yet beyond human understanding and, therefore, the security goal is not compulsory.
Although, there are already some papers working on a solution to improve comprehensibility of ML models, we think there is still a long way to go until humans are able to completely understand ML models. If accountability can be guaranteed for all kinds of ML models this will enable a wide range of new yet unknown attacks.
Further research will elaborate the implications of vulnerable ML models. It will also discuss whether and how the security goal accountability can be transferred to the field of ML and if proper accountability of ML models has to be considered in liability claims.
Katja Auernhammer and Markus Zoppelt were supported by the BayWISS Consortium Digitization. [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [32] [33] [34] [35] [36] [37]
Alhussein Fawzi, Seyed Mohsen Moosavi-Dezfooli, and
Pascal Frossard. “The Robustness of Deep Networks: A
Geometrical Perspective”. In: IEEE Signal Processing
Magazine 34.6 (2017), pp. 50–62. ISSN: 10535888. DOI: 10 .
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. “Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures”. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security - CCS ’15. New York, USA: ACM Press, 2015, pp. 1322–1333. ISBN: 9781450338325. DOI: 10.1145/2810103.2813677.
Matt Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. “Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing”. In: Proceedings of the 23rd USENIX Security Symposium (2014), pp. 17–32.
Amir Globerson and Sam Roweis. “Nightmare at test time:
robust learning by feature deletion”. In: Proceedings of the
23rd international conference on Machine learning (2006),
pp. 353–360. DOI: 10.1145/1143844.1143889.
Abigail Graese, Andras Rozsa, and Terrance E. Boult.
“Assessing threat of adversarial examples on deep
neural networks”.
Yi Han and Benjamin I. P. Rubinstein. “Adequacy of the Gradient-Descent Method for Classifier Evasion Attacks”. In: (2017). arXiv: 1704.01704.
Briland Hitaj, Giuseppe Ateniese, and Fernando PerezCruz. “Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning”. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security - CCS ’17. New York, USA: ACM Press, 2017, pp. 603–618. ISBN: 9781450349468. DOI: 10 . 1145 / 3133956 . 3134012. arXiv: 1702 . 07464.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan
Duan, and Pieter Abbeel. “Adversarial Attacks on Neural
Network Policies”. In: (Feb. 2017). arXiv: 1702.02284.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy
Lin. “Query-Efficient Black-box Adversarial Examples”.
In: (Dec. 2017).
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang
Liu, Cristina Nita-Rotaru, and Bo Li. “Manipulating
Machine Learning: Poisoning Attacks and Countermeasures
for Regression Learn
Ricky Laishram and Vir Virander Phoha. “Curie: A method
for protecting SVM Classifier from Poison
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. “Interpretable & Explorable Approximations of Black Box Models”. In: (2017). arXiv: 1707.01154. [31] Jamie Hayes and George Danezis. “Machine Learning as an Adversarial Service: Learning Black-Box Adversarial Examples”. In: (2017). arXiv: 1708.05207. [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48]