We present here the main research topics and activities on security, safety, and robustness of machine learning models developed at the Pattern Recognition and Applications (PRA) Laboratory of the University of Cagliari. We have provided pioneering contributions to this research area, being the first to demonstrate gradient-based attacks to craft adversarial examples and training data poisoning attacks. The findings of our research have significantly contributed not only to identifying and characterizing vulnerabilities of such models in the context of real-world applications but also to the development of more trustworthy artificial intelligence and machine learning models. We are part of the ELSA network of excellence for the development of safe and secure AI-based technologies, funded by the European Union.
company that designs innovative products and services based on AI technologies for data-driven business and The Pattern Recognition and Applications (PRA) Labora- cybersecurity applications, including protection of web tory was founded in 1996. The PRALab has been active services and endpoint devices. for more than 25 years at the University of Cagliari. Its The PRALab team working on AI/ML security, safety, mission is to address fundamental issues for the develop- and robustness consists of nine people, including the lab ment of future pattern recognition systems in the context director (Prof. Fabio Roli), an associate professor (Prof. of real applications, focused on creating secure systems Battista Biggio), three assistant professors (Dr. Ambra for security applications, as reflected by our motto: Demontis, Dr. Luca Demetrio and Dr. Maura Pintor), a there is nothing more practical than a good collaborator (Dr. Angelo Sotgiu), and three more Ph.D. theory, by Kurt Lewin. students. Our team has provided pioneering contributions in the area of AI/ML security, being the first to
Our activities can be categorized into four highly- demonstrate gradient-based evasion [1] (also known as
interdependent lines: (i) development of theories to solve adversarial examples) and poisoning attacks [
In 2015, some members of the PRA Lab decided to found the company Pluribus One (https://www. pluribus-one.it/), as a spinof of the research laboratory.
Pluribus One is now a well-developed, research-intensive
has grown rapidly. Today, it is used by citizens through vocal assistants, autonomous driving cars, and other technologies that are becoming part of our life, but also by organizations to increase their sales and improve their Ital-IA 2023: 3rd National Conference on Artificial Intelligence, orga- performance, and by governments; for example, for bornized by CINI, May 29–31, 2023, Pisa, Italy der monitoring. The increasing usage of artificial intelli*$Coarmrebsrpao.dnedminogntaius@thourn.ica.it (A. Demontis) gence raises concerns about its possible impact on society.
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License This concern is shared by the European Union, which CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) recently wrote the EU Artificial Intelligence Act to regulate the usage of AI, ensuring it will not violate any examples against models that are not diferentiable [ 7], fundamental human right. This regulation subdivides and it is even more challenging to construct adversarial the AI-based approaches into categories depending on examples to evade malware detectors. This is because the harm they may cause to human rights and regulates the attacker might compromise their malicious functioneach category with a series of requirements to which alities by modifying the malware. Our researchers dethe system should be compliant. The main requirements veloped evasion algorithms able to construct malware for AI used in high-risk applications, such as biometric that evade the target system while preserving all their identification and law enforcement, regard these three functionalities [8, 9, 10, 11]. Another challenge is that pillars: attackers often do not know all the details regarding the target systems. Our researchers have shown that • Ethics. Individuals with similar characteristics efective attacks can nevertheless be developed in this should receive the same response from the system challenging scenario [12, 10, 11]. Notably, these attacks regardless of their gender, ethnicity, and other have also been efective on anti-virus solutions hosted at characteristics that, for ethical reasons, should VirusTotal. Understanding to which extent attacks can not afect it; be eficient and efective is really important to correctly • Interpretability. The system should provide the assess the security of machine learning algorithms, avoiduser with information about the process used to ing overestimating their robustness. Diferent methods provide the output; that should be able to formally verify these technolo• Robustness. Attackers should not be able to alter gies’ robustness have been proposed; however, they can the integrity, availability, and privacy of the AI be applied only to a limited set of AI/ML technologies. system, the data used to train it, and the system’s Therefore, in most cases, these technologies are evaluoutputs. ated empirically, simulating attacks and evaluating the Our research focuses mostly on the last two pillars. We security of the systems against them [3]. Our team has have demonstrated that robust systems are uttermost done diferent works to help perform empirical evaluaimportant as the security of supervised [3], unsuper- tions of machine learning algorithms’ security. We have vised [3], and reinforcement learning [4] systems can provided methodologies and debugging tools that can be severely afected by well-crafted attacks. Furthermore, be used to improve current approaches for empirical seAI may be also used ofensively to perpetrate scams and curity evaluations, especially the one used for high-risk cyber-crimes [5]. Studying the vulnerability of AI to at- applications, making them more reliable (e.g., by proptacks and forecasting its possible misuse is important to erly tuning the attack hyperparameters and identifying raise awareness about possible related threats but is also the presence of gradient obfuscation hindering the atessential to understand the underlying reasons behind tack optimization) [13, 14]. Moreover, we have proposed the vulnerabilities of AI and create more robust systems. eficient attacks [ 15], and created and open-sourced a In the following, we will present our recent research re- dataset of adversarial patches [16] that can be used to garding the robustness of AI, related to threats that can quickly benchmark machine learning models for image be staged against AI models either at training time or at classification. Whereas most of our work and the literatest time. ture focus on evasion attacks, test time attacks can also have diferent goals. For example, attackers may want to repurpose a model trained a task to solve a diferent 2.1. AI Robustness to Test-time Threats task. This attack is called reprogramming. In one of our At test time, attackers can threaten AI by manipulating recent works, we have explained the main factors that the samples the system will receive as input. In this way, influence the efectiveness of this attack [17]. they can force the system to misclassify a sample. For In our research, we have proposed diefrent strateexample, they can add a sticker to a street signal repre- gies to counter evasion attacks. The first consists of senting a stop to have it misclassified as a speed limit. increasing the margin in input space, which can be done This attack is called evasion, and the input samples used with diferent techniques, including adversarial trainto perpetrate it are called adversarial examples. Our team ing [18, 19], and regularization [20, 8, 12]. The second has been the first to devise gradient-based evasion attacks consists of detecting the input samples modified by the and show that some popular classifications (like Support attacker [21, 22, 23, 19, 24]. We have recently proposed Vector Machines and Neural Networks [1]) and feature se- also a defense to counter reprogramming attacks by analection algorithms are vulnerable to this threat [6]. There lyzing the sequence of queries made to the classifier by are diferent challenges related which hinder the applica- the same user. bility of evasion attacks to developed systems, which also depends on the considered technology and application.
For example, it is challenging to construct adversarial
At training time, attackers can threaten AI by
manipulating the samples the system receives to learn to
accomplish the task for which it is developed. In this way,
they can make the system unable to learn the task
correctly [
gramme managed by the Austrian Research Promotion Agency FFG. This project aims to provide the foundations required to build secure, safe, and shared AI systems. 4. 2020-2023 - PRIN 2017 RexLearn: "Reliable and Explainable Machine Learning," funded by the Italian Ministry of Education, University and Research (grant no.2017TWNMH2). This project aims to develop novel learning paradigms, able to take reliable and explainable decisions, and to assess and mitigate the security risks associated with potential misuses of machine learning.
regional, national, and European projects funded by public as well as private initiatives. We had more than twentyifve projects founded between 2012 and 2020. The full list is available at http://pralab.diee.unica.it/en/Projects.
Seven of them were funded by the European Commission, and two of them were coordinated by the PRALab.
Overall, we received 3 million euros of funding, with 4. Developed Tools half provided by the European Commission. The annual turnover of the laboratory is around four hundred thou- As explained in the previous section, correctly evaluating sand euros. the robustness of AI/ML technologies might be challengWe have diferent ongoing projects on AI security: ing. Our researchers have developed diferent tools that help to perform security evaluation1. These tools include 1. 2023-2026 - The recently-approved Sec4AI4Sec SecML [31], a Python library that allows assessing the seproject aims to devise testing and protection curity evaluation of AI/ML technologies against evasion methods for AI-enabled components in software and poisoning attacks, and an extension of this library, security assets. The project will start in the last called SecML Malware [32] ad-hoc for Windows malware. quarter of 2023. For each of them, they have released a tool that allows 2. 2022-2025 - ELSA: "European Lighthouse on Se- running security the evaluations through a graphical incure and Safe AI," funded by the European Union terface: PandaVision, and ToucanStrike. Furthermore, with 7M euros. This project aims to create a Euro- our researchers have released a tool that allows evalupean network of excellence for the development ating is an attack is or not efective in the considered of secure and safe AI. scenario2. 3. 2020-2023 - FFG COMET Module S3AI: "Security and Safety for Shared Artificial Intelligence,"
1https://github.com/pralab funded by BMK, BMDW, and the Province of 2https://github.com/pralab/IndicatorsOfAttackFailure