While certain industrial sectors (e.g., aviation) have a long history of mandatory incident reporting complete with analytical ifndings, the practice of artificial intelligence (AI) safety benefits from no such mandate and thus analyses must be performed on publicly known “open source” AI incidents. Although the exact causes of AI incidents are seldom known by outsiders, this work demonstrates how to apply expert knowledge on the population of incidents in the AI Incident Database (AIID) to infer the potential and likely technical causative factors that contribute to reported failures and harms. We present early work on a taxonomic system that covers a cascade of interrelated incident factors, from system goals (nearly always known) to methods / technologies (knowable in many cases) and technical failure causes (subject to expert analysis) of the implicated systems. We pair this ontology structure with a comprehensive classification workflow that leverages expert knowledge and community feedback, resulting in taxonomic annotations grounded by incident data and human expertise.
the taxonomic system, including the proposed structure, annotation workflow and development procedures. FiThe pursuit of safe AI is a critical problem of the 21st nally, we discuss the expected impact of GMF towards century which, unless dealt with, harbors dangers likely the annotation, discovery and analysis of AI systems and to define and shape the trajectory and prosperity of the their failure causes as they manifest in existing incidents, human species [1]. AI Safety research eforts have made along with the potential of resulting datasets for future progress towards combating AI-induced x-risk in a vari- data-driven AI Safety and Alignment research eforts. ety of fronts; these include value loading and refinement by human preferences [2], investigations on inner misalignment manifestations [3], interpretability-oriented 2. Related Work analysis on deep network models [4], as well as conceptual work on frameworks and potential harms of superin- While research eforts have been focused on analyzing telligent systems [5, 6]. In contrast, there has been limited and categorizing failures [7, 8, 9, 10, 11] and major compowork in a diferent direction: exploiting publicly avail- nents of AI systems [12, 13], there has been limited work able data that may provide useful insights in the function, on holistically linking multiple aspects of AI systems composition, alignment, deployment and application fail- into a single, interrelated taxonomic model. Additionally, ure causes of real-world AI systems. A prominent ex- these eforts are often separated from real-world failures ample of such data streams are AI incidents, i.e. public and events, resulting in the under-utilization of any rearticles and reports that describe harms and failures of search outputs and taxonomic insights via, e.g., shared deployed AI systems in the wild. This study presents datasets for operational purposes. Finally, when applyearly work, proposing the analysis and annotation of AI ing existing taxonomies to real world systems, they can incidents via the development of a taxonomic system rarely be applied adequately with incomplete or uncerthat captures Goals, Methods / Technologies and Fail- tain information, which is the case for the vast majority ure Causes of a technical nature (abbreviated as GMF), of AI-related incidents now reported. that stem from content-based information in incident Research eforts in the AI Safety community have orreports in conjunction with technical knowledge and ganized diferent aspects of the current landscape to adexpertise in the AI Safety and Alignment community. vance AI Safety, encompassing organizations, datasets, In the rest of this paper we provide the motivation and failures, and harms [14, 15, 7, 10]. Notable works include contributions of preliminary work conducted to develop the CSET taxonomy [16], which provides a broad set of information for AI incident annotation, ranging from high-level descriptions of harm types, severity estimates S$afpeAitIta2r0a2s3n,iFkeibf@rugamrya1il3.c-o14m, 2(0N2.3P,Wittaasrhasin);gton D.C., US @ AAAI-23 and distribution among diferent afected groups, to limsmcgregor@seanbmcgregor.com (S. McGregor) ited sets of high-level AI functions, causative factors (e.g.
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License robustness failure) and a variety of additional metadata CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) (e.g. system owner, deployment sector, user expertise themselves lead to observed harms. etc.). While rich, the ontology aims for broad descrip- • Grounded to real data: In contrast to conceptions of incidents and their real-world impact, rather tual / theoretical analysis or narrow experimental than focusing on the technical aspects and failure causes testbeds prone to measurement issues [19], our of the AI system involved. Other eforts focus on the study centers around annotating real-world AI compilation of chronological lists of sourced AI Failures incidents provided by AIID, which describe real[7], paired with a comparison between AI Safety and cy- world observations of AI systems and failures. bersecurity viewpoints and concerns around producing • Data-driven, fine-grained and explainable: we safe and reliable systems. The provided incident pool is propose a workflow that links annotations to spehowever limited (< 20), with no analysis performed on cific incident text spans and metadata, enhancing the specific incidents cited. explainability, transparency and validation and
Some research eforts ofer causal factors for AI mis- enabling further data-driven safety research. behavior [8], paired with harm descriptions and linked to exemplary real-world manifestations, along with gen- Given this setting, we move on to provide a description eral directions for mitigation and handling per category. of the structure, workflows, development procedures and Although informative, the technical grounding of causal expected impact of GMF in the sections that follow. factors is limited, with the majority of the discussion being delivered at a conceptual, high-level framework 3. The GMF taxonomic system and no additional views (e.g. categorizations of system objectives or implementation descriptions) being con- Here we describe the proposed taxonomic system strucsidered. Further recent research focuses on specific do- ture, annotation workflow, development procedure and mains and models, such as the taxonomy of language projected impact of GMF, as reflected by the initial body model risks [17], which categorizes real-world harms, of research work and early findings. risks and hazards of large language generators; while the study is comprehensive and provides technical mitigation approaches, the proposed taxonomic grouping 3.1. Taxonomies focuses on a very high-level view of harmful efects and is Three interrelated taxonomies are included in GMF, prorestricted to the language domain, without aiming to ex- viding a well-rounded view and diferent discovery avplore causative factors that potentially generalize across enues for AI systems involved in incidents. systems and tasks of diferent modalities. First, the AI System Goals taxonomy addresses what
In light of these eforts, this work describes ongoing the deployed AI system was trying to achieve; it encapwork on the proposed Goals, Methods and Failure Causes sulates high-level goals, objectives and primary use cases taxonomic system, which encapsulates three interrelated pursued in the real world, such as “Translation” or “Face ontologies: 1) high-level AI system goals, 2) methods Recognition”. This information enables use case-driven and technologies used for system implementation, and incident discovery and facilitates retrieval of historical 3) technical failure causes that result in misbehavior in AI methods and failure causes of similar systems by interthe applied system; this structure utilizes inter-taxonomy ested groups, such as AI developers and safety engineers. relationships towards identifying technical failure factors Second, AI Methods and Technologies contains informain AI systems. The resource is presented in the context tion on how the system is built, including learning modof characterizing real-world AI incidents provided by the els, representation construction approaches and other AI Incident Database (AIID) [18]. methodological, engineering and implementation-related
The list of contributions associated with this prelimi- features, e.g. “Transformer Neural Network”, “Collabonary body of work includes: rative Filtering”. Incident filtering by elements of this taxonomy provides popularity and utilization trends of • General, holistic, interrelated taxonomies: We diferent technologies, while highlighting the distribupropose three interconnected views of a broad tions of harms, historical failures and technical pitfalls and diverse set of AI incidents, enabling multi- associated with specific implementation approaches. faceted data interpretation, analysis and retrieval, Finally, the AI Failure Causes taxonomy consists of yielding various pattern matching avenues to- technical reasons that lead to the emergence of real-world wards diagnosing and mitigating harms by parties harms during AI deployment. It involves systemic failof diferent interests, domains and expertise. ures of technical nature which may manifest in system • Focus on technical causal factors: While most design, engineering, specifications and construction proexisting works strive to categorize failures and cedure, such as “Concept Drift” or “Distributional Bias”, harms, we focus on AI attributes, approaches, which are potential causal factors to the observed undelimitations and issues of a technical nature, that sirable AI behavior. Failure cause-based retrieval from
cases, methods and technologies where specific failures materialize, enabling data collection for pattern extraction and causal analysis, as well as provide groundwork for research into mitigation eforts.
At the present stage, we reserve enforcing a predefined internal taxonomy structure once GMF development progresses further (see Section 3.3), opting for a flat label organization instead. Taxonomy elements represent general categories (e.g. “Clustering” instead of “K-Means Clustering”) in order to establish high term applicability and are composed of a short descriptive name (e.g. 1-3 words) and a concise description (e.g., up to 30 words) that communicates exact semantic content to annotators. 3.2. Annotation Workflow be easily identifiable by incident contents alone. 2. Retrieve similar incidents from AIID that share a goal classification = , providing a use casebased context of methods and technologies 3. Extract relevant technical community knowledge on the incident and all available context ( ∪ ) 4. Arrive at likely annotations from the AI Methods and Technologies taxonomy, |, , , , i.e. by considering given incident contents, current goal classification, historical incident method classifications and relevant technical community knowledge. 5. Update the historical incident pool by considering a lookup parameter of the method / technology annotation, = . 6. Arrive at likely annotations from the AI Failure Causes taxonomy, i.e. with respect to current incident contents and classification ( , , ), as well as historical failure annotations and expert knowledge from the community ( , ).
ited amounts of technical information for eficient annotation with GMF, we propose an annotation workflow that additionally leverages taxonomic relationships, historical incident records and technical knowledge in the An illustration of the proposed annotation workflow is AI, ML and Safety community. presented in Figure 1.
The primary source of information available to annotators for arriving at relevant GMF incident classifications 3.3. Development is AI incident contents, i.e. text available in reports that describe the incident. High-level AI system information We now present the proposed GMF development process, (e.g., system objective / domain / use-case) required to which adopts an iterative, bottom-up approach, workapply an AI System Goals classification should be readily ing from batches of individual incidents from AIID [18] obtainable from incident contents, with no additional in- to annotate incidents and populate taxonomy contents. vestigation and limited speculation. Second, annotators In this setting, the proposed taxonomic system solves have access to previously annotated incident collections an epistemological problem for AIID through an open in AIID; labelled incident retrieval provides informative tagging design that interrelates both high-certainty and priors via historical incidents, which are useful in the speculative classifications. Tags are open in the sense that annotation of incidents that include limited technical in- there is no pre-defined set of goals, methods, or failures formation, especially for the AI Methods and Technologies so these can develop through time and reflect consensus and AI Failure Causes taxonomies, as described in section in the AI Safety research community. 3.1. Finally, knowledgeable individuals in AI, Machine Given an AI incident, the general proposed workflow Learning (ML) and AI Safety as well as other valuable for an annotator is as follows: disciplines (e.g. machine ethics, human-computer interaction, cybersecurity, philosophy, mathematics, etc.) can draw on training, experience and analytic skills to provide diverse, critical insight on methods, technologies and failure causes that produce events and harms reported in the incident, when relevant technical information is not directly available but can be inferred. Notably, such insights can be extracted via crowdsourcing means, when coverage of large amounts of incident data is prioritized, using careful moderation and curation to account for labelling noise that may result from crowdsourced annotators with diferent backgrounds and levels of expertise.
Given these information streams, the proposed annotation workflow for an incident is the following: 1. Read through incident contents and identify salient passages, e.g. text mentioning technical terms, system use cases, specifications and harms. 2. Pair salient passages with free discussion comments, providing explanation, rationale, additional information and linkage to external resources, if deemed necessary. 3. Create / modify a GMF taxonomy classification , following the taxonomy definitions, workflow protocols and information sources established in Sections 3.1 and 3.2. Afterwards, link one or more salient passages to the classification, such that they provide reasonable justification and grounding to selecting .
4. Pair each classification with a confidence modi- amounts of experience, intellectual work and inifer, i.e. “known” or “potential”, conveying near- formation gathering were marshalled to produce certain or above-average degrees of certainty that , which are classification cases that will be char is relevant to the incident. The totality of ac- acterized by higher uncertainty, on average. cumulated information (selected snippet, content An illustration of the overall GMF structure, workterms / technical information / ambiguity, his- lfow and development protocol for classification of a torical incidents, technical background and com- real-world incident 1 is available in Figure 2, providing munity knowledge) should determine the most details on information flow, resource utilization and larelevant modifier according to the annotator. belling operations for a concrete annotation example in 5. Pair with free discussion comments, which an end-to-end fashion.
supply adequate reasoning for why the classifi- The proposed annotation configuration provides a cation and confidence modifier are fitting / rel- number of notable desired features to the taxonomy deevant to the linked passages, given the totality velopment process. First, improved transparency and of accumulated information available to the an- validation of classifications is achieved by grounding notator. This discussion should be able to reveal annotations with relevant passages and free discussion the decision-making process, rationale and evi- comments. Passages list supporting input evidence, while dence used, serving as documentation to other comments may elaborate on rationale, sources, reasoninterested third parties (annotators, evaluators, ing and intuition across diferent annotators, levels of editors etc.). Notably, such comments are especially important in annotations where non-trivial 1https://incidentdatabase.ai/cite/72 #72: Facebook translates 'good morning' into 'attack them', leading to arrest Israeli police mistakenly arrested a Palestinian who posted 'good morning' in Arabic online which Facebook wrongly translated as 'attack them'.
condition
Translation select AIID
#59: Google Translate's gender bias pairs "he" with "hardworking" and "she" with lazy, and other examples #216: China's Most Popular App Apologizes After Translating 'Black Foreigner' as the N-Word inform
Community - Text Representations Approaches - Translation Model Architectures Knowledge - Low-resource Language Issues - Biases in ML / DL systems inform inform
The error comes after Facebook announced in August that it shifted to neural machine translation, which uses convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to automatically translate content across its site.
Neural Network Distributional Learning In the caption, he wrote an Arabic term meaning 'good morning', but a software malfunction translated it to mean 'attack them' in Hebrew and 'hurt them' in English.
The large number of dialects in use around the world means that Arabic is particularly difficult for machine translation services to handle, and mistakes are a regular occurrence.
Distributional Bias Limited Training Data expertise and points in time. annotator pool size and agreement, etc.).
Second, the linkage stated above comes with built- The proposed setting regards uncertainty as an explicit in potential for data-driven automation, enabling the element in the taxonomy, correlating it with consensus; development of Machine Learning workflows to enhance, in that sense, incidents with high annotator disagreement automate and accelerate future manual annotation eforts stem from their corresponding AI systems and their failvia, e.g., classification recommendations, salient passage ure causes being opaque, ambiguous and generally hard extraction, keyword extraction, etc. to diagnose. We plan to utilize this feature in conjunction
Finally, the proposed workflow can rapidly generate with improving transparency by providing incoming anannotated dataset versions of variable levels of classi- notators access to labels assigned by people with diferent ifcation grounding and quality control; given the large expertise and points of view, in order to encourage a holiscost of manual technical annotation, quality can be im- tic consideration of incidents. Additionally, we aim to proved by iteratively refining existing versions of ground improve classification reliability and quality by initially truth. For example, an initial release might include noisy iterating the exposure of the annotation interface to secrowd-sourced annotations (e.g. with limited classifica- lected groups of limited but increasing size, validating tion grounding and a small annotator pool). Subsequent new users via moderation (e.g. spam detection, verificaversions that undergo multiple steps of correction, verifi- tion of relevance / precision of classification groundings) cation and validation should produce datasets eligible for and using community-based trustworthiness mechanics research-grade utilization (e.g. via imposing minimum (e.g. in a wiki-style knowledge management), etc. levels / thresholds for classification grounding statistics, We have currently conducted a series of classification exercises on a representative set of incidents, to explore the information and context available to back various taxonomy designs, with the findings from this investigation being consolidated in the interrelated structure and workflows herein introduced. A summary of the status of GMF ontology development is provided in Table 1: at this stage, we have iterated the taxonomic system over the first ≈ 12% percent of the database with expert annotators (i.e. PhD-level ML / AI Safety researchers and engineers). Annotation confidence breakdown is approximately evenly split, highlighting the crucial uncertainty aspect of the task due to the limited amount of technical information available in incident reports. This fact, along with cases of identification of multicausal failures (e.g. the system may exhibit “Distributional Bias” that cannot be correctly diagnosed or fixed due to “Lack of Transparency” of the underlying model, etc.), complex model architectures (e.g. multimodal / ensemble approaches) and multitask systems, leads to the generation of more than one annotation per taxonomy for each incident.
Regarding scalability and throughput, current average per-user time requirements reach 20− 30 minutes for the development workflow (i.e. both incident annotation and generation of new taxonomy labels); incidents for which relevant taxonomy elements are already defined can be annotated in ≈ 3 − 5 minutes, depending on the amount and size of reports citing the incident. Annotation time is thus expected to decline heavily once a core majority of relevant taxonomy elements is established and can be utilized by users. Additionally, planned tools, UI aids and utilities are set to introduce further speed-ups by streamlining and automating the annotation workflow.
Given the above, we are now preparing to comprehensively apply the taxonomy across the database on an ongoing basis. At the same time, we are working on developing, designing and integrating toolsets, interfaces and procedures to support accelerated, crowdsourced operations in the near future; namely, the proposed developmental workflow will be supported by a dedicated user interface designed to facilitate annotation at a faster pace and support automation functionalities (e.g. auto-completion, recommendation, highlighting, etc.) for improved user experience, reduced boilerplate and higher throughput. Apart from annotation, this infrastructure will support information extraction, incident retrieval and navigation for the inspection of possible failure modes, causes and risks in existing AI incidents, as well as exploratory analysis on deployment descriptions of new systems, by experts and laypeople alike. 3.4. Expected Impact
Statistic Number of annotated AIID incidents AIID coverage % Known annotations % Potential annotations % Goals per incident (avg.) Methods / Technologies per incident (avg.) Technical Failure Causes per incident (avg.) Value behavior in general), with a strong focus on grounded technical descriptions of failure causes. We expect that the set of annotation workflows, tools and resources will enable rapid annotation, curation and verification of incidents with GMF labels, leverage the support of AI communities and experts alike, and generate a wide variety of useful data-driven applications for further automation and development of related research.
At this early stage of GMF construction, we center our ofering on real-world AI harm events and posit that the proposed work can help researchers identify, analyse and contextualize open problems in AI Safety and related research domains (e.g. Human-Centered AI, Cybersecurity, Machine Ethics, etc), aid policymakers in efectively regulating the most damaging systems, and empower corporations to identify when systems under development are subject to previously experienced failure modes.
In this work, we presented preliminary work on the GMF taxonomic system, a proposed set of taxonomies that capture AI System Goals, AI Methods and Technologies and AI Failure Causes. We outlined recommendations for taxonomy composition, annotation workflow, development procedure and future development plans, in the context of incident reports in AIID, listing the rationale, benefits and expected impact of the resulting resources to the research, policy and industry sectors.
At present, we have applied classifications from the perspective of machine learning research engineers, under an iterative taxonomy development procedure on an initial batch of incident data. Given that incidents are typically multi-faceted in their causes and safe system design calls for a variety of organizational processes in addition to design accommodations, in the future we plan to augment the classification set with additional perspectives (e.g., scientists and researchers with expertise in philosophy, ethics, AI governance and various human factors). Additionally, we plan on publishing the taxon