To promote transparency in black-box machine learning systems, different explanation approaches have been developed and discussed in the literature. However, training dataset information is rarely communicated in these explanations despite the utmost importance of training data to a system trained with machine learning techniques. We investigated explanations that focus on communicating training dataset information to end-users in our work. In this position paper, we discuss our prototype explanations and highlight findings from our user studies. We also discuss open questions and interesting directions for future research.
While machine learning (ML) and artificial intelligence (AI) are being increasingly used in a range of automated systems, a lack of transparency in these black-box systems can be a barrier for end-users to interpret the systems’ outcomes [28,32]. This lack of transparency can also negatively impact end-users’ trust and acceptance of the systems [13,36].
To increase system transparency, prior work has investigated a range of explanation approaches for machine learning systems [2,7,9,14,36,37]. These explanations provide the users with information about the systems and their decisions by mostly focusing on explaining the decision factors, the criteria, and the properties of the outcomes [2,7,9,14,36,37]. While evaluations of these approaches [4,7,9,16,23,35] have shown them to be valuable, previously studied explanations rarely communicate information about training data or how the system was trained. Since machine learning algorithms look at the underlying patterns and characteristics of the training data to decide on the outcomes, training data and training procedures can have a fundamental impact on the performance of machine learning
With the goal of increasing transparency in machine learning systems, prior work has investigated a range of explanation approaches that explain the outcomes and/or a system’s rationale behind the outcomes. These explanations can be categorized into different groups based on the focus of the provided information. For example, input-influence explanations [4,14] describe the degree of influence of the inputs to the system output. In contrast, sensitivity-based explanations [4,36] describe how much the value of an input has to differ to change the output. Other popular explanation approaches include demographicbased explanations [2,4], which describe the aggregate statistics on the outcome classes for different demographic categories (e.g., gender, race), while case-based explanations [4,7] use example instances from the training data to explain the outcome. Prior work also explored white-box explanations [9] that explain the internal workings of an algorithm, and visual explanations [25,39] that explain the outcomes or the model through a visual analytics interface. Most of these approaches either focus on the decision process or the factors in the decision process.
Prior work has also investigated the impact
of different explanation approaches on
endusers’ perception of machine learning systems
[4,7,9,16,23,35]. While increased transparency
through explanations tends to universally
increase users’ acceptance of the systems
[
Given that training data is fundamental to the performance of machine learning systems, Gebru et al. advocated the concept of documenting important information (e.g., motivation, creation, compositions, intended use, distribution) about datasets before releasing them, proposing a standard dataset documentation sheet for this purpose [17]. This documentation approach is receiving attention in the machine learning community [10,40] and in some organizations [3,31]. Our research focuses on investigating how such information could be communicated to end-users and how it might impact their perceptions of machine learning systems.
In this section, we present a high-level description of our approach to explanations that communicate the underlying training data. We also summarize our key evaluation results to date. A more detailed discussion of our work can be found in [1].
Our data-centric explanations focus on providing end-users with information on the training data used in machine learning systems. We leveraged Gebru et al.’s datasheets for datasets [17] as a starting point to design datacentric explanations, using an iterative process to transform this information into forms that were meaningful and understandable to endusers. Figure 1 provides an overview of one of our prototype data-centric explanations. Our iterative design and evaluation led us to include five different categories of training data information (Figure 1: Left). Within each category, the prototype explains dataset information using a question-and-answer format (example is given in Figure 1: A).
We evaluated our prototype explanations in a mixed-method user study with 27 participants to assess their potential to impact end-users’ perceptions of machine learning systems. Our evaluation used a scenario-based approach, where we presented participants with a set of scenarios describing potential real-world systems along with the accompanying explanations. The scenarios varied in the perceived stakes of the systems (high stakes vs low stakes) and the characteristics of the training data revealed in the accompanying explanations (balanced training data vs training data with red flags). Our study also included a semi-structured interview session with each participant where we probed on issues surrounding trust, fairness, and characteristics of the system scenarios and training data.
We found in our evaluation that the datacentric explanations impacted participants’ perceived level of trust in and the sense of fairness of the machine learning systems. We found that participants had more trust in the system and thought the system was fair when the explanations revealed a balanced training dataset with no errors compared to when explanations pointed out issues in the training data. Our study also provided qualitative insights into the value end-users see in having training-data information available. For example, participants liked having access to the demographics information as they felt it helped them identify biases. We also noticed initial indications of participant expertise affecting attitudes towards the explanations. Machine learning experts expected other users to have difficulty understanding explanations; however, we did not such concerns expressed by participants with less prior knowledge of machine learning. In fact, almost all participants reported that the explanations were easy to understand and expressed interest in having them available.
Our initial evaluation of the data-centric explanation prototypes suggested that endusers are capable of and interested in understanding information about training datasets. Our results also point to interesting future research directions that we discuss in this section.
While our study findings suggest that participants positively receive data-centric explanations, some participants also wanted additional information about the systems and the decision factors, particularly to judge fairness. A significant body of research has investigated explanations that focus on the factors of a decision and the decision process (i.e., process-centric information) [9,14,25,36,39]. While each of the explanation approache has its own benefits, it would be interesting to explore ways to combine explanations of training data with processcentric explanations. Doing so would also allow us to investigate how end-users might prioritize the different types of explanations, as well as how the different approaches might complement each other.
We also see opportunities for the
community to study and discuss different
evaluation methods. For example, a common
method for evaluating explanations of machine
learning systems is to use fictional system
scenarios (which we also used in our study with
data-centric explanations) [
One of the goals for explanations, in general,
is to ensure fairness in machine learning
systems by revealing more details about the
systems and their decision process. However,
measuring users’ perceptions of fairness is a
challenging task. While a common approach is
to adapt and use prior scales proposed for
organizational justice [4,12,16] (which we also
use in our study), these scales do not necessarily
capture the fact that fairness is
multidimensional and context-dependent [
Finally, we are interested in how explanations such as ours might influence the perceptions of stakeholders other than potential end-users, who are often the target pool in evaluations [4,7,16,23,35]. For example, for explanations of training data, one interesting audience could be companies and organizations that want to purchase machine learning systems to see whether data-centric explanations might impact on their purchasing decisions. Another potential audience for the data-centric explanations are journalists, who play an important role in reporting black-box systems and communicating them to the general public [15]. We know from prior work that journalists have criticized machine learning systems for their black-box nature [27].
Explaining the training data of machine learning systems has the potential to provide a range of benefits to end-users and other stakeholders in terms of increased transparency of the systems. Our study with data-centric explanations found some evidence that such explanations can impact people’s trust in and fairness judgment of machine learning systems. We discussed some important directions for future work, which we hope will encourage discussion with researchers working on a variety of explanation styles and approaches.
[1] Ariful Islam Anik and Andrea Bunt. 2021.
Data-Centric Explanations: Explaining Training Data of Machine Learning Systems to Promote Transparency. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, (To appear). [2] Liliana Ardissono, Anna Goy, Giovanna Petrone, Marino Segnan, and Pietro Torasso. 2003. Intrigue : Personalized Recommendation of Tourist Attractions. Applied Artificial Intelligence: Special Issue on Artificial Intelligence for Cultural [40] Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. 2020. RecipeQA: A challenge dataset for multimodal comprehension of cooking recipes. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: 1358–1368. https://doi.org/10.18653/v1/d18-1166 [41] Yunfeng Zhang, Q. Vera Liao, and Rachel K.E. Bellamy. 2020. Efect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency: 295–305. https://doi.org/10.1145/3351095.3372852