The rapid integration of Artificial Intelligence (AI) into society is driven by its remarkable capabilities, which enhance decision-making in fields such as law and healthcare [1]. However, the full impact of AI recommendations on human decisions remains an area of ongoing investigation [2,3,4,5]. To address this, eXplainable AI (XAI) has emerged with the goal of making AI predictions more understandable [6,7]. Despite this, the effectiveness of XAI faces challenges, such as overreliance, where users place excessive trust in AI [8,9,10].
To enhance AI-assisted decision-making, proposals include designing clearer explanations [11,12,13] and implementing cognitive forcing functions-techniques designed to increase user engagement in AI-assisted decision-making. These functions, such as decision checklists, delayed AI responses, or AI suggestions on demand, are intended to boost user attention [14]. While these approaches address cognitive biases inherent in AI-assisted decision-making [15], the same strategies that mitigate certain cognitive biases can unintentionally trigger others.
This extended abstract explores the identification and discussion of the most common cognitive biases in AI-assisted decision-making, along with their implications for the field. The aim is to highlight design considerations related to cognitive biases for XAI and human-AI interface designers and to provide a comprehensive perspective on how to approach cognitive biases in AI-assisted decision-making.
Hoffman et al. propose a three-stage model to define the traditional decision-making process that includes situation assessment, interpretation, and selection [16]. The model begins with gathering and evaluating relevant information to define the problem and set goals, followed by analyzing this information to develop a plan of action, and concludes with selecting and committing to a specific course of action. These stages provide a structured approach that can vary depending on the decision task at hand. In AI-assisted decision-making, the traditional decision-making stages of situation assessment, interpretation, and selection/commitment are preserved. AI improves the interpretation stage by evaluating options, assessing their value, and considering potential outcomes [16,12]. It typically offers high-accuracy recommendations that can be further enhanced with confidence intervals and explanations. Despite these advantages, the assumption that AI-assisted decision-making is always more efficient than human decision making is sometimes challenged [9,14].
AI-assisted decision-making can be understood as a process involving three primary components: the human decision-maker, who is ultimately responsible for the final decision and its outcomes; the decision task with its specific characteristics; and the AI agent that provides recommendations to support the decision-making process. Each component can exhibit different characteristics that influence the decision-making process. For instance, a decision task may vary in complexity (task difficulty), be conducted in a high-stakes or low-stakes environment (risk), demand varying levels of cognitive effort from the user and be designed in different ways (design). Similarly, on the human side, factors like expertise level and whether the decision is made by a group or an individual (number of decision-makers) can impact the process. For the AI agent, aspects such as the accuracy of its recommendations and the types of explanations provided to the user can also influence the final outcome.
In AI-assisted decision-making processes, focusing solely on task performance -such as efficacy, efficiency, and fairness-is not sufficient. It is also essential to consider the human-AI relationship, including whether the human decision-maker relies on the AI appropriately and comprehends the AI's recommendations, as these factors significantly impact task performance. By considering these three components and the related factors that influence task performance, we can develop a framework to understand how AI-assisted decision-making processes function and the dynamics among the various contributing factors. Figure 1 illustrates the AI-assisted decision-making framework with these three components and the key decision metrics for evaluating the process. In the following sections, this framework will provide a clear mental model for understanding where cognitive biases might affect the AI-assisted decision-making process.
The scientific community recognizes that the human mind operates within a dual-process system, where certain cognitive processes are rapid, effortless, and intuitive -generated by System 1 -while others are slower and require greater mental effort -generated by System 2 [17]. This understanding is crucial for understanding human decision making, which often occurs under uncertainty with incomplete information. In such situations, decision-makers rely on heuristics-simple, quick judgments-as proxies for unknown answers. These heuristics are typically generated by System 1 and can lead to cognitive biases if not scrutinized by System 2.
While cognitive biases in classical decision-making have been extensively studied, those arising in AI-assisted decision-making are only now gaining attention [17]. This is due to the recent prominence of AI-assisted decision making and the previously unchallenged belief that AI tools inherently enhance decision efficiency [12]. However, recent studies suggest that AI can mitigate in one hand, and reinforce in another, cognitive biases in decision-making. This section provides an analysis of each cognitive bias in AI-assisted decision-making and how they can impact the decision making process, supported by various examples from the literature.
Confirmation bias involves seeking information that confirms existing beliefs, disregarding contradictory data, and making decisions that reinforce initial beliefs [17]. In AI-assisted decision-making, it occurs when AI suggestions align with preexisting beliefs, reducing critical thinking [11]. Users may accept or reject recommendations solely on the basis of alignment, neglecting other factors. This bias is more common in lay users than in experts [18]. Additionally, when looking at explanations, users may selectively focus on parts confirming their beliefs [19].
Automation bias is the tendency to favor decisions made by automated systems, even when they are prone to errors, leading to overreliance [20]. In AI-assisted decision-making, this cognitive effect occurs, especially when the cognitive load of the decision is high [21,22] or when the expertise of the human decider is low. Explainable AI [11] and cognitive forcing functions [23,14] are viewed as solutions that can mitigate this bias.
In contrast to automation bias, algorithm aversion bias leads humans to dismiss algorithmic decisions just because it is a machine [24]. In AI-assisted decision-making, users may prefer human recommendations as they perceive them as easier to understand [25]. In critical tasks, individuals may favor human discretion over algorithmic application of fairness principles, as humans can transcend these principles if necessary [26]. This bias can lead to under-reliance and disuse of AI systems.
The anchor effect, occurs when individuals estimate uncertain quantities, influenced by initial reference points called anchors [27]. These anchors, whether informative or randomly assigned, bias final estimates. This effect is prominent in various contexts, particularly in quantitative estimations like real estate pricing, where initial listing prices affect subsequent estimates [28]. It also affects qualitative judgments, such as sentencing decisions in judicial settings, evidenced by studies that show significant variations based on initial sentencing demands [29].
Cognitive biases that arise from the anchor effect in AI-assisted decision-making stem from direct and indirect anchoring processes. An immediate anchor is the AI's suggestion, influencing decisions by guiding towards similar options and potentially neglecting other factors. This can yield varied outcomes. If the AI system surpasses human capabilities, it improves decision accuracy [30]. In contrast, reliance on less accurate AI recommendations can lead to overreliance [31]. Additionally, when AI recommendations come after humans initiate decision-making, the original human estimate can act as an anchor. Two possibilities emerge: If the AI suggestion aligns with the initial estimate, confirmation bias may prompt immediate adoption, as previously discussed. On the contrary, if the AI suggestion differs, individuals tend to stick to their original estimate and may not rely on the system.
Anchoring bias may manifest itself indirectly in situations involving ordering and framing effects. For example, when individuals receive accuracy information about an AI assistant, it can act as an anchor, reducing trust compared to scenarios without disclosure [32]. Additionally, in repeated use of AI assistants, users may initially perceive high accuracy, leading to inflated trust and anchoring future assessments to this impression, increasing reliance on the system [33]. The opposite scenario may also occur.
One notable human behavioral trait is loss aversion, where losses hold more weight than equivalent gains [17]. This bias can extend to AI-assisted decision-making. Humans may focus more on false positives than false negatives in AI errors [15], leading to algorithm aversion bias and under-reliance. Additionally, in risky decision tasks, individuals tend to trust their beliefs over AI, contributing to a lack of trust, which is challenging to mitigate [31].
Availability bias leads to an overestimation of event frequencies based on easily recalled instances [17]. In human decision-making with AI recommendations, users can incorrectly estimate the frequency of AI suggestions due to memory recall [11], affecting AI reliance. Wang et al. propose presenting base frequencies to mitigate this bias [11]. Furthermore, explanations can also induce availability bias if users recall relevant knowledge. Users may perceive explanations as more or less plausible based on recalled knowledge, potentially reinforcing biased perceptions [19].
The cognitive biases that arise and their effects can vary depending on the characteristics of the decisionmaking task. Within the AI-assisted decision-making framework described in section 2, several biases may occur in relation to the three components. For example, an individual with limited knowledge of AI but high expertise in the relevant field may exhibit algorithm aversion, leading to a lack of trust in AI recommendations [24]. Conversely, a lack of experience on the part of the human decision-maker can result in automation bias. In group decision-making scenarios, there is a tendency toward groupthink-a bias where individuals conform to the majority opinion, potentially increasing overreliance on the AI system [34,35].
The task's characteristics also play a significant role. High-stakes decisions are more prone to loss aversion bias, which may lead to under-reliance on the AI system [15,31]. Conversely, highly complex tasks may result in automation bias, as the human decision-maker might rely more on the AI system due to the task's difficulty [22]. Even task design can influence the decision-making process and the emergence of specific biases. For instance, if the AI recommendation is presented at the outset, alongside the collection of all relevant information, it could trigger anchoring bias, where the AI recommendation serves as an anchor [30]. However, a cognitive forcing function that delays showing the recommendation until after a certain period could lead to confirmation bias, where the human decision-maker has already formed an opinion, and the AI recommendation merely reinforces this decision, reducing critical thinking [18].
Finally, regarding the AI component, a high cognitive load required to interpret the explanations-or even just the presence of explanations-might induce automation bias in the human decision-maker [22].
Each decision-making task is defined by the unique characteristics of its components-human, task, and AI-and the interplay of these factors results in different cognitive biases for each task. Therefore, it is crucial to analyze various cognitive forcing function designs and explanations in each scenario. This analysis should identify not only the biases that need to be mitigated but also those that could potentially be introduced by the explanations or the new design.
The paper focuses on common cognitive biases in AI-assisted decision-making rather than covering all possible biases. Techniques such as XAI and cognitive forcing functions can help address some of these biases, but they can also unintentionally introduce new ones. For instance, explanations can trigger the mere exposure effect, leading to overreliance [15,14]. Additionally, complex explanations that aim for completeness [13] or present arguments for and against each option [12] can induce automation bias due to their high cognitive demands [22].
Cognitive forcing functions are designed to enhance user engagement in AI-assisted decision-making. These kinds of techniques can also trigger cognitive biases similar to those caused by explanations. For example, introducing AI suggestions after the user's initial decision can lead to anchoring effects or algorithm aversion [30]. Moreover, if not carefully designed, these functions can inadvertently reduce engagement by making the decision process overly complex. In conclusion, while these techniques offer valuable solutions, they also present challenges, requiring a nuanced approach to effectively manage potential cognitive biases for each case of AI-assisted decision-making task.
Acknowledgments This research was funded by INESC-ID (UIDB/50021/2020), as well as the projects CRAI C628696807-00454142 (IAPMEI/PRR) and TAILOR H2020-ICT-48-2020/952215 and HumanE AI Network H2020-ICT-48-2020/952026.