Teamness and Trust in AI-Enabled Decision Support Systems: Current Challenges and Future Directions

Abstract

Artificial intelligence-enabled decision support systems (AI-DSSs) can process highly complex information to recommend or execute decisions autonomously, but often at the cost of lacking transparency and explainability. The existence of inherent human limitations in understanding increasingly inexplicable AI-DSSs, however, raise the question of people's roles in the high-stakes, rapid decision-making domains for which AI-DSSs are being developed. In this paper, we summarize the current state of human-AI teaming research in light of how emergent cognitive properties arise from human interactions with AI-DSSs. We also identify important open research questions in accounting for the teamness of AI-DSSs in light of current directions in trust research. Finally, we outline some anticipated challenges in methodological approaches and generalizability when attempting to design studies to answer these questions.

Keywords

• eol>Decision support systems
• Trust
• Human-AI teaming
• Teamness
• AI-DSS

1. Introduction

Decision support systems (DSSs) have traditionally supplemented human cognitive capabilities with computerized information processing to improve decision-making quality and speed [ 1 ]. Initial applications during the 1970s-1990s were largely tools that collated and presented information to support human decision-making, such as in military housing occupancy assignment, oficer manpower planning, and aircraft design compendiums [ 2, 3 ]. In the 2000s, DSS design philosophy shifted towards prosthetic functionalities that recommended decisions and actions altogether [ 1, 4 ]. The acceleration and democratization of machine and deep learning methods in recent years have introduced artificially intelligent DSSs (AI-DSS; [ 5 ]) that are capable of processing highly complex information and executing actions autonomously. AI-DSSs are critical components of the U.S. Department of Defense (DoD)’s multi-domain operations (MDO) in warfare [ 6 ]. For example, the DoD’s MDO roadmap includes the navigation and engagement mechanisms of unmanned combat vehicles [ 7, 8 ].

However, the advanced capabilities of AI-DSSs often come at the cost of reduced transparency and explainability [ 9 ]. This is concerning, as the unintended adoption of faulty DSS recommendations can result in lethal or catastrophic outcomes. An infamous example is from 2004, when US military personnel failed to veto automated engagements made by the Patriot missile system, ultimately causing the fratricide of British and American pilots [ 4 ]. The deployment of AI-DSSs for increasingly complex and sensitive applications risks similar or even more catastrophic outcomes within other high-stakes domains, including national security, law enforcement, and healthcare. Thus, for ethical and legal motivations, keeping people in the decision-making loop remains the status quo for DSSs in such high-stakes domains [ 10 ].

It is important to note that the same advancements behind the rise of increasingly inexplicable AI-DSSs can also lead to more impactful ways of integrating human and algorithmic decisionmaking processes. Teams consisting of humans and AI, capable of surpassing systems of only humans or AI, are now considered to be on the horizon [ 6 ]. But as teammates interact, a team may exhibit diferent extents of group-level team properties, i.e., “ teamness”, compared to other teams or to itself at diferent points in time [ 11 ]. Teamness not only afects the extent to which a team performs at levels greater than the sum of its parts; it also has recursive impacts on how people trust and perceive their teammates, which then influences the teamness of their future interactions. Although it has been theorized that people may not tend to perceive AI or robots as teammates in comparison to other people [ 12, 13 ], the role of teamness in AI-DSSs has not previously been explored in the literature. In this paper, we outline the current state of research about some of the diferent ways that human interactions with AI-DSSs have been construed in light of trust in AI. We then identify open research questions and challenges for incorporating teamness in future studies on the role of trust in AI-DSSs.

2. Human Supervisory Control over AI-DSSs

People typically occupy supervisory roles over DSSs to perform checks and balances, especially in high-stakes and safety-critical domains [ 14 ]. An example can be found in airport security checkpoints, automated face recognition technology (AFRT) systems aid security agents in verifying if a traveler’s identity matches their itineraries and identification documents [ 15 ]. The AFRT system then presents a recommended decision (e.g., “match” or “mismatch”), which is subsequently reviewed by the security agent who can either approve or veto the decision. Put simply, people serve as the ultimate decision-making authority in supervisory control systems like AFRT-assisted border control.

But with AI capabilities already surpassing human accuracy in many circumstances—as in the case of AFRTs and face matching [ 16, 17 ]—it is unclear if it remains appropriate for people to exert supervisory control over AI-DSSs. In applications where AI-DSSs are deployed for rapid decisionmaking, human supervision over AI-recommended decisions may also result in suboptimal system performance. Furthermore, people are subject to numerous cognitive limitations and biases surrounding technology use. To name a few, people are prone to “automation bias”, or the tendency to blindly adopt an automation-recommended decision even while faced with evidence that the recommendation is faulty [ 18 ]. People also often become complacent in the presence of AI, resulting in the inefectual detection of errors or irregularities in AI outputs [ 19 ]. Outside of biased decision-making tendencies, people’s scrutiny of AI-recommended decisions and actions is also subject to limitations of workload, task expertise, and individual diferences, among others [20]. These phenomena have been linked to potentially inappropriate uses of DSSs, such as overreliance on automated decisions, aversion to algorithm-heavy decision processes, and the relegation of human operators to passive decision-making roles [21, 22]. Nevertheless, it has been argued that human supervision over AI-DSSs is vital even when it may not significantly improve system performance [23].

For one, the inclusion of humans in the loop who are vested with final decision-making authority is essential to meet many legal and ethical requirements governing the use of AI in consequential decision domains [24]. The proposed EU Artificial Intelligence Act, for instance, mandates human oversight over AI systems for “preventing or minimising the risks to health, safety or fundamental rights [...] when a high-risk AI system is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse” [25, p. 51]. However, such measures have been criticized for lacking specificity on the expected mechanisms, outcomes, or metrics by which human oversight can ensure the safety and fairness of AI-assisted decisionmaking [26]. A more apparent rationale for human supervisory control over AI-DSSs is that many critical situations may require innovative solutions beyond the scope of algorithmic decision-making [ 4, 23 ]. For example, after a flock of Canadian geese struck both jet engines, pilots of U.S. Airways Flight 1549 minimized their reliance on cockpit DSSs to execute a manual emergency landing on the Hudson River, saving all 155 passengers aboard [27].

3. AI-DSSs as Human-AI Teams

3.1. Teamness in AI-DSSs

The inherent limitations of human supervisory control raise questions about the role of people in high-stakes domains where AI-DSSs are prevalent. But while people are increasingly taking on the role of collaborators rather than supervisors when interacting with AI-DSSs, many AI models perform poorly when cooperating with people [28]. This raises a more important question: could AI-DSSs with humans-in-the-loop result in stronger overall system performance in comparison to humans or AI alone? In 2005, amateur chess players interacting with AI-DSSs in so-called “centaur” teams successfully defeated International Grandmasters and AIs, demonstrating that it is possible for non-experts and AI algorithms to perform better together[29].

Understanding the reasons behind this phenomenon and the mechanisms driving its occurrence is essential for broadening its application to other categories of AI-DSSs involving humans in the loop. A possible explanation lies in the theory that decisions from DSSs are more meaningfully studied as outcomes of joint cognitive systems aimed at achieving common goals[30]. This parallels the concept of interactive team cognition, where teams of interacting individuals achieve results beyond the sum of individual inputs[31]. The idea that humans can form teams with autonomous forms of automation to achieve exceptional results has recently gained traction [32], though not without controversy (cf. [ 13, 33 ]). As with [ 11 ], we posit that “teamness” emerges according to the extent that interdependent interactions between people and AI result in cognitive outputs that cannot be broken down into individual human or AI contributions. This is irrespective of whether unique roles or tasks are apparent in human-AI interactions, which current definitions of human-AI teams stipulate that people and AI teammates must exhibit while performing interdependent tasks towards common goals [32].

3.2. Disambiguating Teammate-likeness, Human-likeness, and Teamness

Another important consideration is that the team cognitive qualities that arise from human-AI team interactions are closely related to people’s perceptions of their AI counterparts. This is why, for example, many AI-DSSs are designed with human-like characteristics to foster trustworthy and likable perceptions (e.g., [34, 35, 36]) by inducing anthropomorphism. Anthropomorphism is the attribution of human-like characteristics to an inanimate object, influenced by people’s perceptions and desires to socially engage with it as they would with other people [ 37 ]. The widespread availability of AI products and interfaces capable of human-like interactions (e.g., Siri, Amazon Echo, ChatGPT) has led to an observed increase in people’s tendencies to socialize with AI, treating and perceiving them as they would other people [ 38 ]. This trend is only expected to accelerate with the increasing prevalence of generative AI algorithms [ 39 ].

On the other hand, people may interact with AI-DSSs in human-like ways to ease interactions without perceiving them as human-like or as teammates—a phenomenon referred to as “ethopoeia” [ 40 ]. For instance, people may withhold criticism, use verbal politeness cues (“please”, “excuse me”, etc.), or talk about an AI using gendered language, yet retrospectively view it as an inanimate object [ 41, 42, 43, 44 ]. Such findings have led some to argue that deliberately portraying AI as teammates may lead to dangerous or misleading expectations and that AI should be instead be designed to be viewed only as tools or “supertools” [ 13, 33 ]. However, it appears that many such arguments result from conflating teammate-like or human-like perceptions and expectations of a non-human agent, which are individual-level cognitive phenomena, with the team cognitive properties that arise from human-AI interactions.

Cautionary warnings against depicting AI as teammates are not unfounded; theories of perceived AI teammate-likeness (e.g., [ 12 ]) support the idea that human-like interaction capabilities may make a person more likely to form teammate-like perceptions of an AI counterpart. However, because there is a tendency for researchers to conflate anthropomorphism and ethopoeia [ 45 ], it bears clarification that human-like perceptions resulting from ethopoetic interactions with AI are not precursors to teammate-like perceptions of (or interactions with) AI-DSSs—though they are likely correlated [ 46 ]. In addition, there is mixed empirical support for the relationship between team decision-making performance—a dimension of teamness—and teammate-like perceptions of AI [ 47, 48 ].

It is an open research question how people perceive the teamness of their interactions with AI also influences their teammate-like perceptions about it, or vice versa. We posit that, as the team cognitive properties that emerge when teammates interact [31] afect humanlike and teammate-like expectations and behaviors that form at the individual level, these individual-level phenomena, in turn, influence the teamness of future team interactions. In other words, there is likely a dynamic feedback loop involving individual-level social perceptions of AI and the teamness of collective actions, similar to other multi-scale team processes like physical coordination and communication [ 49 ]. It has been established that social expectations ultimately drive how people trust and efectively interact with non-human counterparts in a similar feedback loop [ 21, 50, 51 ]. However, the recursive impacts of teamness have not been considered in previous calls for investigating the formation of trusting relationships in human-AI teams (e.g., [ 6 ]).

4. Trust in AI-DSSs: Accounting for Teamness

4.1. Trust in an AI Teammate

Trust has been defined as a person’s willingness to rely on automation as an aid to achieve specific goals [ 52 ]. The efectiveness of DSS-assisted decision-making is understood to be a function of how a person’s expectations are calibrated to its actual performance and process capabilities in situations that the DSS was designed for [ 1 ]. Thus, the relationship between trust and DSS performance has been a prime focus of research over the last three decades [ 21, 51, 52 ].

Many of the aforementioned cognitive limitations and biases that plague human supervisory control over AI-DSSs are related to trust. In general, interacting with AI-DSSs that are also capable of autonomously executing recommended decisions and actions can preclude continuous human engagement in AI-assisted decision-making. This is partly due to combinations of the pressures of making consequential decisions in real-time, the complexity of the environment, and the need for rapid decision-making beyond human capabilities[ 53 ]. Thus, human supervision of AI-DSSs can be characterized by a heightened sense of vulnerability, making trust central to maintaining efective system interactions in the long run. This is similar to how people may demonstrate higher propensities to adopt relational trusting expectations and behaviors in their interactions in human-AI teaming setups [ 51 ].

Nevertheless, diferences in how trust relates to decision-making performance when people serve as teammates to AI-DSSs as opposed to supervisors remain poorly understood. This raises several questions. For example, do people evaluate the trustworthiness of an AI agent diferently depending on whether they are prompted to consider it as a teammate or a tool? How do an AI agent’s teammate-likeness and perceived trustworthiness relate to the teamness of human-AI decision-making interactions? Do patterns of human-AI decision-making interactions over time correlate with trustworthiness perceptions? And do these patterns and correlations change depending on whether people are prompted to consider AI-DSS as a teammate or a tool?

Addressing these gaps may entail establishing the relationships between various trust constructs, human-AI teamness perceptions and qualities, and the factors that are known to afect both, such as anthropomorphism [ 54 ]. For human supervisory structures over AI-DSSs, this may include investigating how teamness and teamness-related social constructs relate to traditional measures of trust in automation [ 55 ]. These include reflective questionnaires that ask for people’s perceptions of the technology (e.g., [ 56, 57 ]) and behavioral metrics on how people adopt AI-recommended decisions (i.e., compliance) or rely on them when human inputs do not appear to be needed [ 58 ]. But if AI-DSSs are to be considered as human-AI teams, establishing the relationships between these traditional trust constructs and measures is simply a precursor to understanding how trust relates to teamness. Teamness is a team-level phenomenon; therefore, we must also consider how trust manifests at the team level beyond perceptions and behaviors of individual team members.

4.2. Human-AI Team Trust: Trust in the Team or Emergent Team Trust?

Team-level constructs of trust (i.e., “team trust”) have received considerable attention in the human-AI teaming literature, albeit inconsistently defined. Team trust has commonly been used to refer to an individual’s trusting attitudes towards their team as a whole, measured through self-report questionnaires (e.g., [ 59 ]). This is consistent with the notion that an individual’s trust in their team is afected by their perceptions of their team’s overall performance [ 11, 60 ]. We refer to this variable as a person’s trust in the team.

In considering how people trust in their team, it should be considered that may not necessarily perceive robot or AI counterparts as teammates in the same way that they would other people [ 12, 13 ]. If this is the case, then some may intuitively interpret the subjective questionnaires that ask for trustworthiness ratings of one’s own team as not including one or more AI agents. On the other hand, explicitly stating that such ratings must include AI counterparts may induce undue perceptions of teammate-likeness or human-likeness, among others, and confound the measurement of team trust. For studies on how trust develops in long-term AI-DSS team settings, the repeated administration of such surveys also risk amplifying these issues of internal validity [61, ch. 2]. Behavioral measures can also be problematic: a human teammate’s execution of a human-AI team’s consensus decision may be attributed to a number of social factors beyond a person’s trust in the team [ 51 ]. We note, though, that the current methodological issues surrounding the measurement of this team trust construct do not mean that it is conceptually unsound. More innovative ways are nonetheless needed to properly account for how teamness informs and is subsequently afected by team members’ trust in their team.

Another way that team trust has been conceptualized is as an emergent team-level phenomenon that generally describes how teammates trust one another, which we refer to as emergent team trust. One way that emergent team trust has been approach involves investigating trust through observable markers in the context of reciprocal human-AI team relationships [ 62 ], akin to behavioral measures of a person’s trust in an individual AI agent. The utility of considering an emergent team trust concept can theoretically be seen, for example, in proposed uses of network models to show how trust relations manifest and propagate in large teams comprising multiple people or AI agents [ 63 ]. But network models generally apply only to teams with more than two members; most human-DSS interactions presumably take place in dyadic settings [28]. Furthermore, as with behavioral measures for individuals’ trust in their team, interpreting the emergence of trusting behavioral patterns at the team level is fraught with issues of causal validity. Valid markers of emergent team trust have been scarcely explored in the various AI-DSS interaction paradigms, and should be addressed in future empirical research.

These two operationalizations of team trust are neither incompatible nor mutually exclusive—an individual’s trust in their team certainly afects the overall trust dynamics as teammates interact with each other [ 51, 64 ]. Indeed, combining these constructs may be appropriate in accounting for the teamness of AI-DSSs, depending on the research question at hand and the teaming context of interest. For instance, in multi-human AI-DSSs, one may consider how individual perceptions of teamness and a team’s trustworthiness relate to each other and to emergent patterns of trusting behaviors. Jointly measuring and interpreting questionnaire data on people’s trust in their team alongside network-based parameters of emergent team trust dynamics may make for a straightforward integration of both approaches if applicable to the team task context.

Defining emergent team trust in terms of aggregations of trust-in-the-team measures may also be considered, e.g., as the sum or average of individual members’ ratings of their trust in the team [ 64 ]. However, we note the need to ensure the construct validity of attempts to integrate both definitions of team trust through aggregation techniques. Caution should be exercised in selecting aggregation techniques that are interpretable and consistent with compilational definitions of emergence (i.e., multi-level, or involving both individual and team-level scales), which inform our current understanding of teamness [ 11 ]. We refer the reader to [ 65 ] for a detailed discussion of various aggregation techniques in light of emergent team phenomena.

5. Methodological Challenges

We acknowledge current methodological challenges in designing studies for investigating trust in AI-DSSs in light of human-AI teaming. First, we anticipate challenges in designing experimental testbeds that will induce interactions that are ecologically valid with real-world decision-making behaviors in the high-stakes domains within which AI-DSSs are expected to ifnd widespread use [ 6 ]. For instance, laboratory simulations of AI-DSS in next-generation combat vehicles (e.g., [66]) cannot be designed to realistically or ethically induce levels of risk perceptions that real-world warfare scenarios involve. In many such simulations for other high-stakes decision-making scenarios, aspects of operational accuracy surrounding real-world task demands may also have to be sacrificed to ensure the feasibility of testbed implementations.

There is also an “assumption gap" in current AI-DSS research, in which theoretical use contexts and application domains require certain AI capabilities to produce relevant outputs for teaming with people, but are not technically feasible due to limitations in current AI technology. For example, current limitations in AI data visualization capabilities mean that heatmaps of salient features cannot be adequately used as an explanation tool for face-matching tasks in AFRT simulations. The current state of the art produces heat maps that do not result in demonstrably diferent levels of team-like performance in AFRT simulation experiments (e.g., [67]). Current experimental paradigms heavily rely on simulated AI capabilities, most commonly in the form of Wizard-of-Oz settings [68] that may not represent realistic AI applications [69].

Another growing challenge is the development of testbeds through proprietary applications, such as those involving large language models like ChatGPT [70] or of-the-shelf games such as Minecraft and Roblox [71]. Many commercial, of-the-shelf models ofer limited explanations of how certain AI capabilities were developed, instead responding, for example, with “As an AI model, I don’t have access to...". When approaching the study of human trust in AI from a teaming perspective, AI should be able to act like a teammate, and acting like so means that the technology has to be there to produce teammate-like behaviors. We also acknowledge, however, that the use of open-source models may result in AI-DSSs that model state-of-the-art but at considerably poorer performance levels (e.g., [67]).

Finally, the development of AI-DSSs that are capable of a suficiently wide range of interaction mechanisms to resemble teaming with people may require the development of automated observational data collection protocols at unprecedented scales. We note that initial eforts are underway [72]; nevertheless, successful applications of such protocols remain virtually non-existent or unpublicized.

6. Conclusion

The continued growth in AI-DSS capabilities and applications in various high-stakes decision domains calls for reconsidering the role of people in human-in-the-loop decision-making. In this paper, we presented how we can advance the current state of human-AI teaming research in light of how human-AI interactions within AI-DSSs can exhibit team cognitive properties, i.e., teamness, to varying degrees. We posit that the teamness of human-AI interactions in an AI-DSS afects the formation of social perceptions like trust at the individual level, which in turn influence the future teamness of the AI-DSS in a dynamic feedback loop. As such, future research should investigate the relationships between these cognitive phenomena and joint decisionmaking performance in the design of AI-DSSs. Transdisciplinary (rather than interdisciplinary) eforts are needed to address the current technological assumption gaps and testbed design challenges in studying the teamness of human-AI interactions with AI-DSSs. Research teams solving these problems must integrate an understanding of AI-DSS development and expertise in human factors, among many others to streamline the development of ecologically valid study methodologies. Overall, there are several challenges and open research questions in the advancement of trust theory in teams of people and AI-DSSs. We believe that a teamness perspective can improve our understanding of future human-in-the-loop AI-assisted decisionmaking paradigms.

Acknowledgments

The authors thank Nia Amazeen for her invaluable insights in the formulation of this topic. We also thank Frank Jäkel, Tilman Nols, Ewart de Visser, Lionel P. Robert, Glenda Hannibal, and Myrthe L. Tielman for strengthening the arguments of this paper through their inputs and questions at the MULTITRUST workshop in Munich, Germany. [20] M. Knop, S. Weber, M. Mueller, B. Niehaves, Human Factors and Technological Characteristics Influencing the Interaction of Medical Professionals With Artificial Intelligence–Enabled Clinical Decision Support Systems: Literature Review, JMIR Hum Factors 9 (2022) e28639. doi:10.2196/28639. [21] R. Parasuraman, V. Riley, Humans and Automation: Use, Misuse, Disuse, Abuse, Hum

Factors 39 (1997) 230–253. doi:10/bbntx8. [22] T. Snow, From satisficing to artificing: The evolution of administrative decision-making in the age of the algorithm, Data & Policy 3 (2021/ed) e3. doi:10/gscn56. [23] R. Parasuraman, C. D. Wickens, Humans: Still Vital After All These Years of Automation,

Hum Factors 50 (2008) 511–520. doi:10/dg3kqc. [24] M. Ananny, K. Crawford, Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability, New Media & Society 20 (2018) 973–989. doi:10/gddxrg. [25] European Commission, Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL LAYING DOWN HARMONISED RULES ON ARTIFICIAL INTELLIGENCE (ARTIFICIAL INTELLIGENCE ACT) AND AMENDING CERTAIN UNION LEGISLATIVE ACTS, 2021. [26] M. Veale, F. Z. Borgesius, Demystifying the Draft EU Artificial Intelligence Act — Analysing the good, the bad, and the unclear elements of the proposed approach, Computer Law Review International 22 (2021) 97–112. doi:10.9785/cri-2021-220402. [27] E. Atkins, Emergency Landing Automation Aids: An Evaluation Inspired by US Airways Flight 1549, in: AIAA Infotech@Aerospace 2010, American Institute of Aeronautics and Astronautics, Atlanta, Georgia, 2010. doi:10/gjgprk. [28] E. K. Chiou, J. D. Lee, Cooperation in Human-Agent Systems to Support Resilience: A

Microworld Experiment, Hum Factors 58 (2016) 846–863. doi:10/f82nh8. [29] N. Case, How To Become A Centaur, Journal of Design and Science (2018). doi:10/gscn58. [30] D. D. Woods, Cognitive Technologies: The Design of Joint Human-Machine Cognitive

Systems, AI Magazine 6 (1985) 86–86. doi:10/grpwb9. [31] N. J. Cooke, J. C. Gorman, C. W. Myers, J. L. Duran, Interactive Team Cognition, Cogn Sci 37 (2013) 255–285. doi:10/gf69qb. [32] T. O’Neill, N. McNeese, A. Barron, B. Schelble, Human–Autonomy Teaming: A Review and Analysis of the Empirical Literature, Hum Factors (2020) 001872082096086. doi:10. 1177/0018720820960865. [33] B. Shneiderman, Intelligent Agents and Supertools, in: Human-Centered AI, Oxford

University Press, New York, 2022, pp. 99–103. [34] J. Murphy, U. Gretzel, J. Pesonen, Marketing robot services in hospitality and tourism: The role of anthropomorphism, Journal of Travel & Tourism Marketing 36 (2019) 784–795. doi:10/ghw345. [35] C. Pelau, D.-C. Dabija, I. Ene, What makes an AI device human-like? The role of interaction quality, empathy and perceived psychological anthropomorphic characteristics in the acceptance of artificial intelligence in the service industry, Computers in Human Behavior 122 (2021) 106855. doi:10/gk49kw. [36] A. Fenwick, G. Molnar, The importance of humanizing AI: Using a behavioral lens to bridge the gaps between humans and machines, Discov Artif Intell 2 (2022) 14. doi:10/gscn6k. [66] H. D. Graham, G. J. Lematta, N. J. Cooke, Exploration of Time-Based Resilience Measurements for Next Generation Combat Vehicle Teams, in: 2021 Resilience Week (RWS), 2021, pp. 1–7. doi:10.1109/RWS52686.2021.9611812. [67] E. K. Chiou, P. Salehi, E. Blasch, J. Sung, M. C. Cohen, A. Pan, M. Mancenido, A. Mosallanezhad, Y. Ba, S. Bhatti, Trust in AI-Enabled Decision Support Systems: Preliminary Validation of MAST Criteria, in: 2022 IEEE 3rd International Conference on HumanMachine Systems (ICHMS), 2022, pp. 1–1. doi:10/gscn6f. [68] L. Riek, Wizard of Oz Studies in HRI: A Systematic Review and New Reporting Guidelines,

JHRI (2012) 119–136. doi:10.5898/JHRI.1.1.Riek. [69] J. Simpson, M. Richardson, D. Richards, A Wizard or a Fool? Initial Assessment of a Wizard of Oz Agent Supporting Collaborative Virtual Environments, in: Proceedings of the 10th International Conference on Human-Agent Interaction, HAI ’22, Association for Computing Machinery, New York, NY, USA, 2022, pp. 299–301. doi:10.1145/3527188. 3563930. [70] OpenAI, Introducing ChatGPT, https://openai.com/blog/chatgpt, 2022. [71] G. J. Lematta, C. C. Corral, V. Buchanan, C. J. Johnson, A. Mudigonda, F. Scholcover, M. E.

Wong, A. Ezenyilimba, M. Baeriswyl, J. Kim, E. Holder, E. K. Chiou, N. J. Cooke, Remote research methods for Human–AI–Robot Teaming, Human Factors and Ergonomics in Manufacturing & Service Industries 32 (2022) 133–150. doi:10.1002/hfm.20929. [72] Artificial Social Intelligence for Successful Teams, About ASIST, https://artificialsocialintelligence.org/about/, 2023.

References

[1] J. M. McGuirl , N. B. Sarter , Supporting Trust Calibration and the Efective Use of Decision Aids by Presenting Dynamic System Confidence Information , Hum Factors 48 ( 2006 ) 656 - 665 . doi: 10 .1518/001872006779166334.

[2] S. Eom , E. Kim , A survey of decision support system applications ( 1995 -2001), J Oper Res Soc 57 ( 2006 ) 1264 - 1278 . doi:10/dh9whx.

[3] H. B. Eom , S. M. Lee , A Survey of Decision Support System Applications ( 1971 -April 1988 ), Interfaces 20 ( 1990 ) 65 - 79 . doi:10/bb85k4.

[4] M. L. Cummings , Automation and Accountability in Decision Support System Interface Design , JOTS 32 ( 2006 ). doi:10.21061/jots.v32i1.a.4.

[5] S. Gupta , S. Modgil , S. Bhattacharyya , I. Bose , Artificial intelligence for decision support systems in the field of operations research: Review and future scope of research , Ann Oper Res 308 ( 2022 ) 215 - 274 . doi: 10 .1007/s10479-020-03856-6.

[6] E. National Academies of Sciences, Human-AI Teaming: State-of-the- Art and Research Needs , 2021 . doi: 10 .17226/26355.

[7] N. J. Cooke , V. Gawron , Human Systems Integration for Remotely Piloted Aircraft Systems , in: Remotely Piloted Aircraft Systems: A Human Systems Integration Perspective , John Wiley & Sons, Ltd, 2016 , pp. 1 - 14 . doi: 10 .1002/9781118965900.ch1.

[8] K. M. Fahey , M. J. Miller , DoD Unmanned Systems Integrated Roadmap 2017 -2042, Technical Report , Department of Defense, 2017 .

[9] C. Rudin , Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead , Nat Mach Intell 1 ( 2019 ) 206 - 215 . doi: 10 .1038/ s42256-019-0048-x.

[10] M. De-Arteaga , R. Fogliato , A. Chouldechova , A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores , in: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20 , Association for Computing Machinery, New York, NY, USA, 2020 , pp. 1 - 12 . doi:10/gjzbr6.

[11] N. J. Cooke , M. C. Cohen , W. C. Fazio , L. H. Inderberg , C. J. Johnson , G. J. Lematta , M. Peel , A. Teo , From Teams to Teamness: Future Directions in the Science of Team Cognition , Hum Factors ( 2023 ). doi:10/gscn55.

[12] K. T. Wynne , J. B. Lyons , An integrative model of autonomous agent teammate-likeness , Theoretical Issues in Ergonomics Science 19 ( 2018 ) 353 - 374 . doi: 10 .1080/1463922X. 2016 . 1260181 .

[13] V. Groom , C. Nass , Can robots be teammates?: Benchmarks in human-robot teams , Interaction Studies 8 ( 2007 ) 483 - 500 . doi: 10 .1075/is.8.3.10gro.

[14] T. B. Sheridan , Human Supervisory Control, in: G. Salvendy (Ed.), Handbook of Human Factors and Ergonomics , 4th ed., John Wiley & Sons, Ltd, 2012 , pp. 990 - 1015 .

[15] J. Sanchez del Rio , D. Moctezuma , C. Conde , I. Martin de Diego, E. Cabello, Automated border control e-gates and facial recognition systems , Computers & Security 62 ( 2016 ) 49 - 72 . doi:10/f88nq8.

[16] A. J. O'Toole , X. An , J. Dunlop , V. Natu , P. J. Phillips , Comparing face recognition algorithms to humans on challenging tasks , ACM Trans. Appl. Percept . 9 ( 2012 ) 16 : 1 - 16 : 13 . doi:10/ gg2g6x.

[17] A. J. O'Toole , F. Jiang , D. Roark , H. Abdi , Predicting Human Performance for Face Recognition , in: W. Zhao , R. Chellappa (Eds.), Face Processing: Advanced Modeling and Methods , Academic Press, New York, NY, USA, 2006 , pp. 293 - 319 .

[18] L. J. Skitka , KATHLEEN. Mosier, M. D. Burdick , Accountability and automation bias, International Journal of Human-Computer Studies 52 ( 2000 ) 701 - 717 . doi: 10 .1006/ijhc. 1999 . 0349 .

[19] C. D. Wickens , B. A. Clegg , A. Z. Vieane , A. L. Sebok , Complacency and Automation Bias in the Use of Imperfect Automation , Hum Factors 57 ( 2015 ) 728 - 739 . doi:10/f7kzg2.

[37] N. Epley , A. Waytz , J. T. Cacioppo , On Seeing Human: A Three-Factor Theory of Anthropomorphism, Psychological Review 114 ( 2007 ) 864 - 886 . doi: 10 .1037/ 0033 - 295x . 114 . 4.864.

[38] A. Purington , J. G. Taft , S. Sannon , N. N. Bazarova , S. H. Taylor , "Alexa is my new BFF": Social Roles, User Satisfaction, and Personification of the Amazon Echo , in: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA '17 , Association for Computing Machinery, New York, NY, USA, 2017 , pp. 2853 - 2859 . doi:10/ffrc.

[39] S. Erwin , On National Security | Analyzing intelligence in the age of ChatGPT , 2023 .

[40] C. Nass , J. Steuer , E. Tauber, H. Reeder , Anthropomorphism, agency, and ethopoeia: Computers as social actors , in: INTERACT '93 and CHI '93 Conference Companion on Human Factors in Computing Systems, CHI '93 , Association for Computing Machinery, New York, NY, USA, 1993 , pp. 111 - 112 . doi:10/dms7hh.

[41] C. Nass , Etiquette equality: Exhibitions and expectations of computer politeness , Commun. ACM 47 ( 2004 ) 35 - 37 . doi:10/fkdx68.

[42] C. Nass , Y. Moon, Machines and Mindlessness: Social Responses to Computers, J Social Isssues 56 ( 2000 ) 81 - 103 . doi:10/cqzrs6.

[43] M. C. Cohen , M. Demir , E. K. Chiou , N. J. Cooke , The Dynamics of Trust and Verbal Anthropomorphism in Human-Autonomy Teaming , in: 2021 IEEE 2nd International Conference on Human-Machine Systems (ICHMS) , IEEE, Magdeburg, Germany, 2021 , pp. 1 - 6 . doi:10/gscn52.

[44] K. Fischer , Interpersonal variation in understanding robots as social actors , in: 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , 2011 , pp. 53 - 60 . doi: 10 .1145/1957656.1957672.

[45] L. Takayama , Making sense of agentic objects and teleoperation: In-the-moment and reflective perspectives , in: 2009 4th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , 2009 , pp. 239 - 240 . doi:10/fq85t8.

[46] J. B. Lyons , K. Sycara , M. Lewis , A. Capiola , Human-Autonomy Teaming : Definitions, Debates, and Directions , Frontiers in Psychology 12 ( 2021 ). doi:10/gkpvf9.

[47] J. C. Walliser , E. J. de Visser, E. Wiese, T. H. Shaw , Team Structure and Team Building Improve Human-Machine Teaming With Autonomous Agents , Journal of Cognitive Engineering and Decision Making 13 ( 2019 ) 258 - 278 . doi: 10 .1177/1555343419867563.

[48] G. Tokadlı , M. C. Dorneich , Autonomy as a Teammate: Evaluation of Teammate-Likeness, Journal of Cognitive Engineering and Decision Making ( 2022 ) 15553434221108002 . doi: 10 . 1177/15553434221108002.

[49] J. C. Gorman , T. A. Dunbar , D. Grimm , C. L. Gipson , Understanding and Modeling Teams As Dynamical Systems, Frontiers in Psychology 8 ( 2017 ) 1053 . doi:10/gh445p.

[50] R. Parasuraman , C. A. Miller , Trust and etiquette in high-criticality automated systems , Commun. ACM 47 ( 2004 ) 51 - 55 . doi:10/djndvp.

[51] E. K. Chiou , J. D. Lee , Trusting Automation: Designing for Responsivity and Resilience, Human Factors: The Journal of the Human Factors and Ergonomics Society ( 2021 ) 001872082110099 . doi:10/gjvcr2.

[52] J. D. Lee , K. A. See , Trust in Automation: Designing for Appropriate Reliance , Hum Factors 46 ( 2004 ) 50 - 80 . doi:10/dr6jf9.

[53] S. Zieba , P. Polet , F. Vanderhaegen , S. Debernard , Principles of adjustable autonomy: A framework for resilient human-machine cooperation , Cogn Tech Work 12 ( 2010 ) 193 - 203 . doi:10/b7839z.

[54] M. C. Cohen , M. A. Peel , M. J. Scalia , M. M. Willett , E. K. Chiou , J. C. Gorman , N. J. Cooke , Anthropomorphism Moderates Relationships of Dispositional, Perceptual, and Behavioral Trust in a Robot Teammate, in: Proceedings of the Human Factors and Ergonomics Society Annual Meeting , in press.

[55] S. C. Kohn , E. J. de Visser, E. Wiese, Y.-C. Lee , T. H. Shaw , Measurement of Trust in Automation: A Narrative Review and Reference Guide , Frontiers in Psychology 12 ( 2021 ).

[56] J.-Y. Jian , A. M. Bisantz , C. G. Drury , Foundations for an Empirically Determined Scale of Trust in Automated Systems , International Journal of Cognitive Ergonomics 4 ( 2000 ) 53 - 71 . doi:10/fdt84r.

[57] E. T. Chancey , J. P. Bliss , Y. Yamani , H. A. H. Handley , Trust and the Compliance-Reliance Paradigm: The Efects of Risk, Error Bias, and Reliability on Trust and Dependence , Hum Factors 59 ( 2017 ) 333 - 345 . doi:10/ggkkfn.

[58] J. Meyer , J. D. Lee , Trust, reliance, and compliance, in: J. D. Lee , A. Kirlik (Eds.), The Oxford Handbook of Cognitive Engineering , Oxford Library of Psychology, Oxford University Press, New York, NY, US, 2013 , pp. 109 - 124 .

[59] H. D. Graham , M. C. Cohen , J. Hrabovsky , D. Orth , N. J. Cooke , Team and Individual Trust Progression for Human-Autonomy Teaming in Next Generation Combat Vehicles , in: Proceedings of the Human Factors and Ergonomics Society Annual Meeting , volume 66 , SAGE Publications Inc, 2022 , pp. 157 - 161 . doi: 10 .1177/1071181322661054.

[60] A. C. Costa , R. A. Roe , T. Taillieu, Trust within teams: The relation with performance efectiveness , European Journal of Work and Organizational Psychology 10 ( 2001 ) 225 - 244 . doi: 10 .1080/13594320143000654.

[61] W. R. Shadish , T. D. Cook , D. T. Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference , Houghton Miflin , Boston, 2001 .

[62] J. Landfair , T. Nguyen , C. Magaldino , P. G. Amazeen , L. Huang , M. Demir , Dynamic Modeling of Trust in Automation in Human-Autonomy Teaming , Proceedings of the Human Factors and Ergonomics Society Annual Meeting 66 ( 2022 ) 1174 - 1178 . doi:10/ gsdzf6.

[63] L. Huang , N. J. Cooke , R. Gutzwiller , S. Berman , E. Chiou, M. Demir , W. Zhang, Distributed dynamic team trust in human, artificial intelligence, and robot teaming , in: Trust in Human-Robot Interaction , 1 ed., Academic Press, 2020 , pp. 301 - 319 . doi: 10 .1016/ B978-0 -12-819472-0 . 00013 - 7 .

[64] M. J. Scalia , J. L. Harrison , S. Zhou , D. A. Grimm , J. C. Gorman , Interaction with an Autonomous Team Member Determines the Relationship between Team Trust and Team Performance , in: 2022 IEEE 3rd International Conference on Human-Machine Systems (ICHMS) , 2022 , pp. 1 - 4 . doi: 10 .1109/ICHMS56717. 2022 . 9980616 .

[65] S. W. J. Kozlowski , K. J. Klein , A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes , in: Multilevel Theory, Research, and Methods in Organizations: Foundations, Extensions, and New Directions, Jossey-Bass/Wiley, Hoboken, NJ, US, 2000 , pp. 3 - 90 .