This paper presents the idea of mutually adaptive trust calibration in Human-AI Teams (HATs). Mutually adaptive trust calibration in HATs is established when both the human agent and the machine agents continuously adapt to one another, in terms of beliefs, attitudes and behaviors, to optimize trust and team performance. This goal requires new concepts, definitions, models, and measures. We highlight our past and recent studies that advance this important objective.
Human
Human understands machine (human-readable algorithms)
2.1. Trust Calibration Definitions
Mutual trust is a fundamental property and predictor of good performing teams. We define trust as “the continuous process of setting and updating a discrete interaction policy towards another agent in risky situations.” Some people trust AI a lot which can lead to over-trusting, leading to misuse and potentially disastrous outcomes. Others distrust AI which can cause undertrust, leading to disuse and unnecessary additional workload [3]. For instance, people tend to apply broad heuristics to trusting other agents such as the system-wide strategy, which occurs when one faulty system “bleeds over” negatively into the perception of similar, but well performing systems [4]. Ideally, people have calibrated trust, an optimized state when the perceived trust matches actual machine trustworthiness (see Figure 2). Early on, we identified and studied teams composed of people and unmanned vehicles and found that trust increased with experience even though the robot made many mistakes [5]. Over-trust can be adjusted downwards by dampening expectations and under-trust can be adjusted upwards through trust repair strategies [6,7].
ss Over-trust à misuse e n i h t r o Dampening w t s rTu libra IA Ca d e v i e c r e P
st
Tru ted
Repair Under-trust à disuse t s u r t e r o m t s u r t s i d e r o m
Actual AI Trustworthiness 2.2. A Model for Mutually Adaptive Trust Calibration
Recently, we have established a model for mutually adaptive trust calibration in HATs [8]. This work presents a new model explaining the role and process of establishing mutually adaptive longitudinal social trust calibration throughout the life cycle of a HAT (see Figure 3). The HAT Trust Model describes the development and role of trust calibration in HAT collaboration. HAT consists of four parts including 1) Relationship Equity, 2) Social Collaborative Processes, 3) Perceptions of Team Partner, and 4) Perceptions of Self. Central to our model is the idea of relationship equity which describes the cumulative result of the cost and benefit relationship acts that are exchanged during repeated collaborative experiences (including social and/or emotional interactions) between two actors. The middle part of the model describes the collaborative task performance between the teammates. Together, they perform a joint activity with the purpose of achieving a common goal. Collaboration is risky: actions may fail and circumstances may change. Therefore, the individual actors monitor the behavior and collaboration of themselves and their teammates. Based on their observations, actors aim to establish appropriate trust stances towards one another (A in B and B in A) to mitigate the potential risks involved in accomplishing the joint task. One part of the model includes passive trust calibration process: Based on team members’ perceptions of one another, actors predict one another’s trustworthiness. Taking into account their current formal work agreements and informal way of collaboration, they then (sub)consciously assess the risk involved in the collaboration as it currently is, and decide upon a trust stance towards one another. Another process of the model is the active trust calibration process: This process is based on an actor’s awareness concerning their involvement in team trust calibration. This awareness enables both actors to engage in deliberate attempts to influence, aid, or hampeInrtertnahtieonatlrJouursntal ocfaSolciibal Rroabtotiiocsn process. 3. Mealimsiutatrioens, mcooerdninattinogfpaTralrleluacstivtitiiens, oHrcoummmunaicant--Acuontdoitionn oteammmyemTbeersainmto scomplacent states and mising information relevant to the team [20,21,84,92,150]. use, whereas undertrust can cause inefficient monitoring and To valida2t.e3 mPerocdepetlisonosfomfTueatmuaPlalrytnaedr(aGpretyiv)e trust calibrauntiboalnan,cietdiwsoerkslosaedn.Itnioatlhetrowdoredsv, etrulostpcaalibnradtioen mis pcriur-ically assess trust building, development and repair inloops mdesacrnib-eaduintothneopmerofdoyerml, athneceH.mRTsThr.otruugsht pthroecefesesdlbeaacdikns this ciahl ufor optimal team m tea Essential endeavorThiesbgluoeo-gdreymbeoxaessuinrdeicsatoefthterupasstsivaentrdusttrcualsibtrwatioornthitnoecsosnti[n9uo-u1s2i]n.crRemeecnetanltulpyd,atweseofhtahve eteacmatmaelmobgeurse’d the process: Based on team members’ perceptions of one another, state of tahcteors predict one another’s trustwortheinnetss. Taking into wetruhstasvtaencems taowpaprdesdonethaniosthteoranMd aanyoeverr’asll orerduigctiionnaolf trust art of trust measurem [13] and model [1a4cc]o.uWnt ethedirivcuidrreentmfoermaasluwroersk oagfretermuesntts ianntdoinsfoerl-f-rempisocarltib,rbateiohnsa.vWieoarssaulmae nthdat,pfohr ytesaimomloegmibcearslthmat are benevolent and sincere, the development of appropriate treuastsures and showmaslowmayeofecxolalamborpatlieons, othfeyhthoewn(stuhb)ecsonescmiouesalysaussreesss arsetanucsesewdillibnentehfietthneier xcotllsabeocrattiiovenesff.orts; team members tdheecirdisekupinovnoalvterudsitnsttahnececotollwabaordrastoionne aasnoitthceurr[r1e6n,t9ly0]i.sT,ahnedy can compensate for each others’ flaws, while relying on each 3.1. TthernumsatyindegcidtehtoeaMdjuost rthaeilr Jcuolldabgormatioenntotsmiotigfaate GPotThe-rsE’ sntraenbgtlhes.d Robot
the assessed risks, for example by proposing formal work
We exaDgpurrelieonmgreetnhdtes noherxobtwycorelallaabxominragativtcheheoiencxcieasstiinoagng, ewthnoertkacattgrorareseimnobeetnadtisn. to 2r.4esPperocnepdtiotnos omfSoerlfa(lYeqllouwe)ries is perceived and
trusted bayddhitiuonmal ainnfoqrmuaetiosntcioonnceerrnisng[t1he5ir]t.ePamamrteimcbiper’as ntrutsst-werTehetyaesllkowedbowxesitihndqicuateerthyeiancgtivtehtreushtucamlibaranti-olnikpreo-agent
with thewgoortahilneosfs.fTihgiusirnifnorgmaotiuont mwayhdeetvhiaeterfrtohmethaegoerigninta,lprecsesesn:Ttheidspraosceass his ubamsedaonnliakneactroor’sbaowtaroenresas cwonecebrnc-lient,
was morparelAdliydcetiqocunoa,tmerelyspuclaetilnitbgeranintetad ptarrunesdtdiscttaiconnoceeusrralodmr,oonbrgemthiestcrtaeulaimbsrtametiedomn..- Painrnetgsisctheienpiarabilnnevstobslvoetrhmaaetcntetoidrns tteotahemnegtarmugset oicnraldaiebllriabcteiooranmt.eTaphttieesmtaewptnasrcteo-e and
perceivebdersmleoardatloitmyoroe fefbfeocttivhe caoglleabnotrastioans: Ohviegrthrustyceatn fo uinnfludenicte, laaidc,korinhagmpberetchaeutrussetcailtibrcaotiounldpronceosst. Tphrisovide
justifications for its moral judgments. While both agents were also rated highly on
trustworthiness, participants had little intention to rely on such an agent in the future1.23This work
presents an important evaluation of a morally competent algorithm integrated with a human-like
platform that could advance how moral robot advisors are trusted in the future [16-18].
perfectly reliable agents (#2–4) than with the unreliable agent (p < .05, Agent Number as a within-subjects variable (Fig. 6). The analysis
prop < .01, and p < .001 respectively). Furthermore, in the informed con- duced a significant main effect for Agent Number (F(
Agent Number) x 2 (In-for all aAgennottsheinr vtahreiaubnleinofofrimnteedrecstonwdaistiovnersificaactrioosns rbeleihaabviliiotyr. dViedrinficaottion A significant 3-way interaction occurred between Agent Number,
with agent number as adifferbs3eihg.an2vificio.arnStwlyya,ssdtFe(efin3e,md23-7bw)y=ai0dp.5aer0t,icpTip=arn.u6t’8ss,dηte2cE=isif0of.n0e1t o.crtesvieiwnthHeiumamge aInnfor-mAatuiontoannd oRemliabyilitTy,eF(a3, m47s1) = 25.60, p < .001, η2 = 0.14).
a significant main effect Imparigoer vteorificmaatikoinng raesdpeocnissieontimaebopurtovthideedcoarnreicntdniecsastioofn tohfethAeTRde.-Two Follow-on analyses showed that, in the informed condition, there was a
, η2 = 0.33, as well as agree mofesacsruurteisnyofgivveernifictaotiiomnagebsehwahveionrrewveiereweudse.dVeirnifictahtiisonsturdeys.poTnhsee first main effect for Agent Number such that the unreliable agent was trusted
2
rmation Condition (F(1,time Twmaeosasadunraeelsymztheode wnraisttehtraata4wt(AheigcethnhtpeNarutemicfibpfeearn)ctxst2ievl(eIecntfneodremtsoastrieoovnifeCwoinnidmtietaigorenvs) e(i.ne.tmioucnhslesos nthatnrtuhesotthcear laigbenrtas,tFi(o3,n23w4) e= 6p2r.0o0v,pi<d e.00a1n,η e=x0a.m44.ple
lso a main effect for thex 2 (Rveerliiaficbatiliiotny) AraNteO)V.TAhweistehcoAngdenmteNausumrebewraassthaewtiitmheins-psuebnjtercetvsievwariin-g an There was also an interaction between Agent Number and Reliability
.001, η2 = 0.21). Theseaobflea(imFriageg.e5c)oe.nnTchetea UapnaAartliyVcsiipssapnurtophdauedcreeldveciatsesodigtrnoiyficdaoncstoo(nmi.eta.riinmoealfgfseectrteufvodierwyAgrteeshnptoantsecsoucuhnthtaet rtreudst waasblioawserkfonr othwepnerfaecstlythreeliasbylesatgeemnts-iwntihdee70t%rust
interactions, and theseNeuffmebtciemrte()F[.(43V,]e4r.i4ficT4at)hio=ins18bb.1e0hia,avpsi<ori.sw00at1sh,aηnea=lytze0e.nd09wd)ieathnndaca4yls(oAfgaoemnrtaNionupemfefbeercrta)xto2 rresliatboilitay pcopndliytiotnrcuomsptarbedrtoo atrdusltyforrtahetpheerfrecttlyhraelniabelexagheinbtsiitning
2
e first significant inter-for In(fIonrfmoramtiaotnioCnoCnodnitdioitnio(nF)(1x,21(4R8e)li=ab4i6li.t8y0),ApN<OV.0A01w,iηth2 =Ag0e.n2t3N).uTmhbeer as the 100% reliability condition, F(
August 9, 2018 Time: 9:14 # 8 3.3. Neural Correlates of Automated Agents
We also investigated the neural underpinnings and mechanisms of trust in automated agents [21]. We used two event-related potentials measured by electroencephalography as an indicator de Visser et al. Neural Measures of Trust in Automation of trustworthiness (see Figure 5). We demonstrated that this marker could distinguish between high and low reliability.
New efforts are under way to realize the vision of mutually adaptive trust calibrating HATs. New developments in generative AI will enable novel types of communication that can allow us to improve human-AI teaming, which is not as good as human-human teaming [22]. We are also investigating how trust calibration efforts trade-off with situation awareness and workload to improve teaming performance. Other efforts are focusing on quantifying the relationship equity construct that some have found useful for the prediction of cross-task and long-term AI teaming [23]. Furthermore, recent reviews have found trust repair, dampening and explanation efforts are not always effective and determined there is a need to develop better, predictive models to enhance such interventions [24, 25]. Lastly, we are examining how AI teammates can align with human decision-makers in terms of values and decision styles, which focuses on the integrity (process) and benevolence (purpose/intent) aspects of trustworthiness (as opposed to the ability / performance dimension of trustworthiness). Combined, these efforts may help to further advance and improve trust calibration in human-AI teams. [13] Kohn, S. C., de Visser, E. J., Wiese, E., Lee, Y. C., & Shaw, T. H. (2021). Measurement of Trust in
Automation: A Narrative Review and Reference Guide. Frontiers in psychology, 12: 604977.
[14] Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995). An integrative model of organizational
trust. Academy of management review, 20(
CRC Press. [21] de Visser, E.J., Beatty, P., Estepp, J.R., Kohn, S., Abubshait, A., Fedota, J.R. & McDonald, C.G. (2018). Learning from the slips of others: Neural correlates of trust in automated agents.
Frontiers in Neuroscience, 12, 309. [22] Walliser, J. C., de Visser, E. J., Wiese, E., & Shaw, T. H. (2019). Team Structure and Team Building Improve Human–Machine Teaming With Autonomous Agents. Journal of Cognitive Engineering and Decision Making, 1555343419867563. [23] Sharp, W. H., Jackson, K. M., & Shaw, T. H. (2023). The frequency of positive and negative interactions influences relationship equity and trust in automation. Applied Ergonomics, 108, 103961 [24] Esterwood, C., & Robert, L. P. (2022, August). A literature review of trust repair in hri. In 2022 31st IEEE International Conference on Robot and Human Interactive Communication (ROMAN) (pp. 1641-1646). IEEE [25] Pak, R., & Rovira, E. (2023). A Theoretical Model to Explain Mixed Effects of Trust Repair Strategies in Autonomous Systems