=Paper=
{{Paper
|id=Vol-3154/short3
|storemode=property
|title=Trustworthy Automation for Large-Scale Collaboration : a Proposed Exploratory Study
|pdfUrl=https://ceur-ws.org/Vol-3154/short3.pdf
|volume=Vol-3154
|authors=Clélie Amiot,François Charoy,Jérôme Dinet
|dblpUrl=https://dblp.org/rec/conf/chi/AmiotCD22
}}
==Trustworthy Automation for Large-Scale Collaboration : a Proposed Exploratory Study==
https://ceur-ws.org/Vol-3154/short3.pdf
Trustworthy automation for large-scale
collaboration: a proposed exploratory study
Clélie Amiot, François Charoy and Jérôme Dinet
Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France
Abstract
With collaboration happening on an ever-larger scale, problems arise. Automation is a possible technology
that could help make large-scale collaborations more efficient. We interest ourselves in how automation
integration can be done without introducing additional troubles and propose a research protocol to
investigate the effects of automation on collaboration with regard to trust.
Keywords
trust, automated assistant, large-scale collaboration
1. Introduction
Along with technological and communication advances, collaborative endeavors are organized
at an ever-larger scale, be it by the number of collaborators, geographical scope, or complexity
of the projects undertaken. However, with this new scale, new challenges emerge, leading to an
increased cognitive load for collaborators and, consequently, the possibility of accidents.
There is an interest in automating some of the tasks of large-scale collaboration to free
collaborators’ cognitive capacity as well as improve their context-awareness and support their
decision-making. This can be done by using automation to keep track of all critical contextual
information and bringing it to the attention of a collaborator when needed, document changing
procedures and prevent conflicting records, and disseminate all new information to relevant
parties with the appropriate level of visibility.
Nevertheless, to successfully support large-scale collaboration with automation, we need the
automated tools produced to be adequately trustworthy for their users to avoid misuse or disuse.
Our research looks at how a cognitive agent designed for large-scale collaboration automation
can be integrated into a team in a way that maintains adequate trust and does not disrupt the
collaboration.
2. Automation and Large-Scale Collaboration
Collaboration is described as large-scale when hundreds of people or more are working toward
the same goal. The following events are examples of large-scale collaboration:
AutomationXP22: Engaging with Automation, CHI'22, April 30, 2022, New Orleans, LA
Envelope-Open clelie.amiot@loria.fr (C. Amiot); francois.charoy@loria.fr (F. Charoy); jerome.dinet@loria.fr (J. Dinet)
Orcid 0000-0003-4431-4856 (C. Amiot); 0000-0002-0640-6613 (F. Charoy); 0000-0002-9546-8604 (J. Dinet)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
• The organization of the Olympics, from the construction of the facilities and volunteer
recruitment to the broadcasting of all events worldwide. Last year in Tokyo, it involved
11,656 athletes and 79,000 support staff [1] of 206 nations for a total budget of at least
15.4 billion dollars [2].
• The release of a Marvel movie, such as Avengers: Endgame (2019), which on the span of
a 20 months production schedule, involved more than 4 thousand credited cast and crew
from various trades, with 30 separate companies contracted for visual effects only [3].
• Natural disaster management, like the American wildfires of 2020, where 59 thousand
wildfires were reported across all 50 states and led to the evacuation of hundreds of
thousands of people coordinated by 76 federal and state agencies [4].
Large-scale collaborations present unique challenges caused by their scope [5, 6]:
1. Collaborators, more often than not, do not know each other. They have no existing
familiarity or trust between them, and they have to rely instead on organizational trust
[7].
2. Collaborators fill different positions and, as a result, do not have the same expertise or
vocabularies, making misunderstandings more likely.
3. Collaborators might often belong to different organizations that do not share the same
work processes and administrative procedures. This increases the number of things to
keep track of for collaborators and raises the risk of mistakes [8].
4. Coordination costs get higher: more people have to be kept informed of changes and
coordinated with to prevent the same task from being done twice, in the wrong order, or
while impeding on other tasks’ resources.
5. Collaborators might not have the same level of clearance and be allowed to share all pieces
of information they have with each other. For example, emergency services coordinating
evacuations during a natural disaster should not communicate the same information to
the power company than they would to the press.
6. Collaborators often come from different countries and, consequently, different time zones,
resulting in interactions distributed in time and space, making coordination harder by
producing delays and making direct communication challenging.
7. As a result of their distribution across countries, collaborators also do not always have a
language in common, forcing them to rely on translations that may not be as faithful.
We advance that automating some of the tasks done by people during large-scale collaboration
with an agent could make collaborations simpler, faster, and less error-prone.
First, agents can automate some of the more menial tasks in a collaborative endeavor. For
example, in Wikipedia, syntax and format checking are automated by bots [9]. This type of
automation leaves more time and thinking space for collaborators to concentrate on more
critical tasks.
Secondly, automation can also be used to reduce coordination costs directly. For example, a
cognitive agent can keep information circulating while people are working in different time
zones and reduce delays (in the same way that chatbots are helpful for customer service).
Additionally, they can keep track of all up-to-date information necessary without forgetting
to keep all and only relevant personnel in the loop. Finally, having people checking in with
an assistant would also reduce confusion by encouraging collaborators to use standardized
vocabulary to refer to the same work processes.
3. The importance of trustworthiness
Before implementing an automated tool designed for large-scale collaboration, we need to
ensure that its anticipated users trust it. Indeed, when implementing a tool, even if it is working
perfectly, it will only be used as envisioned and efficiently if its users trust it. Users need to
trust the information it communicates, its validity, its conformity to the organization, and its
consistency before using it and making the decision of propagating it to others. Moreover, a
lack of trust can hinder the tool’s goal of improving performances or even go against it. Indeed,
a breakdown in trust would make people reluctant to relay the tool’s information, or worse, not
trust the information and double-check it by using secondary channels, resulting in increased
coordination costs and collaborator’s cognitive load compared to a situation where no tool was
implemented. This increase in redundant communications can lead to conflicting information
between people making decisions and allow for more miscommunications, confusion, and errors
[10, 11].
According to the literature, factors that make an automated tool more trustworthy for its users
and make them more willing to rely on its actions and information, aside from its performances,
are [12, 13, 14]:
• the ability to negotiate and influence its decision ;
• clarity on what data and reasoning are used to come to its decision;
• transparency on its limitations;
• fluid transition between full automation and human control.
4. Future works
Most studies regarding trust in automated tools for collaboration tend to look at tools designed
for specific collaborative tasks, like patient triage or artistic ideation, and rarely look at collabo-
ration that is not between an automated assistant and a single person. Few studies have been
done that tackle the problems inherent to large-scale collaboration. Our research intends to fill
this gap and address the research questions that arise from the implementation of automation
in large-scale collaboration:
• What is the preferred communication configuration (separated private channels or group
chat, broadcast of all new information or information only given on request) with an
automated assistant?
• Which level of assistant initiative makes users more comfortable and does not over-solicit
them?
• In which conditions are users willing to share their information with the assistant?
• Do users rely on secondary communication to verify the assistant’s information?
Figure 1: Proposed interface of the experimental platform
• Does greater trust in the assistant improve performances?
To answer those questions, we developed an experimental platform (as depicted in Figure 1)
to investigate those questions. Groups of participants are tasked with solving simple modules
by coordinating with others and with the help of an automated assistant. The platform allows
users to either communicate in a group chat or private discussion with or without the automated
assistant, as specified by the experimenter before the start of the experiment. This experiment
aims to test different collaboration configurations (communication channels available, assis-
tant capacities, difficulty of modules’ resolution) and analyze the participants’ performances,
interactions, and experiences quantitatively and qualitatively.
5. Conclusion
Automation is a promising but still underused tool for large-scale collaboration’s challenges.
One of its main challenges is making its integration trustworthy for potential users. We describe
an experimental protocol that aims to resolve some of the unknown of group interaction with
automation during a collaborative task and determine the best approach for designing an
automated assistant.
References
[1] J. McCurry, 79,000 people flying in for tokyo olympics, japanese media reports, 2021. URL:
https://www.theguardian.com/sport/2021/may/20/organisers-of-tokyo-olympics-press-
ahead-despite-covid-fears.
[2] A. Cervantes, The tokyo olympics’ staggering price tag and where it stands in history,
2021. URL: https://www.wsj.com/articles/the-tokyo-olympics-staggering-price-tag-and-
where-it-stands-in-history-11627049612.
[3] Avengers: Endgame (2019) - full cast crew, 2021. URL: https://www.imdb.com/title/
tt4154796/fullcredits/.
[4] N. I. C. Center, National interagency coordination center - wildland fire summary and
statistics annual report 2020, 2021.
[5] A. W. Eide, I. M. Haugstveit, R. Halvorsrud, J. H. Skjetne, M. Stiso, Key challenges in
multiagency collaboration during large-scale emergency management, in: AmI for crisis
management, international joint conference on ambient intelligence, Pisa, Italy, 2012.
[6] S. Morrison-Smith, J. Ruiz, Challenges and barriers in virtual teams: a literature review,
SN Applied Sciences 2 (2020) 1–33.
[7] F. D. Schoorman, R. C. Mayer, J. H. Davis, An integrative model of organizational trust:
Past, present, and future, 2007.
[8] G. L. Kolfschoten, F. M. Brazier, Cognitive load in collaboration: Convergence, Group
Decision and Negotiation 22 (2013) 975–996.
[9] L. Zheng, C. M. Albano, N. M. Vora, F. Mai, J. V. Nickerson, The roles bots play in wikipedia,
Proceedings of the ACM on Human-Computer Interaction 3 (2019) 1–20.
[10] D. A. Winsor, Communication failures contributing to the challenger accident: An example
for technical communicators, IEEE transactions on professional communication 31 (1988)
101–107.
[11] R. Parasuraman, V. Riley, Humans and automation: Use, misuse, disuse, abuse, Human
factors 39 (1997) 230–253.
[12] S. D. Ramchurn, F. Wu, W. Jiang, J. E. Fischer, S. Reece, S. Roberts, T. Rodden, C. Greenhalgh,
N. R. Jennings, Human–agent collaboration for disaster response, Autonomous Agents
and Multi-Agent Systems 30 (2016) 82–111.
[13] M. Wirz, D. Roggen, G. Troster, User acceptance study of a mobile system for assistance
during emergency situations at large-scale events, in: 2010 3rd International Conference
on Human-Centric Computing, IEEE, 2010, pp. 1–6.
[14] B. Nesset, D. A. Robb, J. Lopes, H. Hastie, Transparency in hri: Trust and decision making
in the face of robot errors, in: Companion of the 2021 ACM/IEEE International Conference
on Human-Robot Interaction, 2021, pp. 313–317.