With collaboration happening on an ever-larger scale, problems arise. Automation is a possible technology that could help make large-scale collaborations more eficient. We interest ourselves in how automation integration can be done without introducing additional troubles and propose a research protocol to investigate the efects of automation on collaboration with regard to trust. Along with technological and communication advances, collaborative endeavors are organized at an ever-larger scale, be it by the number of collaborators, geographical scope, or complexity of the projects undertaken. However, with this new scale, new challenges emerge, leading to an increased cognitive load for collaborators and, consequently, the possibility of accidents.
Collaboration is described as large-scale when hundreds of people or more are working toward the same goal. The following events are examples of large-scale collaboration: • The organization of the Olympics, from the construction of the facilities and volunteer recruitment to the broadcasting of all events worldwide. Last year in Tokyo, it involved 11,656 athletes and 79,000 support staf [ 1] of 206 nations for a total budget of at least 15.4 billion dollars [2]. • The release of a Marvel movie, such as Avengers: Endgame (2019), which on the span of a 20 months production schedule, involved more than 4 thousand credited cast and crew from various trades, with 30 separate companies contracted for visual efects only [ 3]. • Natural disaster management, like the American wildfires of 2020, where 59 thousand wildfires were reported across all 50 states and led to the evacuation of hundreds of thousands of people coordinated by 76 federal and state agencies [4].
Large-scale collaborations present unique challenges caused by their scope [5, 6]: 1. Collaborators, more often than not, do not know each other. They have no existing familiarity or trust between them, and they have to rely instead on organizational trust [7]. 2. Collaborators fill diferent positions and, as a result, do not have the same expertise or vocabularies, making misunderstandings more likely. 3. Collaborators might often belong to diferent organizations that do not share the same work processes and administrative procedures. This increases the number of things to keep track of for collaborators and raises the risk of mistakes [8]. 4. Coordination costs get higher: more people have to be kept informed of changes and coordinated with to prevent the same task from being done twice, in the wrong order, or while impeding on other tasks’ resources. 5. Collaborators might not have the same level of clearance and be allowed to share all pieces of information they have with each other. For example, emergency services coordinating evacuations during a natural disaster should not communicate the same information to the power company than they would to the press. 6. Collaborators often come from diferent countries and, consequently, diferent time zones, resulting in interactions distributed in time and space, making coordination harder by producing delays and making direct communication challenging. 7. As a result of their distribution across countries, collaborators also do not always have a language in common, forcing them to rely on translations that may not be as faithful.
We advance that automating some of the tasks done by people during large-scale collaboration with an agent could make collaborations simpler, faster, and less error-prone.
First, agents can automate some of the more menial tasks in a collaborative endeavor. For example, in Wikipedia, syntax and format checking are automated by bots [9]. This type of automation leaves more time and thinking space for collaborators to concentrate on more critical tasks.
Secondly, automation can also be used to reduce coordination costs directly. For example, a cognitive agent can keep information circulating while people are working in diferent time zones and reduce delays (in the same way that chatbots are helpful for customer service). Additionally, they can keep track of all up-to-date information necessary without forgetting to keep all and only relevant personnel in the loop. Finally, having people checking in with an assistant would also reduce confusion by encouraging collaborators to use standardized vocabulary to refer to the same work processes.
Before implementing an automated tool designed for large-scale collaboration, we need to ensure that its anticipated users trust it. Indeed, when implementing a tool, even if it is working perfectly, it will only be used as envisioned and eficiently if its users trust it. Users need to trust the information it communicates, its validity, its conformity to the organization, and its consistency before using it and making the decision of propagating it to others. Moreover, a lack of trust can hinder the tool’s goal of improving performances or even go against it. Indeed, a breakdown in trust would make people reluctant to relay the tool’s information, or worse, not trust the information and double-check it by using secondary channels, resulting in increased coordination costs and collaborator’s cognitive load compared to a situation where no tool was implemented. This increase in redundant communications can lead to conflicting information between people making decisions and allow for more miscommunications, confusion, and errors [10, 11].
According to the literature, factors that make an automated tool more trustworthy for its users and make them more willing to rely on its actions and information, aside from its performances, are [12, 13, 14]: • the ability to negotiate and influence its decision ; • clarity on what data and reasoning are used to come to its decision; • transparency on its limitations; • fluid transition between full automation and human control.
Most studies regarding trust in automated tools for collaboration tend to look at tools designed for specific collaborative tasks, like patient triage or artistic ideation, and rarely look at collaboration that is not between an automated assistant and a single person. Few studies have been done that tackle the problems inherent to large-scale collaboration. Our research intends to fill this gap and address the research questions that arise from the implementation of automation in large-scale collaboration: • What is the preferred communication configuration (separated private channels or group chat, broadcast of all new information or information only given on request) with an automated assistant? • Which level of assistant initiative makes users more comfortable and does not over-solicit them? • In which conditions are users willing to share their information with the assistant? • Do users rely on secondary communication to verify the assistant’s information?
• Does greater trust in the assistant improve performances?
To answer those questions, we developed an experimental platform (as depicted in Figure 1) to investigate those questions. Groups of participants are tasked with solving simple modules by coordinating with others and with the help of an automated assistant. The platform allows users to either communicate in a group chat or private discussion with or without the automated assistant, as specified by the experimenter before the start of the experiment. This experiment aims to test diferent collaboration configurations (communication channels available, assistant capacities, dificulty of modules’ resolution) and analyze the participants’ performances, interactions, and experiences quantitatively and qualitatively.
Automation is a promising but still underused tool for large-scale collaboration’s challenges. One of its main challenges is making its integration trustworthy for potential users. We describe an experimental protocol that aims to resolve some of the unknown of group interaction with automation during a collaborative task and determine the best approach for designing an automated assistant. [1] J. McCurry, 79,000 people flying in for tokyo olympics, japanese media reports, 2021. URL: https://www.theguardian.com/sport/2021/may/20/organisers-of-tokyo-olympics-pressahead-despite-covid-fears. [2] A. Cervantes, The tokyo olympics’ staggering price tag and where it stands in history, 2021. URL: https://www.wsj.com/articles/the-tokyo-olympics-staggering-price-tag-andwhere-it-stands-in-history-11627049612. [3] Avengers: Endgame (2019) - full cast crew, 2021. URL: https://www.imdb.com/title/ tt4154796/fullcredits/. [4] N. I. C. Center, National interagency coordination center - wildland fire summary and statistics annual report 2020, 2021. [5] A. W. Eide, I. M. Haugstveit, R. Halvorsrud, J. H. Skjetne, M. Stiso, Key challenges in multiagency collaboration during large-scale emergency management, in: AmI for crisis management, international joint conference on ambient intelligence, Pisa, Italy, 2012. [6] S. Morrison-Smith, J. Ruiz, Challenges and barriers in virtual teams: a literature review,
SN Applied Sciences 2 (2020) 1–33. [7] F. D. Schoorman, R. C. Mayer, J. H. Davis, An integrative model of organizational trust:
Past, present, and future, 2007. [8] G. L. Kolfschoten, F. M. Brazier, Cognitive load in collaboration: Convergence, Group
Decision and Negotiation 22 (2013) 975–996. [9] L. Zheng, C. M. Albano, N. M. Vora, F. Mai, J. V. Nickerson, The roles bots play in wikipedia,
Proceedings of the ACM on Human-Computer Interaction 3 (2019) 1–20. [10] D. A. Winsor, Communication failures contributing to the challenger accident: An example for technical communicators, IEEE transactions on professional communication 31 (1988) 101–107. [11] R. Parasuraman, V. Riley, Humans and automation: Use, misuse, disuse, abuse, Human factors 39 (1997) 230–253. [12] S. D. Ramchurn, F. Wu, W. Jiang, J. E. Fischer, S. Reece, S. Roberts, T. Rodden, C. Greenhalgh, N. R. Jennings, Human–agent collaboration for disaster response, Autonomous Agents and Multi-Agent Systems 30 (2016) 82–111. [13] M. Wirz, D. Roggen, G. Troster, User acceptance study of a mobile system for assistance during emergency situations at large-scale events, in: 2010 3rd International Conference on Human-Centric Computing, IEEE, 2010, pp. 1–6. [14] B. Nesset, D. A. Robb, J. Lopes, H. Hastie, Transparency in hri: Trust and decision making in the face of robot errors, in: Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction, 2021, pp. 313–317.