Contemporary automated systems rely mainly on conventional audiovisual feedback technologies based on 2D displays and audio. Rapid technology advances - especially but not restricted to edge computing, augmented and virtual reality, and adaptive interfaces - can enable mediated reality systems that will revolutionize interactions between humans and automated systems. In this position paper, we outline the potential of mediated reality to create a “good” automation experience. Specifically, we outline two potential scenarios facilitating mediated reality during human-automation collaboration. Guided by these examples, we highlight the potential benefits of using mediated reality, outline future research directions to achieve them, and specify the open challenges that are likely to be encountered. The paper argues for the viability of mediated reality and raises the need to align community eforts in realizing the full benefits of human-automation interactions.
Rapid developments and deployment of ubiquitous
computing technologies — such as edge computing, machine
learning, artificial intelligence, and augmented and
virtual reality — have acquainted both novices and experts
with automated systems in their everyday lives.
Arguably, the main design challenge faced is in creating
systems that can be used by experts and non-experts
pectations across diferent domains such as automated
vehicles [
CEUR human-robot workspaces [12]).
tional systems. We explore mediated reality as an adaptive tool that can enhance the automation experience for experts and novices. Mediated reality describes a perceived reality that is augmented, enhanced, deliberately diminished, or otherwise altered in real-time [13, 14]. With that, it goes beyond AR, which “only” adds virtual ing and plausible alteration of the environment by mediated reality still seems visionary, approaches comprising augmented reality, diminished reality (DR), and a combination of both already provide a glimpse of what might be possible in the future. For example, AR applications are already used to add digital content to the real world using head-mounted displays [15], and smartphones [16, 17].
time by reconstructing the background [18].
When combined, we believe that such technologies can enhance the automation experience by providing a form of human engagement in human-automation interaction that is yet to be explored. In the following, we outline two examples of how mediated reality can contribute to this domain and describe essential research directions necessary to fulfil this vision. 2.
benefits of mediated reality. We outline one example in the private context for non-professionals (Section 2.1) and © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License one in an industrial setting for specialists (Section 2.2).
Attribution 4.0 International (CC BY 4.0) (a) User’s view of an obstructed crossing (b) Processing steps of the system (c) Mediated view with removed parts of a building (DR) and extra UI elements (AR) 2.1. Automated Vehicles Driving automation has received much attention in recent years, especially with the release of consumer vehicles equipped with automated driving features. This domain highlights the importance of a good (driving) automation experience, as the human has to work closely with the automation system (and the car) to use the system safely. For example, in SAE Level 3 [19], the automation system can request the driver to take over. This is no longer the case in higher levels, such as SAE Level 4 Figure 2: Mediated view of a human that works together [19]. Regardless, a good experience with the automation with an “intelligent” robot. The robot is in motion, and a system requires the driver to trust the system, which can vmiratcuhailnheaanrde apuotionmtsaattictahlelyphriodbdleenmf.oPraarctsleoafr tvhieewro(bhoigthalnigdhtthede be done by communicating information about internal with red borders), and important information is provided (e.g., and external states (e.g., Oliveira et al. [20]). the vital turned-of safety switch).
A mediated reality system ofers potential solutions to
this challenge. In 1a, a vehicle approaches an obstructed
crossing. This could be dangerous in case of a take-over
request or make the driver uneasy because they must in situ recommendations [22]). These can be extended
trust the automation system. A mediated reality system by applying a real-time mediated reality approach to
encould use techniques such as environment reconstruc- hance collaboration in human-robot teams further.
Prition, tracking, and registration — all parts of Augmented marily where robots and operators work together on a
and Diminished Reality — to provide a mediated view of task, having a shared goal, and contact is possible (or
reality. The system could also estimate the user’s state, even desired), a mediated view can enhance team
perforthe overall context, and the (shared) goals to provide the mance. For example, if both work together on the same
best level of information. Such a view could be similar part during an assembly or maintenance task.
to the one in 1c where the crossing is not obstructed In such a task and environment, AR has been shown
anymore, and UI elements such as stop signs, lines, and to support human-robot teams by, for example, adding
warning signs support the driver in building situation contextual information such as task instructions, future
awareness or communicate the system’s understanding. robot trajectories, visual cues for essential components,
and more [
An accurate model of the environment must include How can we encourage sustained interaction with autoinformation about geometry, lighting, and materials for mated systems? An automation system that communia realistic modification of reality. Without it, the mod- cates with a human operator via a mediated reality has ified reality would be perceived as unconvincing, aes- not only the means to prompt explicit requests from the thetically unpleasant, implausible, or eerie — similar to user but also to nudge the user towards collaborative an augmented reality where an object is not correctly interactions. For example, this can happen by modifyembedded into the scene. In addition, the system must ing perceived reality to choose a preferable action over understand the human operator — their preferences and a viable but worse alternative. In addition, it can also requirements — so that the automation system can col- highlight the benefits of usage by mediating reality. For laborate with the user by mediating reality without being example, in a home automation system paired with an intrusive or annoying. For this, it needs to know the state AR-HMD, the automation system could mediate reality of the human operator to make assumptions about his so that it is the most natural thing for a human operator current skill level, mental load, situational awareness, and to interact with the system and, by that, sustain interacmore. It could achieve this by making inferences from tion beyond the initial excitement phase. This could be the human operator’s explicit behaviour (e.g., speech done, for example, by representing a reality that anticirequests) and implicit behaviour (e.g., gaze, posture). Au- pates the positive consequences of interacting with the tomated systems might even have access to information automation system. Here, the system can adapt if the that fellow human operators do not (e.g., psychophysi- interaction patterns become natural and habitual. ological activity, statistics of competence levels), which How to design for the overall reliability of humancould support its inference of whether or not support is automation teams? In addition to traditional 2D displays required. Finally, an automation system needs to know and plain augmented reality, mediated reality has other about the (shared) goals, tasks, and intermediate steps means to support reliable cooperation in teams of huto truly support the human operator during a task, be man(s) and automation system(s). Here, mediated reality it driving or a workflow within the industrial context. can work towards a consistent team performance by its To reiterate, this is necessary to make the mediation of unique means of adding information and making usually reality as natural, efective, helpful, and non-disturbing hidden structures and entities visible to the human opas possible. This can be achieved by preemptively provid- erator. Humans can then integrate this knowledge into ing necessary information or removing parts of reality their planning and action processes and perform better to make the following tasks easier to perform. While in the human-automation team. This means, if designed many of these variables can be predetermined for some carefully, a mediated reality approach promises to make particular scenarios, a mediated reality system benefits the interaction between the human and the automation most from real-time assessment of the environment, the experience more transparent and, at best, leads to comoperator, the goals, and tasks. This way, a real-time munication between the two that is at least as efective modification that is situation-dependent and respects the as the one between two human team members. given environment and human states can happen. How to support the development of trust in these assem
Following the workshop’s theme, we now outline how blies? Especially complex automation systems can use mediated reality can improve human engagement and mediated reality to support the development of trust and the experience of interaction with an automation system manage and maintain it. For example, a home automain the following paragraphs on particular challenges. In tion system can use an “x-ray” view to display the result particular, we highlight the potential of mediated reality of its actions in reality in real-time, even when hidden, and motivate future research challenges. to the human operator. Such an automation system that facilitates mediated reality techniques can also implicitly for an easy design and development of mediated reality communicate its understanding of the world (as seen in prototypes (e.g., similar to ARCore[25] and ARKit[26]). Figure 1) and potentially its insecurities in a collabora- 6. Qualitative and quantitative studies that assess users’ tive task. In addition, such a system can also add and reactions to mediated reality in a human-automation highlight audiovisual content in real-time. For example, team need to be performed. in a technical context, this could mean that the system 7. In this paper, we mainly discussed visual mediated shows the human operator how an internal mechanism reality. However, work on spatial audio [27], auditory ilof a complex workpiece works (“x-ray”) or is influenced lusions [28] and also haptics [29], and smell [30] promise by an action of the team and asks the human operator if future extensions of this genuinely multi-sensory vision. this understanding is correct. Given that this informa- Of course, any mediated reality-based approach intion on what the automation system knows and what it volves significant ethical issues. This includes long-term doesn’t know helps when building trust [23, 24], a medi- consequences and aspects like how the human reacts to ated reality approach ofers additional channels for this, or after sustained interaction with a modified reality and compared to traditional AR or 2D displays. if there is any efect on the experience of “real reality”
How to ensure and support the well-being of human after such a sustained interaction. It is also essential to operators? For this specific challenge, mediated reality ensure that no user in a human-automation team that rehas the potential to hint towards dangers or dangerous lies on mediated reality is unaware that they are facing a situations by a) highlighting them and b) making them mediated environment. While this is not a problem with visible or c) a combination of both. Such a procedure is today’s displays, it is relevant to consider these aspects shown in Section 2.1 where the automation system makes early on. An overview is provided by [31, p. 239]. the danger visible, annotates it, and by that, ensures the well-being of the human operator. Mediated reality, in general, can be used in human-automation teams to 5. Conclusion & Outlook communicate intention, short-/medium-/ and long-term goals, processing steps, and available information about the automation systems’ state and context.
diated reality for increasing the experience in humanautomation teams. Focusing on visual alterations, including augmenting and diminishing reality, our examples 4. Research challenges illustrate how mediated reality can improve the human automation experience. Following this, we described The contributions of mediated reality to an enhanced au- several areas where mediated reality can contribute and tomation experience naturally revolve around the techno- present these contributions (takeover requests, sustained logical means to augment or deliberately diminish reality. interaction, overall reliability, trust, and operator’s wellThat is, modifying perceived reality for a human opera- being). While mediated reality technology is arguably tor to ensure that shared goals are achieved efectively, still in its infancy, we showed that the potential benefits eficiently, safely, and satisfactorily. To realize this vision, for human-automation teams of this output modality are the following core challenges need to be addressed: plentiful. Thus, in the future, researchers must work 1. For a beneficial mediated reality, further advance- on soft- and hardware, UI/design patterns, management ments in software (real-time tracking, registration, recon- strategies for a shared context, lab and field studies, and struction) and hardware (tracking and semi-transparent multimodal mediated reality to truly benefit from medisplay) for (audio-)visual displays are necessary. diated reality in human-automation teams. Still, with 2. One of the more ethical challenges is not simply to today’s technology stack, it is already possible to impledesign a mediated environment that presents an altered ment and test prototypes of mediated reality user interreality as realistically and plausibly as possible but to faces, for example, by simulating them in virtual reality. consider that it can also lead to “reality confusion” in We highly encourage doing precisely that, e.g. by testing which the human user is no longer able to determine the above-mentioned use cases and contributions. what is real and what is not - even if he tries. 3. Appropriate design patterns and strategies for dynamically mediating reality are needed so that (potentially Acknowledgments dynamic) modifications are not annoying, distracting, eerie, or inefective. 4. Systems for context tracking and estimation are necessary so that the mediated reality can adapt according to the human state, the shared goals, and the environment. 5. Toolkits/frameworks need to be developed that allow
Teaming between Humans and Production Systems” coifnanced by tax funds of the Saxony State Ministry of Science and Art (SMWK3-7304/35/3-2021/4819) on the basis of the budget passed by the deputies of the Saxony state parliament.
The virtual scene in Figure 2 is generated in Unity with assets of: • Alarm button, • Building, • Electric engine, • Environment map, • Machine + Rotor, • Hand, and • Robot.