Design Dimensions for Holographic Intelligent Agents: A Comparative Analysis

Abstract

Humanoid intelligent agents, or 'Holographic AIs', as we prefer, are trending, promising improved delivery of personalized services on smart glasses and in Augmented Reality. Lacking clarity of the concept and missing recommendations for their features, however, pose a challenge to developers of these novel, embodied agents. In this paper, we therefore conduct a comparative analysis of nine intelligent agents who can interact with both physical and virtual surroundings. We identify, select, and investigate four distinct types of nonplayer game characters, chatbot agents, simulation agents, and intelligent tutors in order to, subsequently, develop a framework of features and affordances for holographic AIs along the axes of appearance, behavior, intelligence, and responsiveness. Through our analysis, we derive preliminary recommendations for developers of Holographic AIs: the use case determines appearance; dialogue management is key; awareness and adaptation are equally important for successful personalization; and environmental responsiveness to events both in the virtual and digital ream is needed for a seamless experience.

Keywords

• Artificial Intelligence
• Human-Computer Interaction
• Virtual Humans
• Intelligent User Interface
• Holographic AIs
• Augmented Reality

-

It may be a controversy that the new predominant skeuomorphism (‘container-shape’) of future Operating Systems for Spatial Computing will be dressed up as intelligent assistants - like Siri, Alexa, or Cortana, just in 3D. There is a compelling argument though that the intermediary between user and the system functionality could mimic social interaction with a human assistant, delivering emotionally charged, reactive conversation that users may prefer over interaction through other user interface (UI) metaphors [ 1 ].

Face-to-face interaction with anthropomorphic conversational agents has a long history in Human-Computer Interaction, but, alas, not necessarily a positive one. Critics find that anthropomorphic agents “hinder rather than enhance productivity” [ 2 ]. Multimodality, including verbal/non-verbal interaction, facial expression, and eye contact (see [ 3 ]), social awareness, autonomy, and responsiveness all play their special role thereby in replacing mechanized input-output. Artificial intelligences (AIs) today are often able to pick up on subliminal cues of their human interaction partners, enabling them to engage in conversation on a level where at times they may even outperform human ability [ 4 ]. Gratch et al. [ 5 ] consequently postulate for the creation of interactive human-like UIs that we need to pay attention to “body animation, facial animations, perception, cognitive modelling, emotions and personality, natural language processing, speech recognition and synthesis, non-verbal communication, distributed simulation, [gamification]”.

Augmented Reality (AR) can be delivered using holographic displays [ 6 ] such as Microsoft’s HoloLens or Magic Leap. Spatial mapping builds up a representation of physical surfaces, so that apps can adapt to room shape/size, and voice, gesture, and gaze interaction allow to observe and accurately control digital information projected onto a user’s view of the real world. Campbell et al. define agents embodied in AR as AuRAs [ 7 ] based on the concept of MiRA, “an agent embodied in a Mixed Reality environment” [ 8 ]. They, however, do not systematically analyze the difference in features of agents in AR. We propose to introduce the new term ‘holographic AI’ instead, as the terms of artificial / embodied / intelligent / conversational / virtual / mixed reality agents are ambiguous for AR and their definitions do not acknowledge the importance of environmental responsiveness as its key feature. To us, a ‘Holographic Artificial Intelligence’ is an anthropomorphic user interface, projected into the real world, which interacts with and responds to both its physical and virtual surroundings. To the user, a Holographic AI is always embedded in the physical environment [ 9 ], mediating between system functionality and the user, and interacting with the world, providing a pleasing and personalized wrapper for service automation.

Within this paper, we seek to contribute to a more unified view of what successful holographic AIs are, to help overcome teething problems for this novel technology that led many contemporary implementations to appear rather unconvincing (including some of our own work in progress). This paper therefore uses the comparative analysis of a selection of holographic AIs for the exploration of the parameters in a reference framework suitable for guiding development of pedagogical holographic AIs.

We do not speculate whether education and training really have a justified demand for holographic AIs. We merely acknowledge that these are an emerging technology. Under this assumption we are mainly interested in how successful holographic AIs implementations work in the context of their concrete application. We explore the design space, identify drivers to their success from the cases we selected and their comparative analysis, and express recommendations for their development.

The rest of this paper is organized as follows. Section 2 describes the methodology applied to select and analyze relevant cases of holographic AIs. In Section 3, we introduce a new typology based on comparisons among the different applications. Section 4 summarizes the findings regarding features. Section 5 adds discussion and recommendations, and with Section 6 we conclude to also outline our future plans. 2

Methodology

Generally speaking, case studies are particularly well suited for exploring phenomena in the context of their application [ 10 ]. Studies using only a single case design, however, are not very useful for generalizing recommendations [ 11 ], such as recommendations about what seem to be major drivers of success or failure, systematically linking conditions and outcomes. For this, Comparative Analysis offers a suitable methodology [ 11 ], which we are applying in this paper. In Comparative Analysis, we compare cases by inspecting their commonalities and differences, which allows us to draw conclusions on what holds across contexts.

For this, we first identify key questions, then initial parameters. Next, we find relevant cases, from which we select the ones to study in-depth, focusing on heterogeneity and avoiding those too similar. Subsequently, we describe key features in the context of their application. Our further analysis allows us then to extract key commonalities and key differences, which we integrate into a framework.

Our guiding research question is: What design features drive the success of holographic AIs and what can consequently be recommended for their development.

To answer it, we collected products that explicitly describe conversational virtual assistants in AR from recent years. To set focus also on recent work and work in progress, we obtained relevant cases by searching pertinent news, blogs, and conducting open searches in YouTube and Google, using keywords such as “holographic virtual agents”, “virtual holographic products/human”, “HoloLens virtual human”, “holohuman”, “holographic partner”, “smart assistant in AR”, “conversation agents in AR”, “HoloLens virtual human”, “holographic system”, “HoloLens AR game”. Additionally, we added two cases from our own work (HoloCare, Wekit / MirageXR). Search results identified were checked, reviewing first titles in the list, then, if relevant, introduction, the full website, including videos, interviews, and, ultimately, the programs. Only products that fitted to our definition of holographic AIs were selected. In total, we identified 16 virtual agents as potential cases, removing – after review – those which are not placed into physical space (and are only available in a virtual world), as well as those who do not actually interact with the environment, who only rather statically occupy display real-estate (‘talking heads’ / avatar on screen). Finally, only nine holographic AIs in total were selected for further investigation. Among the agents investigated, we were able to identify four distinct usage contexts: non-player game character, chatbot agent, simulation agent, and intelligent tutor. While the first two serve as a generic comparison to inform (with applications possible in education or training), the latter two focus on education and training contexts. 3.1

Game Characters

Bowman et al. [ 12 ] define a 3D user interface as “a UI that involves 3D interaction”, in which the user’s tasks can be executed directly in a 3D environment. In AR, game characters appear to interact directly with users in acting out storylines.

AR game characters are able to provide life-size, humanoid avatars with exaggerated animations to improve perception of the characters’ actions. Game characters are controlled by the game AI, and typically narrate the background that users need to advance the storyline, and it requires the users to work with the virtual character in scripting the storylines. The graphics of game characters tend towards realism, highresolution textures, and smooth animation. For example, Young Conker [ 13 ], an AR game for the Microsoft Hololens, recognizes and transforms furniture, floors, and walls into the game stage using spatial mapping so that a main character, a squirrel, runs and jumps around the room (see Fig.1a). In Young Conker, players control the cartoon squirrel’s motion with voice commands and gestures (air-tap). Although the digital animal can blink its eyes and possesses lip-sync, it cannot actually engage players in dialogue. Similarly, in Fragments [ 14 ] (see Fig. 1b), multiple agents navigate the story, react to players’ decisions. The game characters can stand, sit on a real-world sofa or chair, lean on the physical walls of the user’s room, gaze at players, and precisely locate their positions through the spatial mapping. 3.2

Simulation agents

Holographic AIs that simulate real events are useful, for example, in healthcare training. Pearson Immersive (now GIGXR) created the HoloPatient app (Fig. 1c) that uses volumetric videos to present standardized clinical patients to the learner. The program enables multiple students to conduct teamwork [ 15 ]. Although the users focus on patients’ physical and psychological states, they cannot talk with the virtual patients, and extra panels are used for interaction. Therefore, it is hard for a user to empathize with unresponsive, unemotional patients. At the same time the realism is compelling and more authentic, compared to the other animated approaches.

GhostPacer [ 16 ] is a mixed reality fitness training product able to project a holographic training partner via smart glasses into the real world in order to jog with the user (see Fig.1d). The appearance of the partner is a basic human shape: it lacks facial animations; but it can track routes, match the speed of the user, and provide evaluation data so as to encourage the user.

Simulation agents rely on the user choices to produce difference situations or results. Holographic AIs play role of navigation, it not only simulates specific events, but also to a series of possibilities that will perform based on the user choices. Therefore, users can mix with real events based on such virtual experience.

Chatbot

Chatbot agents are conversational agents that are able to answer frequently asked questions, providing information in natural language [ 17 ].

With its AR hologram, Gatebox has created a holographic AI character named Azuma Hikari [ 18 ]. She is a blue-haired anime girl that is projected into a transparent tube, akin to a hologram pyramid (see Fig.1f). This holographic AI character employs various animations to perform corresponding scenarios, for example, like snapping her fingers when lights need to be turned on/off. The character communicates with the user, recognizes room temperature and the user’s face using motion sensors and cameras. Similar is Holographic Cortana (based on the character in the Halo video game series, see Fig.1e) [ 19 ], displayed on a holographic pyramid. Cortana uses natural language processing and information panels (e.g., weather forecast).

In 2019, Microsoft created a hologram that can project a person into a life-sized, speaking, Japanese virtual avatar [ 20 ]. In an effort to imbue realism in its hologram, Microsoft utilized high-resolution cameras and lighting. AI, text-to-speech synthesis, and natural language translation are used for capturing and recording a user voice in HoloLens to generate a personalized voice signature in real-time (see Fig.1g).

These novel UI chatbot agents are an improvement on monotonous interactive processes. Advanced sensor tracking enables the chatbots to be more responsive to the user and their environment. 3.4

Intelligent tutors

Intelligent virtual tutors are not limited to the educational field. They can be used for sports training, or training staff in handling risky procedures. For example, HoloCare (see Fig.1h) is a HoloLens application aimed at patients recovering from cancer surgery, aimed at improving quality of life and outcomes following treatment. In this case, an agent gathers health information to generate an exercise plan and guide the users through the selected exercises using life-sized 3D animations. Dialogue management and speech output are the main methods to quiz the user. MirageXR (formerly Wekit) produces so-called ‘ghost tracks’ of experts, that are animated subsequently to trainees as agents, relying on built-in sensors in the smart glasses for capture and anchored replay (see Fig.1i). Internet of Things (IoT) technology is used to reconstruct these experts in AR [ 21 ]. The floating agents can recognize physical objects and machines, and provide instruction for performing procedures. This helps trainees to compare performance and discover shortcomings by reviewing outcomes.

The appearance of these intelligent tutors is not wholly realistic or human, and they are effective in terms of helping the user improve their professional skills, stimulating stamina, or in cultivating companionship between learners and virtual agents. In both cases, emphasis is on the capture of what the expert does, as only with such representation (in the dialogue model or in the recorded ghost tracks) there is knowledge relevant to the learner.

Features for the design of holographic AIs

When comparing these four distinct types, we found that there are obvious differences between them, e.g., regarding the size of the agents (see Table 2). We propose to group them along their major hallmarks into appearance (A), behavior (B), intelligence (I), and responsiveness (R).

Appearance is an important attribute in social models that determines how motivating the virtual character can be [ 22 ]. The agents in the aforementioned examples have humanlike appearance and characteristics, which can be life-sized or pet-sized. They are, however, imbued with different levels of personification. In view of this, there are two factors of appearance: size mapping and realism. Appearances of holographic AIs are mostly humanoid to some extent, maybe with the small exception of the ghosts in Wekit/MirageXR.

Behavior is composed of animation and expressiveness. It is not merely animations, but subtle expressions that deliver more potential or explicit information within different storylines and conversations. For instance, game characters focus more on body movements, and the expressing of emotions in games is a functional capability which is context-based [ 23 ]. Chatbot agents emphasize facial expressions that offer realistic face-to-face communication (e.g., friendly and smiling faces). Besides, the animations of simulation agents are connected to the user’s choices as well as events and content of the dialogue, while the behavioral structure of intelligent tutors is a scaffolding, which focuses on key points designed to engage learners and produce affective feedback (using nodding, gaze, gestures, and facial animations).

Intelligence is focused primarily on how agents interact with users, and how smart and flexible they are. It refers to the ability to perceive the human user and react in an appropriate manner by adapting to users’ demands and executing tasks intended. The components of intelligence in this dimension include awareness, comprehension, and adaptation. Awareness relates to a virtual agent’s capability to identify and analyze real objects, contexts, and users’ faces. Understanding looks at the capabilities regarding processing of conversation. Adaptivity is about the degree to which the agent reacts to user behavior. Adaptation of game characters depends on the story arc, the ability to perceive the state of the game that is related to characters and users’ actions, or the capacity of agents to react to relevant goals [ 24 ]. Adaptation in the context of simulation relies on the user’s options and actions, and different behaviors cause different results, which stimulate users into reacting and acting in better ways. In terms of intelligent bots, natural language processing and dialogue management recognize users’ emotions, intonation, and conversational content producing correspondent behavior. Chatbot agents also apply recommendation systems as a tool to execute efficient communication based on user preference [ 25 ]. Besides, digital tutors apply natural language processing but also use it for formative assessment. Further, innovations in IoT and computer vision technology for intelligent tutors have resulted in improved user engagement. This enabled the development of applications employing agents in situ in workplaces requiring special industrial training.

Responsiveness refers to the capacity to react quickly and intuitively to the virtual and real spatial surrounding of the user and properties as well as events perceived in it, maintaining a certain level of corporeal presence for the holographic agent [ 7 ]. The agents’ awareness (see above: intelligence) thereby determines whether and how holographic AIs can respond to user behavior and/or physical particularities.

Table 1 captures and compares the different features of the reference framework of these four categories of holographic AIs.

Discussion and premilitary recommendation

Appearance and behavior represent exterior traits, which together account for the impression of the virtual agent. Intelligence and responsiveness are internal cores of a holographic AIs, responsible for tasks scheduling and execution, and adoption of corresponding appropriate behavior. The obvious difference between these across the types of holographic AIs can be summarized as follows.

Game characters are typically based on pre-structured storylines to perform various body animations, and the way of conversation is more similar to voice commands (yes/no, short answers). Simulation agents rely on mimetic events in the model of the simulation, reaction of holographic AIs and contexts guide users to make better choices. Talespin’s Barry Thompson [ 26 ], for instance, simulates an environment where trainees learn how to fire Barry by observing his behaviors and picking suitable utterances to speak with him. Chatbot agents collect data of preferences to provide individual recommendations by natural language processing. Multimodal interactivity also considers how to communicate with users in natural and flexible ways, which, in turn, make users feel that holographic AIs always accompany them. Teaching of course is a more critical goal for intelligent tutors, requiring them to recognize errors, correct it, generate feedback and teaching plans. Educational applications do not rely on specific teaching content to set conditions for agents’ behaviors in a specific environment but aim at a more general response to learners’ inputs compared to simulation agents. For instance, virtual patients need to perform precise behavior and produce according appearance (e.g., skin discoloration) to match the needs of the simulation. The features of intelligent tutors, however, can refer to pedagogical simulation agents. Both agents need to emphasize adaptivity and comprehension by combining knowledge about topics, available learning content, and students’ levels and acceptancy.

To summarize the insights gained in this study, we make the following four preliminary recommendations for the development of different holographic AIs:  Recommendation 1: Use case defines appearance: Photorealistic appearance and a rich set of animations help where improved engagement is required. If the use case is more restrictive in modality, e.g., aimed at highlighting movement, a more minimalistic style may benefit instead.  Recommendation 2: Natural language processing with dialogue management is key when implementing hands-free interaction, also supporting more engaging conversation (open-ended, less predetermined), not just simple speech recognition. Holographic AIs may wish to react and respond in multiple modalities, e.g., to determine user intent, recognize user behaviors, and activate non-verbal responses.  Recommendation 3: A personalized experience requires awareness and adaptation, which is paramount to providing an acceptable level of intelligence: recognize the activity of the user, adapt and react to it, generate corresponding response - this will help to produce humanlike performance.  Recommendation 4: Responsiveness relies on real-time mapping and analysis of the physical surrounding, when implementing environmentally responsive behavior like interaction with real-world objects. Multimodal interactivity in AR should not only focus on how to interact with users, but seamlessly integrate interaction both with the virtual and physical world.

Educational applications should respond to different scenarios in the physical environments needed for specific subjects. Currently, educational applications lack personalization despite of 1:1 interaction possible in AR. Multiple events and rich and frequent assessments are paramount in helping students review and correct their mistakes. Virtual teachers have the option to capture students’ psychological states or even track attention (via, e.g., eye- and gaze-tracking). Holographic AIs can play a role as teaching assistants to deliver student-centered services. The types of holographic AIs we identified in this paper also can motivate different learning activities and related services. The revised Bloom’s taxonomy, for example, groups learning along six levels [ 27 ], all of which need to be leveraged to ensure full utility for our virtual teachers or teaching assistants. For example, the cued recall possible with XR plausibly can explain better memory retention, while step-by-step guidance in XR contributes to better understanding and can motivate the development of problem-solving abilities. The practical application of knowledge in situ and in practice is gateway to higher level learning analysis and evaluation. This is supported by theory. Experiential learning no longer focuses on mere knowledge delivery, but rather promotes practical experimentation. The role of holographic AIs as a facilitator of collaborative learning experiences and as a supplement of social learning has not yet been investigated but seems to offer significant potential. 6

Conclusion and Future Work

In this paper we have explored the characteristics of different holographic AIs. We have reviewed state-of-the-art examples of four categories of autonomous and intelligent 3D agents: (i) game characters, (ii) chatbot agents, (iii) simulation agents, and (iv) intelligent tutors. We have considered the four aspects of appearance, behavior, intelligence and responses in our framework, which might serve as a standard for the design of virtual human characters and holographic AI technologies.

In terms of education and training, holographic AIs need to provide personalized service and optimized adaptivity with the help of acquired knowledge. Furthermore, real-time feedback (updating also the teaching plan) in time is also critical to ensure quality of learning gains.

The work conducted here serves to inform the research and development of Hanna, a holographic tutor for geometry (and other applications). It will apply natural language processing, dialogue management, and eye tracking to implement multimodal interactivities and user-centered services based on this framework. In the future, we will continue to investigate holographic AI agents, and consider how their acceptance, usability as well as user experience can be improved.

Acknowledgements

Work on this paper was supported by the European Union’s Horizon 2020 research and innovation program through the ARETE project (grant no 856533).

References

1. Sevilla , G.: AI company UneeQ seeks to take the next step in the evolution of intelligent chatbots: Enter the Digital Human ( 2020 ). https://archive.aweber.com/ispr2/Cnw0x/h/AI_based _digital_humans_are_next .htm, last accessed 2021 /05.

2. Catrambone , R. , J. Stasko. , J. Xiao. : Anthropomorphic Agents as a UI Paradigm: Experimental Findings and a Framework for Research , GVU Technical Report;GIT-GVU-02-10 , Georgia Institute of Technology ( 2002 ).

3. Chollet G. et al.: Multimodal Human Machine Interactions in Virtual and Augmented Reality . In: Esposito, Hussain, Marinaro, Martone (eds.): Multimodal Signals: Cognitive and Algorithmic Issues . In: LNCS , Vol. 5398 . Springer: Berlin ( 2009 ).

4. Russel , S. , Norvig , P. : Pearson: Artificial Intelligence: a modern approach. 3rd edition . Pearson: Wilmington ( 2016 ).

5. Gratch , J. , Rickel , J. , André , E. , Cassell , J. , Petajan , E. , Badler , N.: Creating interactive virtual humans: some assembly required . In: IEEE Intelligent Systems 17 ( 4 ), 54 - 63 ( 2002 ).

6. Lin , H. , Wu , Y. : Augmented reality using holographic display . In: Optical Data Processing and Storage 3 ( 1 ), 101 - 106 ( 2017 ).

7. Campbell , A.G. , Stafford , J.W. , Holz , T. , O'Hare , G. : Why, when and how to use augmented reality agents (AuRAs) . In: Virtual Reality 18 ( 2 ), 139 - 159 ( 2014 ).

8. Holz , T. et al.: MiRA-Mixed Reality Agents . In: International Journal of Human-Computer Studies , 69 ( 4 ), 251 - 268 ( 2011 ).

9. Khvatova , K. : What is an augmented reality agent ? University of Milano-Bicocca ( 2017 ).

10. Yin , R. : Case Study Research and Applications. Sixth Edition . Sage ( 2018 ).

11. Goodrick , D. : Comparative Case Studies. Methodological Briefs Impact Evaluation No. 9 , UNICEF Office of Research: Florence ( 2014 ).

12. Bowman , D.A. , Kruijff , E. , LaViola , J.J. , Poupyrev , I.: 3D User Interfaces: Theory and Practice . Addison Wesley Longman: USA ( 2004 ).

13. Young Conker, https://www.asobostudio.com/games/young-conker, last accessed 2021 /05.

14. Fragments , https://www.asobostudio.com/games/fragments, last accessed 05/ 2021 .

15. Pearson ( 2017 ), https://www.pearson.com/uk/web/pearsontq/news-and-blogs/ 2017 /04/ introducing-holopatient.html, last accessed 2021 /05.

16. GhostPacer ( 2020 ), https://ghostpacer.com, last accessed 05- 2021 .

17. Rickel , J. , Marsella , S. , Gratch , J. , Hill , R. , Traum , D. , Swartout , W. : Toward a new generation of virtual humans for interactive experiences . In: IEEE Intelligent Systems 17 ( 4 ) 32 - 38 ( 2002 ).

18. Azuma Hikari, https://www.gatebox.ai/en/hikari, last accessed 2021 /05.

19. Holographic Cortana Appliance: Working Concept, http://unt1tled.com/blog/cortanahologram-working-concept, last accessed 2021 /05.

20. Demo : The magic of AI neural TTS and holograms at Microsoft Inspire 2019 , https://www.youtube.com/watch?v=auJJrHgG9Mc, last accessed 2021 /05.

21. MirageXR, https://platform.xr4all.eu/wekit-ecs/mirage-xr/, last accessed 2021 /05.

22. Baylor , A. L. : Promoting motivation with virtual agents and avatars: Role of visual presence and appearance . In: Philosophical transactions of the Royal Society of London. Series B, Biological sciences 364 , 3559 - 3565 ( 2009 ).

23. Carroll , J. M. , Russell , J. A. : Do facial expressions signal specific emotions? judging emotion from the face in context . In: JPSP , 70 ( 2 ): 205 - 218 ( 1996 ).

24. Petri , L. , Björk , S. : Gameplay Design Patterns for Believable Non-Player Characters . 3rd Digital Games Research Association International Conference 4 , 416 - 423 ( 2007 ).

25. Cerezo , J. , Kubelka , J. , Robbes , R. , Bergel , A. : Building an Expert Recommender Chatbot . In: Proceedings of the 1st International Workshop on Bots in Software Engineering , pp. 59 - 63 . IEEE Press, Canada ( 2019 ).

26. Cnet , https://www.cnet.com/news/vr-lets -you-fire-a-virtual-employee-to-practice-doing-it-forreal , last accessed 2021 /05.

27. Anderson , W.L. , Krathwohl , R.D. , Airasian , W.P. , Cruikshank , A.K. , Mayer , E.R. , Pintrich , R. , Raths , J. , Wittrock , M.C. : A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives . Addison Wesley Longman: Boston, MA, USA ( 2001 ).