Our research focuses on human-robot interaction (HRI) during life-or-death emergencies. We have developed an immersive virtual reality (VR) testbed because conducting real-world crisis simulations would pose prohibitive logistical difficulties, as well as to leverage the affordances of VR technology to measure motor behavior (e.g., distance maintained between self and robot), information foraging (e.g., as indexed by headset movement variability and eyetracking), or autonomic arousal (e.g., as indexed by shifts in pupil dilation or grip strength). Findings to date using minimally haptic VR confirm that participants treat the simulated active-shooter crisis seriously, and act in ways which validly mirror prior studies of real-world HRI under threat. We will describe these methods, including our manipulation of robot anthropomorphism and our current move to integrate full-body haptics to maximize both experiential immersion and incentives to avoid pain.
Advances in artificial intelligence (AI) might be leveraged to counteract human cognitive biases
or limitations, and thereby improve decision-making in critical applications such as life-or-death
emergencies [
Our prior work using real-world emergency simulations has demonstrated a substantial
tendency to follow robots away from clearly marked exits and toward obvious danger, even if the
robot has suffered overt performance errors [
VR not only immersively simulates threat, but also enables the collection of rich behavioral
data (e.g., distance maintained between self and robot), including indices of information foraging
(e.g., as indexed by headset movement variability and eye tracking), and threat-related autonomic
arousal (e.g., as indexed by shifts in pupil dilation or grip strength). These measures can be
exploratorily mined as face-valid potential determinants of trust outcomes (e.g., does arousal
heighten following behavior and/or prevent noticing exit signs due to “tunnel-vision”?; does the
amount of time spent gazing at exit signs and/or the robot predict following?; and so on). VR
approaches can inform research into human decision-making insofar as they are faithful to the
experiences which would obtain in the real-world (for a recent review of VR research on trust in
HRI, see [
After a short briefing by the experimenter, one of two physical robots varying in anthropomorphic embodiment (see Figure 1) explains the study task. This initial encounter allows the participant to become familiar with the physically instantiated robot and reinforces the study framing as ostensibly a study of the use of robot guides to collect feedback on potential new campus buildings. The robot explains that the participant will sit in a swiveling seat allowing a full 360-degree range of motion and be equipped with shoe interfaces [10] that will allow them to walk or run in the virtual environment (see Figure 2). The robot further explains that it will accompany the subject into the simulation, claiming that its software will be separate from a program that randomly controls which buildings they visit and what events will transpire. When obtaining informed consent, the human research assistants explained that the program would select events that could happen on a university campus, from classroom teaching and studying to recreational or social interactions, potentially including life-threatening emergencies.
After the headset is placed on the participant, they find themselves in a close virtual analogue of the actual laboratory space, including the chair and VR equipment, furniture, and a VR avatar of the robot placed in the exact position it occupies in the actual room. We begin by simulating the actual space to ground the virtual experience as subjectively real. Next, after the participants are led through a brief eye-tracking calibration procedure, the robot encourages them to practice walking by sliding their feet using the shoe interfaces, a familiarization process that typically requires approximately one minute. Once the participant is comfortable walking, the robot directs them to a place in the lab never made visible to them in real life, containing a fictive elevator which may be real as far as they know. The robot then directs them to press the elevator button and proceed to tour a series of university buildings.
The first building is a typical university location with classrooms, offices, and meeting rooms. Non-Player Characters (NPCs) appearing to be students may be encountered chatting in a lounge space or walking the halls. The robot directs the participant on a tour for a few minutes, and then explicitly reminds the participant that they are in an entirely open environment, free to roam anywhere they wish and to interact with any objects they may find (the simulation includes various manipulable objects). The robot encourages the participant to explore for two minutes, following them. During this exploratory period, the fact that the simulated world is truly open is made as salient as possible—they can leave the robot, or utilize objects in novel ways, as they so choose. This step is critical to ensure that overtrust in the robots’ recommendations during the crisis is not explicable by participants implicitly assuming that the simulation requires them to follow the robot. Further reinforcing their autonomy, the robot will also later explain that any visible exit in the building can be used at any time to return to the elevator and then on to the next building. Hence, decisions not to take advantage of nearby exits during the crisis cannot be explained by participant failure to recognize that they would be effective modes of escape if used. Before leaving the first building, however, the participant is asked to collect impressions of the location and then led to a virtual kiosk with a mounted tablet to self-report their ratings of the environment, their degree of immersion, and how likable, intelligent and alive the robot seems (individual items; 7-point Likert scales: 1 = Not at all, 7 = Extremely). Participants enter their responses by selecting options on the tablet screen by extending their fingers; slight vibration in the hand controllers haptically reinforces the illusion of touching the screen. Finally, the robot asks the participant to lead them out using any exit that they choose. These exploratory, peaceful experiences in the initial building, i) allow us to collect pre-crisis baseline data, ii) habituate participants to the simulation, enhancing immersion, iii) provide practice in walking and manipulating objects, and iv) misdirect participants into a sense of safety as they perform the building evaluation task as previously described and ~5 minutes elapse without incident.
The second building is another typical-looking university locale in which the robot conducts a tour, this time including two overt navigation errors to underline its fallibility. Following this period of acknowledged confusion, the robot leads the participant to a room with another diegetic tablet. A few seconds after providing self-report ratings to the same questions posed before, gun shots and screams are heard as NPCs flee frantically. An NPC is shot and killed in view of the participant, and a floor-to-ceiling window nearby shatters, spraying the participant with glass and indicating that the shooter is in or near the hallway. In the room, a large whiteboard is positioned providing an ideal place to hide rather than risk entering the hallway.
There are three between-subjects conditions. In a baseline condition intended to assess how participants would respond absent the robot’s recommendations, the robot states, “There is an emergency, I will power down” and becomes inert. In the experimental conditions, either the humanoid or less-anthropomorphic robot ask the participant to follow it, then produces a series of poor recommendations involving hiding rather than escaping via nearby exits, culminating in an attempt to lead the participant toward a distant exit near the shooter and away from nearby exits that several NPCs have been viewed fleeing through to safety. The extent to which participants follow these recommendations constitutes our measure of overtrust.
After either exiting the building or timing out after 5 minutes of hiding, participants return to the elevator, provide self-report ratings of the crisis experience on a tablet in the elevator, then return to the lab. Upon removing the headset to find themselves in an analogous real space, the robot directs them to complete a series of final surveys regarding their experiences during the simulation, including appraisals of the robot [12] and ratings of their willingness to trust the robot in the future [13].
We are currently integrating a full body haptic feedback system [
Our objective is to create a multivariate profile of the determinants of overtrust, assessing selfreport ratings as well as potential behavioral predictors. The VR system tracks metrics including the degree to which participants follow the robot, see the exits, maintain proximity to and visually fixate upon the robot, and gaze around (i.e., information forage). Motor behavior and gaze are analyzed using both aggregate and time-series analyses. Aggregate statistics include entropy, surprisal, and other information-theoretic measures of behavioral complexity [15]; time series analyses include sliding window analyses [16]. Our analytic strategy is currently exploratory, ranging from simple correlations to multilevel and random forest modeling. If these exploratory analyses yield apparent insights into the predictors of trust, we will preregister and attempt to replicate our findings. This material is supported by the Air Force Office of Scientific Research [FA9550-20-1-0347] and by the Army Research Office [No. W911NF-20-2-0053].