Telepresence is a technology that allows users to experience a real-time experience as if they were in a remote location. However, there is often a delay in communication before the user sees the image of the control feedback. Such communication delays significantly reduce the operability at the remote site so is the user's sense of agency in control of, for example, a remote vehicle. To reduce perceptible communication delays, we propose a remote vehicle maneuvering telepresence system that displays a predicted view by using odometry and 3D point clouds while showing live video when the odometry catches up with the vehicle's current pose. As a result of evaluation experiments, we confirmed that the proposed system significantly improves subjective operability and workload.
awareness; or using predictive techniques.
There are several approaches to predictive technology
Teleoperation is the electronic remote control of a machine to reduce the impact of delay. These approaches are
cateor vehicle, and remotely operated vehicles (ROVs) have gorized as dynamic system models and free models. Free
a wide range of applications on the ground, underwater, model approaches include information superposition
modin the air, and in space [
Remote control presents multiple challenges, one of and large amounts of information about the environment
which is delay in operation. In this study, we define it and the vehicle. Image processing-based approaches [
0000-0002-0486-1743 (M. Perusquía-Hernández); Therefore, we investigate whether a simple predictive
(0H00.0U-c0h0i0y2a-m65a3);50-8040309-0(0N0.3-I2so2y6a0m-1a7);0700(0K0.-0K0i0y2o-k6a1w1a9)-1184 display can improve operator performance and reduce
sub© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License jective workload during remote control. An experiment
CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutiRon 4W.0 Iontrekrnsathioonapl(CPCrBoYc4e.0e).dings (CEUR-WS.org) was set up to investigate changes in operator performance
and workload when piloting a vehicle under the following
three diferent video display conditions. The conditions
were: 1) RGB video (delayed live video), 2) point cloud
video (a naïve predictive display), and 3) automatic
switching between RGB and point cloud video. Participants were
asked to perform two simple navigation tasks: a driving
task and an observation task using a vehicle captured
by a first-person camera. Data collected included task
performance, perceived workload, and demographics. A
Likert scale questionnaire was administered to = 15
participants to test two hypotheses related to task
performance and subjective workload using ANOVA [
The challenges of telepresence, delays, and their detrimental efects are described from two perspectives: operator performance and subjective workload. Means to compensate for delays, especially forecasting techniques, are described.
Draper et al. [
There are multiple approaches to mitigating the detrimental
efects of delays. The first option is to increase the level
of automation (LOA) to reduce operator workload and
improve safety [
We assume a situation in which a remote vehicle-type telepresence is remotely piloted. The user, hereinafter referred to as the “operator”, is supposed to operate the vehicle and view the camera mounted on the vehicle by keyboard input while watching the PC screen.
In general, there is a delay between the time when a
2.2. Type of Delay control command is inputted and when it is reflected in
The literature [
DS-700AC
Predicted Image using Point Cloud & Odometry
Distance PPoriendticCtleodudIm&aOgedoumsientgry Vehicle Movement
Transferred
Delay Delayed TImimaege realized by ROS. For example, a single command to move forward causes the vehicle to move 10 cm forward, and so on. The odometry computation module estimates the current position and attitude of the vehicle by integrating the angle of rotation of the wheels. By performing this estimation in the local environment as well, it can estimate how the remote vehicle should behave according to the input maneuvering commands, and render a 3D point cloud from the estimated position and orientation to present the operator with a realistic reproduced image that immediately responds to the maneuvering commands without waiting for the actual image to arrive. The realistic reproduced image that immediately responds to the control command is presented to the operator without waiting for the actual 3.3. Software image to arrive. If the local environment and the remote To render the 3D point cloud using the RGB and vehicle’s position and orientation are judged to match depth information acquired by the ZED Mini, we used based on the amount of delay and odometry information, depth_sensing.py, available on GitHub from Stereothe delayed RGB image (live video) is presented as is, labs, the distributor of the ZED Mini. The point cloud instead of the 3D point cloud. This prevents deterioration rendering part of this program was modified to render of the video quality at the remote location due to inaccurate based on the vehicle odometry information. The moveor missing depth information when the vehicle is stopped. ment of the virtual camera in the 3D point cloud and the movement of the vehicle in the real world as a result of 3.2. Hardware the control is shown in Figure 3.
We implemented the system so that the control commands entered by the operator are sent to the vehicle via a socket communication. The program1 available on The hardware used for the prototype is shown in Figure 1 and the communication of the system is shown in Figure 2.
A ZED Mini was used for the stereo camera and a Pioneer 3-DX was used for the vehicle. To transfer the images from the stereo camera to the operator’s PC, we used a USB 1https://github.com/kumahika/delivery_navigation GitHub that can control the Pioneer 3-DX was modified to control the vehicle via keyboard input.
To investigate the impact of the proposed method on the
performance and workload when maneuvering a vehicle,
we conducted an evaluation experiment comparing three
feedback conditions. Participants were asked to perform
a driving task in which they drove a vehicle on a route
with repeated curves in all the diferent conditions, and
an observation task in which they answered questions
based on images from three monitors facing in various
directions during the driving task. These tasks were
designed because they involve multiple types of tasks,
especially lateral movement tasks. The benefits of the
prediction technique are highly task-dependent [
The following hypotheses are tested to investigate the efectiveness of the proposed method.
H1 An improvement in worker performance with a hybrid method when compared to single RGB and point cloud methods.
H2 An improvement in the subjective maneuverability of the worker with a hybrid method method when compared to single RGB and point cloud methods.
Figure 4 shows what the operator sees in the video while the vehicle is moving forward, stopping, and turning in each of the three conditions. In RGB, the images are presented as captured by the camera in Figure 4(a) forward, Figure 4(d) stop, and Figure 4(g) turning. On the PC, the images are presented as a 3D point cloud in all cases shown in Figure 4(b) forward, Figure 4(e) stop, and Figure 4(h) turning. By immediately reflecting the maneuvering commands input to the virtual camera in the 3D point cloud, the reproduced image seen when the maneuver is reflected is presented without waiting time. However, the reproduced image using the 3D point cloud image has many missing parts, and therefore, the image quality is inferior to the RGB condition. In the proposed HB method, the camera presents the image as it captures it only when the vehicle is stopped (Figure 4(f)), and presents the image as a 3D point cloud when the vehicle is moving (e.g., forward (Figure 4(c)) or turning (Figure 4(i))). As soon as a maneuvering command is inputted, the system switches to presenting the image in the 3D point cloud, and when the vehicle catches up with the virtual camera, the system switches to presenting the image as it was captured by the camera.
To avoid order efects and learning efects, the order in which the conditions were implemented was randomized using the 3 × 3 Latin square method. Participants were not informed of the characteristics and functions of each condition, and their ability to use them was left to their intuitive comprehension.
= 15 volunteers participated (age range: 22-28 years, average age: 23.5 ± 1.5 years, 3 females). All experiments 4.2. Experimental Conditions in this research were approved by the ethical review board The three conditions are RGB (delayed live video), PC of the Nara Institute of Science and Technology with (Point Cloud), and HB (Hybrid, i.e., the proposed method). review code 2022-I-41.
As previously described, the intentionally added delay was 2 seconds for all conditions, meaning that the total 4.4. Task efective delay was about 2.5 seconds.
In general, experiments on forecasting technology measure RGB This is a naïve remote control method that presents performance using two metrics: course completion time RGB images (live video) with large delay as they and task score. The former measures the time required to are. complete a course for the task of moving a vehicle along a predefined path. In the latter case, the task is to move PC The 3D point cloud is drawn from the vehicle’s esti- toward a given target and complete the indicated task. In mated current position and orientation based on this experiment, the participants were asked to look over odometry information, and a reproduced image is a fence and answer formulas displayed on three monitors presented that immediately reflects the maneuver- facing in diferent directions. The formulas included ing of the vehicle. addition with a decimal point (e.g., 12 + 3.4 =?), and the HB The proposed method combines RGB and PC. Nor- task was designed in such a way that the participants would mally, a PC is used, and is switched to RGB make a mistake if they overlooked the decimal point. adaptively when the real vehicle catches up with the position and orientation of the virtual camera in the 3D point cloud.
(a) RGB: Moving forward (b) PC: Moving forward (c) HB: Moving forward (d) RGB: Standstill (e) PC: Standstill (f) HB: Standstill (g) RGB: While turning (h) PC: While turning (i) HB: While turning
A 17.3-inch laptop computer with an Intel Core i9-9900K CPU 3.60 GHz and Windows 10 was used as the operator’s remote control computer. The laptop screen functioned as a monitor, and the WASD and QEZXC keys on the keyboard were used to control the vehicle. The M key was used to record the time stamps that the participants themselves pressed at the beginning and end of each of the two tasks. The keyboard and mouse were used to answer the questionnaire.
A photograph of the test course is shown in Figure 5(a) and its description in Figure 5(b). The green area in Figure 5(b) was the driving task route, and the red area was judged to be out of course. No time limit was set for the driving task, and participants were instructed to drive as safely as possible. If the participants went of course, the number of times they went of course was recorded while the driving task and the experiment continued. The course line of the test course consisted of plastic chains so that the experimenter would be able to recognize by sound when the vehicle went of the course or rode up on the course. The participants, who were the operators, were not aware of the sound of the plastic chain because they were wearing noise-canceling headphones playing white noise during the maneuver.
We used three monitors facing diferent directions at the two observation points (Figure 6) placed in the gray area shown in Figure 5(b). Each of the three monitors displayed a diferent formula which flowed from right to left. A fence was placed in front of the monitors so that the monitors’ how it worked. Therefore, the use of the prediction display was left to the intuitive understanding of the individual participant.
Participants’ subjective delay experience in each condition was investigated, and a perceived delay time question was asked in the questionnaire to compare each condition. By asking about perceived delay, we hoped to provide an indication of the efectiveness of the predictive display in reducing the perceived delay of the system.
As evaluation indices, four items were used: time, the number of times each maneuver key was entered, the number of times each maneuver key was released, and the vehicle’s xyz coordinates, quaternion number, and movement speed were recorded from the time the participant started the experiment until the end. These indices were continuously collected during the execution of the task in each condition. After each condition, the participants were asked to respond to a questionnaire created with Google Forms. The questionnaire consisted of 15 questions: 10 Likert-type questions in which participants were asked to select one of seven levels from 1 (strongly disagree) to 7 (strongly agree), and five open questions in which they were asked to write freely about the content of the questions.
See Figure 8 for the text of each question. screens are visible only from the front. The operator moved the vehicle within the observation task area, the yellow area in Figure 5(b), and maneuveres it to a position where the monitor screen was visible. The purpose of this arrangement was to prevent the operator from completing the observation task without maneuvering the vehicle, and to encourage the operator to maneuver the vehicle to the position in front of each of the three monitors. The observation task was terminated by having the operator verbally answer all of the formulas displayed on each of the three monitors. The number of incorrect answers and their causes were recorded.
Participants were first briefed on the experimental flow
using slide materials after reviewing the experimental envi- The task completion time for each of the driving and
ronment to understand the situation and the issues involved. observation tasks in each condition is presented in a
boxParticipants were seated in a chair at a desk with a laptop and-whisker diagram. Tests were conducted to confirm
computer, with their backs to the area where the vehicle if there were any significant diferences, as shown in
Figwas moving. During the experiment, participants did not ure 7. A one-way repeated measures analysis of variance
have direct visual contact with the moving vehicle and (ANOVA) was conducted to determine if the diferences
wore headphones playing white noise to avoid the auditory in the three conditions were statistically significant. The
perception of the vehicle. After agreeing to participate Holm method was used to correct for multiple comparisons.
in the experiment, participants completed a demographic For the driving task, there was a significant diference
bequestionnaire. The experiment was described in terms tween RGB and HB (p<0.05), and the proposed condition
of the experimental procedure, the test course, the types reduced the task completion time. For the observation
of tasks, the content of each task, and the maneuvering task, a trend toward significance was found between PC
method. At the beginning of each condition, there was a and HB (p<0.10). Also, there was no significant
difer30-second practice period to familiarize themselves with ence between RGB and HB. Therefore, we confirmed that
the conditions. Thereafter, the participants performed the the proposed condition was as eficient as RGB for the
driving task and the observation task consecutively, with observation task. From these results, H1 “Significant
the end of the observation task marking the end of one con- improvement in worker performance” was supported.
dition. In the driving task, the experiment was continued The Likert scale ratings in each condition are presented
even if the vehicle went of the course, and the number of in a box-and-whisker diagram for each question, and
times this occurred was recorded by the experimenter. In tests were conducted to confirm significant diferences,
the observation task, participants were asked to answer a as shown in Figure 8. Simply treating the ratings as an
total of three questions on three monitors, and the number interval scale [
Significant diferences were found between RGB and PC, and HB and PC, respectively. These results supported the alternative hypothesis.
Significant diferences between RGB and PC, and between RGB and HB were confirmed for impression, sense of security, comfort, discomfort, operability, and delay, at p<0.001. A significant diference of p<0.01 was found between RGB and PC, and between RGB and HB in the sense of being able to control by one’s own will.
For the question of visibility, a significant diference of p<0.001 was confirmed between RGB and PC, and between PC and HB.
A significant diference of p<0.001 was found between RGB and PC, and between PC and HB in the question of whether the pilot was able to maneuver as intended. A significant diference of p<0.01 was found between RGB and HB. A significant diference of p<0.01 was found between the RGB and PCs in the sense of local control. A significant diference of p<0.001 was found between RGB and HB.
From these results, we confirmed that H2 “Significant improvement in subjective maneuverability of the worker” was supported.
Below we list representative participants’ comments obtained for each condition per question. Overall, the comments were more positive for the HB condition than for the others. 4.9.1. RGB • “There was a considerable delay, and it took some time for the intended operation to be reflected, during which time I felt uneasy.” (Male, 23 years old), • “The delay was so large that I felt insecure about whether the input was correct or not.” (Male, 22 years old), • “I felt a considerable delay, which made it dificult to make fine adjustments and made me uneasy.” (Male, 22 years old), • “I felt uneasy because I did not know how much I had to press a button to move forward or backward.” (Female, 24 years old) • “If there had been no delay, I would have been able to control the vehicle as intended.” (Male, 23 years old). 4.9.2. PC • “The robot responded quickly, so it seemed to stop when the timing was right.” (Male, 25 years old), • “The robot’s position was easy to grasp without any delay.” (Male, 22 years old), • “There was no stuttering or switching of images, and I felt that the robot could be operated comfortably.” (Male, 23 years old), • “I was able to control the robot as I expected.” (Male, 23 years old), • “There was no problem in driving, but the image was unclear when performing observation tasks.” (Male, 23 years old), • “The image was rough and there was no image when turning.” (Male, 24 years old), and • “When the visibility of the image was extremely poor, I strongly felt that it was a remote control.” (Male, 24 years old). 7 6 ltrscea45 ikLe32 1 7 6 ltrscea45 ikeL32 1 ***
*** **
** PointCloud Condition PointCloud Condition PointCloud Condition
Ours Ours Ours
RGB
Ours
RGB (a) Good impression of the maneuver. (b) Secure in maneuvering. (c) Comfort in maneuvering.
Ours
RGB
Ours
RGB (d) Comfortable with the controls. (e) Maneuvering as intended. (f) Operability was good.
Ours
RGB
Ours
RGB
Ours (g) The video was easy to view. (h) Not notice any major delays.
(i) Piloting of my own volition. *** *** *** *** PointCloud
Condition ***
** PointCloud
Condition ***
*** PointCloud
Condition ** *** 7 6 ltrscea54 ikeL32 1 7 6 ltrscea54 ikeL32 1 7 6 ltrscea54 ikeL32 1
RGB RGB ***
*** ***
*** PointCloud Condition PointCloud
Condition ***
*** RGB
PointCloud Condition 7 6 ltrscae45 ikeL32 1 7 6 ltrscea45 ikLe23 1 7 6 ltrscea45 ikLe32 1
RGB
Ours PointCloud
Condition (j) Feeling that piloting the vehicle locally, not remotely. 4.9.3. HB (Proposed Method) • “It was the easiest to maneuver.” (Male, 24 years old), • “It was easy to operate with little delay.” (Female, 24 years old), • “The front/rear maneuvering was almost no delay at all.” (Male, 24 years old), • “The observation task was quite comfortable, although the impression was not so diferent from other conditions when driving.” (Male, 23 years old), • “The response time was fast, so it seemed that the camera stopped when the user wanted it to stop.” (Male, 25 years old), • “There was some discomfort in the switching between low and high-quality images.” (Male, 23 years old), • “The field of view was narrow and it was dificult to operate when turning, although I did not feel much delay when moving straight ahead.” (Male, 23 years old), • “It was stressful to have to wait until the delay was resolved because the image did not catch up with the rotation.” (Male, 24 years old), and • “While turning, there was a delay while waiting for the image to be displayed.” (Male, 24 years old).
Dealing with delays is one of the major challenges in telepresence. Therefore, we proposed a telepresence system that automatically switches between predictive images and live images using 3D point clouds according to the situation. It also presents images that show the user’s maneuvering immediately so that the user does not perceive any delay in the images. As a result of evaluation experiments, we confirmed that the proposed system actually improves subjective operability and ease of viewing. In the future, we aim to resolve issues such as the phenomenon of the edges of the screen being blurred and many parts not being rendered when turning on PCs and HBs. This can be achieved, by switching between RGB images and 3D point cloud-reproduced images and by devising a new method for rendering point clouds.