Drone pilots face obstacle awareness challenges in case of bad lighting conditions, distractions, or when flying in any direction that is not in the camera view. Possible obstacles include static and dynamic obstacles such as other drones, humans, animals, or even brick walls within buildings. We propose a tactile system to indicate obstacles, including their distance from the drone, in the 3D space around the user (see Figure 1).
Earlier work and concepts on human-drone interaction was neatly summarized in [4] and explained in further detail by Baytas et al. [1]. Our obstacle awareness concept presented in this paper extends the idea of augmenting spatial awareness for humans [2] and aims to instead increase spatial awareness of a human controlling a remote drone. Earlier approaches to this challenge were able to show promising results for a 2D navigation task using ultrasound sensors attached to a drone and a vibrotactile belt [13]. We aim to extend Spiss et al.'s obstacle awareness system to 3D use cases, which cannot be displayed properly by the tactile belt used in [13].
In our previous work, we presented HapticHead [6, 7, 5], a vibrotactile display around the head consisting of a bathing cap with a chin strap and a total of 24 vibrotactile actuators (see Fig. 2). We were able to show that our prototype can be used in 3D guidance and localization scenarios for people with normal vision in both virtual (VR) and augmented reality (AR) scenarios. The system can indicate directions all around the user and guide the user to look at a defined point in space with a median deviation of 2.3°to the actual target. This precise guidance capability may also be used to make users aware of obstacles in the space around them. The previous work further included a scenario in which blindfolded users were able to feel the presence of real physical objects in the 3D space around them and subsequently were able to find and touch the objects (see [7] and Fig. 3).
A suitable 360°obstacle detection system for drones is needed as an input for our proposed system. There are a variety of systems and technologies that could serve as an obstacle detection system, such as multiple stereo cameras working together [12,10], 3D LIDARs [9], or even a system using, e.g., HyperOmni Visions (HOVIs) [11]. These input systems would need to filter and extrapolate static and dynamic obstacles, including their distance and 3D viewing angle from the drone camera perspective. The detected obstacles should further be filtered so that obstacles further away than a threshold distance would be excluded from the results, as these can be deemed harmless at the given moment.
In our prior work, we introduced a 3D guidance algorithm for arbitrary actuator configurations such as HapticHead [7]. This guidance algorithm proved to be quite efficient and fast in guiding study participants to look in the indicated direction in 3D, including elevation. The same algorithm can be used in obstacle awareness scenarios as well. Just like in [7], the depth to obstacles may also be indicated by a vibrotactile pulse-pattern and intensity modulation which gets faster and stronger, the closer an object is.
The spatial mapping of the vibrotactile feedback is drone centric: The output occurs relative to the drone that the user is controlling and is mapped in a one-to-one fashion to the HapticHead. The front of the drone is mapped to the front of the head. Obstacles appearing in front of the drone are haptically displayed on the forehead. Obstacles that appear to the right of the drone appear on the right side of HapticHead, and so on. This yields in natural mapping of the drone coordinate system to the head coordinate system. To the user it feels as if he or she is flying as a pilot inside the drone, intuitively feeling obstacles along its way.
When indicating multiple obstacles at the same time with the proposed tactile interface, a user will likely suffer from a loss of localization accuracy. For one, if two obstacles are close together, the user will only be able to perceive one of them, as the vibrotactile pulse-pattern would become confusing if two obstacles overlap from the perspective of the drone and thus allocate the same actuators on the Hap-ticHead. Arguably, this limitation is no deal breaker, as the user can still feel the distance of the closer of the two (or more) objects. Furthermore, if more than two obstacles are indicated at the same time, even if they are not allocating the same actuators, a loss of accuracy is still likely. This results from sensory congestion/overload or funneling illusion effects if too many actuators are active at the same time [3,8].
As a solution to these issues, we suggest to only indicate the closest two or three obstacles at the same time and merge obstacles that are close together, only indicating the closer obstacle.
As indicated in the introduction, the proposed system may be used in a variety of use cases related to drone operation and handling. These include:
1. Flying into any direction that is not in the camera view (e.g., side-, back-, down-, or upwards);
2. operating a drone at night or in bad lighting conditions;
3. operating a drone around areas with many static or dynamic obstacles such as other drones, humans, animals, plants, or walls within buildings;
4. operating a drone while being distracted (e.g., by other humans).
In the first three cases, the system would provide tactile guidance to the closest two or three obstacles, so that the user can intuitively navigate his drone out of a dangerous situation. In the fourth case, the system would provide tactile warnings in case an obstacle is close which reminds the user to redirect his attention back to the drone.
Another use case would be accessibility: visually impaired drone operators should have a much easier time avoiding obstacles due to the additional tactile feedback channel.
In conclusion, we propose a tactile obstacle awareness system for drone operators, which may be used in a large variety of use cases. Future work may implement the proposed system and test the assumed benefits in a real environment.