Conventional safe route-finding systems display routes on a map to avoid dangerous areas based on offline data such as statistical data on crime rates. However, such dangerous level changes in a dynamic manner, which can be detected by people walking around. This paper presents a system that shares the detected data measured by edge devices owned by walking people for dynamic safe route finding in real-time. This system evaluates the safety of routes in real time using and displays the route in AR on HMDs and smartphones. This paper reports findings from a simulation and edge device (smartphone) implementation of spatial safety assessment, safe route finding, and route visualization.
Violent crimes such as assault and injury
(Assault is defined in Japan as a crime in which
the victim was not injured, while an injury is a
crime in which the victim was injured) are crimes
that occur close to people, have a significant
impact on the victim's mind and body, threaten
their peaceful daily life, and may increase social
unrest. According to a study on violent crime
conducted by the Ministry of Justice's Legal
Research Institute in Japan[
Conventional navigation systems are designed to arrive at a destination smoothly and quickly. To achieve a safe and secure society, we research, develop, and implement navigation systems that can avoid crimes and unpleasant events by detecting them in advance. The navigation system uses edge devices such as surveillance cameras and smartphones to measure the level of danger in real-time and search for safe routes. The system also visualizes the detected dangerous elements and the safe routes.
We are convinced that this system can contribute to solving various social problems. For example, the system can be used for urban planning to prevent crimes, support services to maintain social distance in infectious disease environments such as COVID-19, detect people in trouble and automatically ask for help from people around them, support services to protect wild animals, and services to find lost cats.
In this paper, we report on the findings and problems encountered in simulating the level of danger assessment, safe route search, and safe route visualization of a location using edge devices. We also report our findings and problems in a prototype system using a smartphone. 1.3.
Many methods have been proposed for safe route finding that minimizes the level of danger and distance on the path and displays the path on a map.
Galbrun et al. proposed a safe route search
method based on Total-Paths (total level of danger
on a path) and Max-Paths (maximum level of
danger on a path) [
Sukru et al. propose a mobile app for the social
distancing of people in a COVID-19 environment.
It allows each user to manually register their
health status and then share their current location
on the network in real-time, predicting the
walking paths of other users and alerting them
with sound and vibration to the possibility of
approaching users with poor health status [
As a mechanism for sharing information sensed by the user, the Joint Tactical Information Distribution System (JTIDS) and the Intervehicular Information System (IVIS), which are used in military applications, can be used as references. These systems share the position, heading, altitude, and speed of detected aircraft, ships, and vehicles with all platforms participating in the network.
For safe route finding, it is necessary to
evaluate the safety of a location. According to
Stacy et al., a person's gender, facial expression,
gaze, appearance, clothing, attitude, and physique
are known to help determine whether a person is
suspicious [
The freshness, spatial resolution, and accuracy
of the original data are important for assessing the
level of danger. Ideally, the data should be
updated in real-time and evaluated by an impartial
third party with a spatial resolution of about 3.5 m
[
Therefore, we are conducting research, development, and social implementation of an information-sharing system and navigation system that can detect and avoid criminal damage and unpleasant events in advance, featuring the following three elements. (1) Edge devices accurately measure and share the level of danger of a location in real-time. (2) A wide-area safe route-finding method based on the level of danger measured by edge devices, and a narrow-area safe route-finding method for successfully passing dangerous people in a narrow space while maintaining social distance. (3) Social Distance and safe route visualization and human guidance methods 2.1.
We have conducted a system study to achieve a navigation system that can detect and avoid criminal damage and unpleasant events in advance. This system consists of six subsystems as shown in Figure 3.
Safty (1) Edge Devices (2) Location danger level evaluation system (3) Wsiedaer-cahresaysstaefmeroute (4) Nasreroawrc-harseyastseamfe route visu(a5l)izSaatfieonrosuytsetem Figure 3: System Overview (6) Data sharing server (1)Edge devices
A device equipped with various sensors such as a camera, microphone, thermo, GPS, etc. We assume surveillance cameras, smartphones, and wearable devices. (2) Location danger level evaluation system
The system evaluates and shares the safety of a location based on information acquired by various sensors in edge devices. We can use other information such as sound (conversations, gunshots, explosions) or temperature (human body, road surface, fire, freezing) to evaluate the safety of the location. This information can be used to set various conditions in addition to safety, such as easy walking routes, cool routes, and routes with many pedestrians. (3) Wide-area safe route search system
This system performs a wide-area route search to maximize safety and minimize distance based on the safety level and distance of the location evaluated by the edge device. (4) Narrow-area safe route search system
This system searches for a safe route in a narrow space (e.g., a street or a station) to pass by a dangerous person while maintaining a social distance from him or her. (5) Safe route visualization system
This system visualizes shared information like dangerous persons detected by other edge devices. This system also visualizes the routes and social distance calculated by the safe route search system. Wide-area safe routes are displayed on a map, and narrow-area safe routes for passing each other are displayed on the HMD. (6) Data-sharing server
A server that accumulates real-time spatial safety information. It stores information uploaded from each edge terminal and distributes it to each terminal in real-time. 2.2.
We will evaluate the usefulness of the designed system. However, our system includes subsystems that are difficult to implement at this time. In particular, as mentioned in section 1.3, there is no established method to evaluate the safety of a location with sufficient accuracy on an edge device. In Addition, there are also no small, lightweight, safe, optical see-through HMDs that can be easily worn. Because of these difficulties in implementing our system in the real world, we first conducted our system evaluation in a simulator.
We constructed a virtual Akihabara city [
Figure 4 shows the results of mapping the current positions of PF, PD, and PS detected by 100 edge devices on a 2D map. We drew bubbles (spheres) colored according to the attributes at the detected person's location. This bubble is drawn at the location of the PF, PD, and PS at the time the PE detected them, but they could move. Also, the map would be filled with bubbles of newly detected PFs, PDs, and PSs. Therefore, we set the transparency of the bubbles to indicate how new the information is, and set them to gradually disappear in one minute so that the map would not be filled with bubbles. Users can see that the disappearing bubble is old information.
When a PE with an edge device sets a destination, a wide-area safe route system performs the route search. We have extended the Dijkstra method, which is used for route finding in navigation systems, to perform wide-area safe route finding considering distance and level of danger in space. Dijkstra's method is an algorithm in graph theory that can find the shortest distance from a start node to a goal node and its path. We generated a graph structure from map data in which the intersections are represented as nodes I and the roads connecting the intersections as edges E. We defined the Edge Danger Level (EDL) as the level of danger between intersection In and the adjacent intersection In+1, where EDL is the sum of the DP of PF, PD, and PS that exist between the intersections. We set a danger threshold value DLth, and if EDL exceeds the DLth, the path is considered impassable and the node connection is disconnected. In this simulator, we set the threshold DLth = 5. The system evaluates the EDL of all edges, updates the graph structure, and finds the shortest and safest path using the Dijkstra method.
Considering the threshold DLth = 5, edge E(I2,
I3) is cut off because it is impassable, and the
shortest path search is performed on the remaining
edges, resulting in the path shown in Figure 5(b).
E(I2, I3) and E(I3, I4) are passable, but there are PF
and PD, so a narrow-area safe path search is
performed to find a path that allows them to pass
each other while maintaining the social distance.
Figure 6 shows an example of a narrow-area safe
path search. The system generates a graph
structure by dividing the space to be traversed into
meshes. The size of the mesh was set to 3.5 m, a
social distance defined by Edward et al[
Figure 7 shows the mapping results of the current positions of PF, PD, and PS detected and shared by 100 edge devices and the wide-area safe route on a 2D map. We deployed PE0, a person tasked with moving from the start point S to the goal point G. PE0 is equipped with an edge device and moves along the route indicated by the edge device's wide-area safety route search. PE0 is equipped with an edge device and moves along the route indicated by the edge device's wide-area safe path search. Although the shortest route is to go straight from S to G, the blue color indicates that the route search is performed in consideration of the level of danger. When PE0 needs to pass PF or PD on the route, the system performs a narrowarea safe route search and shows PE0 a route to pass them while maintaining social distance. Figure 8 shows the view from PE0. In this experiment, PE0 is assumed to be wearing a seethrough HMD. System display PF, PD, and PS detected and shared by other edge devices. The system also displays a narrow-area safe route search result. The system superimposed bubbles with sizes corresponding to the social distance for each attribute of PF, PD, and PS, and the safe passing route (green).
Simulator experiments have shown that the system works effectively within the virtual Akihabara. Therefore, we developed a prototype system using a smartphone as shown in Figure 9. The edge device on the simulator we implemented was an ideal device that could detect people and their attributes within a radius of 30 meters without error. However, since such an ideal device does not exist in reality, our prototype system simply detects a person from a smartphone's RGBD camera image (without a level of danger assessment) and shares the person's location coordinates. We developed a smartphone application in Unity and installed it on an iPhone 13 pro max (Figure 10). The app obtains the position coordinates (latitude, longitude, and altitude) of the smartphone from the Global Navigation Satellite System (GNSS), the azimuth and orientation of the device from the magnetic compass and gyro, and detects the person using the RGB camera and LiDAR, and also gets the distance to the person. The distance to the person is also obtained. The application then calculated the position coordinates of the detected person and displayed a translucent bubble (red sphere) with a diameter of 5 meters. The location information of the detected person is shared by Unity's photon plug-in and is superimposed on the same location from another smartphone. The location coordinates of the detected person are uploaded to the data server (Nifty Cloud mobile backend), and the information of the detected person can be viewed from a web browser (Google Map).
Data Server
Web Server 2.5.
As shown in Figure 11 we placed PF, a person assumed to have a fever, in the park. Next, smartphones were placed at PE1, PE2, and PE3. The location of PE1 is in the same park as PF and has no obstacles, so PF can be detected directly, but from the location of PE2 and PE3, PF cannot be detected directly due to the obstacles (height difference and trees).
PF
PE1
PE3
PE2 Figure 13 and 14 show the view of PF from PE2 and PE3. From PE2 and PE3, the PF cannot be seen directly due to the height difference and the trees, respectively. However, a red bubble is displayed at the location of the PF based on the information from PE1, so that the user can recognize the presence of the PF.
A red bubble with a diameter of 5m representing the PF is shown in Figures 12, 13, and 14, but these bubbles are not occluded by obstacles such as the ground or trees in real space, making it difficult to get a sense of distance. In the simulator, the 3D geometry of the entire townscape is known, so the bubbles can be occluded, but the prototype system cannot measure the 3D geometry of the real world, so occlusion is not possible. The LiDAR in the iPhone13 can measure real-world 3D geometry at close range (about 15m). The AR Foundation tool provides an Automatic Environment Occlusion function using LiDAR, but it does not work on objects at long distances as in this experiment. Therefore, We generally assume that there are fewer obstacles in the sky, we floated the bubble at 30 m above the detected PF and displayed a thin cylinder up to the bubble. The results are shown in Figures 15 and 16.
Figure 17 shows the results of mapping the PFs detected by PE1 on Google Map (aerial photo) with images. This allows users in remote locations to determine the location of PFs.
We have developed a prototype system that can detect and avoid criminal damage and unpleasant events in advance, using a simulator and a smartphone. Through experiments, we confirmed that the system works and that the information about the detected persons can be shared. 3.1.
Once the simulator was run and the locations of PFs, PDs, and PSs were identified on the 2D map, we felt that "we should avoid that dangerous area where PFs and PDs are densely located," and did not feel that much need for wide-area safe route search. It may be sufficient to have information that "that area is dangerous". In addition, when we had to pass by a PF or PD, the AR display of social distance in the form of a bubble made us feel secure, but when we had to pass through their bubble, we felt great stress despite it being a simulation. On the other hand, when there was a PS that was supposed to be a police officer, etc., I was able to move with ease even if there was a PD. However, the bubble was often invisible behind a building, and when we turned a corner, we were sometimes surprised when the bubble suddenly appeared, and we felt that the AR display needed to be improved in this regard. The edge device on the simulator is an ideal device to detect PF, PD, and PS within a radius of 30m without error, but in reality, such an ideal device is difficult to implement. Therefore, in the future, we would first like to narrow down the detection target to people with fever and conduct experiments in the real world to determine the social distance from people with a fever to verify the effectiveness of the system. We would also like to study a safer route-finding method by incorporating PF, PD, and PS path forecasts into the wide-area and narrow-area safe route-finding method proposed in this study.
In this paper, we have evaluated the effectiveness of the system subjectively, but in the future, we would like to evaluate the effectiveness of the system quantitatively. For example, as indicators of quantitative evaluation, we are considering the frequency with which PDs can not be avoided, how many were able to take the safe route, the average of the shortest distance to a PD, and the distance increased by the safe route proposed by the system.
As mentioned in section 3.5, the bubble representing the location of the detected PF is not occluded by objects in the real world, making it difficult to get a sense of the distance. We floated the bubble at 30 m above the PF, but the same problem occurs when there is an object in the sky. In the future, we would like to estimate the 3D geometry of the object in front of the PF from the video and perform bubble occlusion. Another issue is the accuracy of GNSS. We experimented in a park with no obstacles in the sky, but the GNSS accuracy was at best about 5 m horizontally and 3 m vertically, with errors of 30 m or more in some situations. Therefore, a method to estimate self-position by measuring the 3D geometry of an object is possible. For example, position estimation can be performed by matching point cloud data obtained by LiDAR sensors installed in recent smartphones with real space, and something similar has already been implemented in automatic driving. For this purpose, it is necessary to convert the entire real space into point cloud data. However, point cloud data in cities and buildings are available through the PLATEAU project [23] promoted by Japan's Ministry of Land, Infrastructure, Transport, and Tourism, and we would like to consider improving the accuracy of self-position estimation using this method in the future.
Through these research activities, we will promote the social implementation of information-sharing and navigation systems that can detect and avoid criminal damage and unpleasant events in advance.