-Search and rescue operations after natural disasters are time critical operations. Unmanned aerial vehicles (UAVs) present an opportunity to accelerate this execution. We propose MODCAP, a system whose purpose is to combine new technologies, such as UAVs and Fog Computing, with traditional rescue techniques resulting in a more effective search for potencial survivors in any kind of emergency incident. In this paper we describe a Fog Computing architecture that deals simultaneously with real-time constraints and synchronization goals for a hybrid human drone collaboration. The architecture allows the drone fleet to work with ground personnel leading to better response time, dynamic adoption of search patterns as well as visualizing geographic differences before and after an incident. Index Terms-drone, UAV, search-and-rescue, platform, natural disaster, Fog Computing, IoT
Natural disaster such as avalanches and landslides are timecritical situations in which every second counts in the search for remaining survivors. On-site rescue teams often have a limited view of the situation in advance and need to quickly know where to start their search. Drones, also known as unmanned aerial vehicles (UAVs), present an opportunity to quickly get an overhead view of the situation. In 2016, over 670,000 drones were registered in the United States alone 1. While several of these drones are for personal use, most have on-board cameras, which present a possibility for distributed search missions when aided by computer vision.
To do so, we use Fog Computing, [
In this paper we present MODCAP, a Multi Operational Drone Collaboration Platform, based on a Fog architecture. MODCAP manages multiple drones for areas affected by a disaster. MODCAP also keeps all involved search parties synchronized about the search status. It partitions the search areas into different sections and assigns each of them to a different drone. MODCAP supports different types of missions, such 1Source: https://www.openfogconsortium.org/fog-computing-fognetworking-crucial-to-commercial-drones/ as survivor search, equipment delivery, collection of samples in the affected areas, and geographic information collection.
MODCAP also considers dynamic flight paths, air traffic control, current weather reports while flying, as well as allowing for real-time communication between ground station and UAVs. MODCAP provides first responders and others with the situational awareness they need to make effective and fast decisions and to forecast how the disaster area will evolve.
Section II describes related work in both the areas of distributed drone missions and Fog Computing. Section III explains the MODCAP architecture in detail and its core requirements for search and rescue. Section IV presents the implementation of MODCAP with four drones, in a case study that was carried in the mountains. Section V finally concludes with an outlook of the applicability of using MODCAP with commercial UAVs for search and rescue operations.
Search and rescue missions after natural disasters
traditionally relied on “boots on the ground”, or the intense deployment
of people and equipment. This puts many additional people
at risk in hazardous terrain and requires the expensive risky
deployment of large aircraft or helicopters. In [
Mohamed et al. describe UAVFog, a UAV-based Fog
Computing platform [
UAVFog has been prototypically implemented. However, [
In [
Mayer et al. describe in [
In [
1) Dynamically calculate individual search patterns based on available drones 2) Search path adaptation based on geometry of the area 3) Human operators should be informed of the search status It is important for the search path calculation to be dynamic, due to the possibility that new UAVs register themselves with the system after the start of the operation. The new drone would then receive a search area not yet covered by the ones already in operation, and the previous UAVs would reduce their search areas, and thus be faster.
When calculating these paths, it is important to note that different flight patterns are possible (see Figure 1). Whenever a single UAV is available, MODCAP uses a creeping line pattern, as it will search for the entire area. However, if several drones are available, we use a sector search pattern, for it is easier to divide the area.
While Figure 1 shows the search paths in a 2D-view, the vertical component is critical in high slopes areas, such as alpine montains where this system was tested. To take this into consideration, MODCAP uses Geographic Information System (GIS) data of the inspected area.
Furthermore, these search paths may need to be dynamically adjusted if a change caused by the natural disaster causes the calculated path to be obstructed. For example, an avalanche or moved tree may cause the drone to have to change its altitude, or make it impossible to continue on the predicted path. In this situation, the MODCAP system should be informed, and a new path should be calculated.
New paths may also need to be calculated if the identification of an object needs more precise data. While MODCAP should be able to identify any object relevant for the search operation, an important classification are humans, both survivors and victims of the natural disaster. In a search and rescue scenario UAVs collect images from above and far away, making the recognition harder than those possible with modern deep learning techniques.
To achieve these requirements, the flexibility and availability
advantages of commercial drones and the Fogxy architecture
[
By assigning different search missions to available drones, the MODCAP system allows the parallel execution of search efforts. As soon as any of the drones detects a relevant object or potential survivors in their search areas, the human operator who initialized the mission is informed. Additionally, depending on the location of the victim, the same or other UAVs could be faster to provide emergency support equipment than teams on the ground. This different mission assignment between drones and humans is the main reason why the system described is highly cooperative.
The MODCAP architecture offers several advantages. The components in each layer is shown in Figure 2.
By having a Fog layer directly on the affected area, not only do we ensure that the UAVs do not need any internet connection, but also, new UAVs that become available after the start of the mission can be integrated to the ongoing operation. This is done with the orchestration drone, who can communicate directly with other mission drones in the field, and at the same time is close enough to the orchestration gateway, to ensure data synchronisation both to the cloud server, and to the data aggregator, who informs the orchestration drone if any geographic data changes are captured. Smart devices such as smart watches or phones used by victims can also help the MODCAP system locate any survivors. Fig. 2. Top-Level Design of MODCAP
The hierarchical nature of Fog Computing allows us to meet
the challenges and enable real-time video processing through
drone orchestration. We applied the Fogxy pattern [
Cloud Layer: for computational intensive aspects of the system. This includes rendering aerial imagery into a geographic information system data, search path calculation and distribution, as well as the use of a pre-trained machine learning model to identify people in the area. This layer does initially require an internet connection for downloading any previously available geographic information data, as well as for machine learning training. This data is then transmitted to the Fog Layer through the synchronisation link to the orchestration gateway.
Fog Layer: serves as an intermediary between the cloud and the field. This layer is responsible for data and task synchronization between all the components of the system. It ensures that each registered drone gets a specific and different search area, as well as that any detected people get immediately communicated to the emergency operation center. As an affected disaster area may suffer internet connection limitations, this layer only requires a connection to the server on the cloud layer, and not to any external online services.
Field Layer: takes place in the physical field affected by the natural disaster. It is responsible for following the orders from the Fog layer, such as searching for any survivors or collecting data about the impact of the disaster.
Upon notification of a disaster by an operator, the system opens the possibility for any drone in the area to register itself in the search rescues. Depending on the number of registrations, it calculates different areas and missions for each of the respective UAVs. Each UAV receives an individual flight path, with a mission to scan for objects and survivors, gather geographic information, or transport equipment, depending on the drone’s capabilities. Combined, this forms a registration protocol that allows drones to register even after the search operation has already begun. Figure 3 shows the system interface, with the search path assigned to a drone, and its current location (seen in Figure 3 by the red dot).
The MODCAP system was implemented and tested with different types of personal drones: The DJI Phantom 4 Pro, Phantom 3 Standard, and Matrice Series (https://www.dji.com/de/products/drones). Each drone was connected to the Fog layer through a local Wi-Fi network. The Fog layer is responsible for the different mission calculation and assignments, explained below. The server providers in the Fog layer were composed of not only tablets which allowed for fast and synchronized drone communication, but also a data aggregation server, responsible for processing the geographic data incoming from the UAVs and calculating the difference caused by the natural disaster.
To efficiently integrate these different components and be
able to iteratively test each part of the system, agile
methodologies were used. While these have often been adopted
for software engineering, the process poses a challenge for
hardware components, where late changes are harder and more
complicated. One technique that can make agile hardware
development easier, is the use of modular components [
Furthermore, these modular and agile aspects were relevant when considering possible drones. While most commercial UAVs have on board cameras needed for assisted people location, the same can not be said about payload capabilities. One advantage of 3D-printed plastic components is their comparable light weight, which makes them easier to transport on board of commercial UAVs. These drones, on the other hand, usually require flat surfaces to land on, and attaching something underneath them can imply making it harder to take off and land. As a workaround for this problem, 3D-printed components were only attached to the UAVs while these were already in flight or hover mode and removed before landing. While this solution worked on a prototype level, a mature system could surely improve this aspect.
One advantage of MODCAP is the ability to distribute several UAVs among different search paths, which decreases the search duration. The search algorithm chosen by the system depends on the total area to be overflown and the number of drones available for the complete mission.
In addition to the mission distribution, another important aspect for MODCAP is the human interaction between system, drones and people serving the area. Traditionally, search and rescue missions have employed transceivers on the ground to locate any survivors. The interference caused between these devices and the electronic on board of most UAVs makes it hard to pair these two technologies, but in no way diminishes the value offered by the radio devices. By having on ground search teams not only operate such transceivers, but also limit the total search area, quickly communicate between affected locations, access the rescue operations risk and approach, the MODCAP system offers and requires a high collaboration between UAVs and on ground search teams.
In conclusion, the MODCAP system offers several advantages compared to traditional search and rescue operations. Not only does the leverage of multiple drones allow for a bigger search area which gets covered faster, but also these commercial drones enable access to aerial cameras, which provide a higher flexibility in observing hard-to-reach areas.
Important aspects of the MODCAP system rely on the Fog architecture. It connects MODACP to any available UAVs, as well as calculate and assign missions. Beyond area searching, these include emergency equipment delivery. Modular components which can be attached to the drones allow different equipment to be handled accordingly to the target area. These components are important for the application of MODCAP to larger emergency and search and rescue operations.
The prototype implemented in our case study shows promising results as far as mission distribution and search operations go, however in practice, hardware limitations such as energy supply restricting flight time could play a key role in the applicability of the system. Other possible technical limitations include personal UAVs not possessing cameras with a resolution high enough to detect people on the ground. While this could be overcome by reducing the altitude of the drone, this may not always be possible. Furthermore, certain operations aspect of MODCAP can greatly vary from one country to the other due to drones and resources availability. Assessing these circumstances is the next step for our suggested system.
While these may not have been overcome yet, the integration of different search phases, geographic data analysis and rescue missions illustrates how UAVs and human can increase their capabilities by collaborating.