=Paper=
{{Paper
|id=Vol-2308/aviose2019paper01
|storemode=property
|title=MODCAP: A Platform for Cooperative Search and Rescue Missions
|pdfUrl=https://ceur-ws.org/Vol-2308/aviose2019paper01.pdf
|volume=Vol-2308
|authors=Mariana Avezum,Andreas Seitz,Bernd Bruegge
|dblpUrl=https://dblp.org/rec/conf/se/AvezumSB19
}}
==MODCAP: A Platform for Cooperative Search and Rescue Missions==
https://ceur-ws.org/Vol-2308/aviose2019paper01.pdf
MODCAP: A Platform for Cooperative Search and
Rescue Missions
Mariana Avezum Andreas Seitz Bernd Bruegge
Chair for Applied Software Engineering Chair for Applied Software Engineering Chair for Applied Software Engineering
Technische Universität München Technische Universität München Technische Universität München
Munich, Germany Munich, Germany Munich, Germany
m.avezum@tum.de seitz@in.tum.de bruegge@in.tum.de
Abstract—Search and rescue operations after natural disasters as survivor search, equipment delivery, collection of samples
are time critical operations. Unmanned aerial vehicles (UAVs) in the affected areas, and geographic information collection.
present an opportunity to accelerate this execution. We propose MODCAP also considers dynamic flight paths, air traffic
MODCAP, a system whose purpose is to combine new technolo-
gies, such as UAVs and Fog Computing, with traditional rescue control, current weather reports while flying, as well as allow-
techniques resulting in a more effective search for potencial ing for real-time communication between ground station and
survivors in any kind of emergency incident. In this paper we UAVs. MODCAP provides first responders and others with
describe a Fog Computing architecture that deals simultaneously the situational awareness they need to make effective and fast
with real-time constraints and synchronization goals for a hybrid decisions and to forecast how the disaster area will evolve.
human drone collaboration. The architecture allows the drone
fleet to work with ground personnel leading to better response Section II describes related work in both the areas of
time, dynamic adoption of search patterns as well as visualizing distributed drone missions and Fog Computing. Section III
geographic differences before and after an incident. explains the MODCAP architecture in detail and its core
Index Terms—drone, UAV, search-and-rescue, platform, natu- requirements for search and rescue. Section IV presents the
ral disaster, Fog Computing, IoT
implementation of MODCAP with four drones, in a case study
I. I NTRODUCTION that was carried in the mountains. Section V finally concludes
with an outlook of the applicability of using MODCAP with
Natural disaster such as avalanches and landslides are time-
commercial UAVs for search and rescue operations.
critical situations in which every second counts in the search
for remaining survivors. On-site rescue teams often have a II. F OUNDATIONS
limited view of the situation in advance and need to quickly Search and rescue missions after natural disasters tradition-
know where to start their search. Drones, also known as ally relied on “boots on the ground”, or the intense deployment
unmanned aerial vehicles (UAVs), present an opportunity to of people and equipment. This puts many additional people
quickly get an overhead view of the situation. In 2016, over at risk in hazardous terrain and requires the expensive risky
670,000 drones were registered in the United States alone 1 . deployment of large aircraft or helicopters. In [2], Buluvsek
While several of these drones are for personal use, most have presents how UAVs can use different search paths for maximal
on-board cameras, which present a possibility for distributed and efficient area coverage, which is an essential criteria for
search missions when aided by computer vision. search and rescue operations.
To do so, we use Fog Computing, [1] an architecture Mohamed et al. describe UAVFog, a UAV-based Fog Com-
that enables both intensive computing processing on cloud puting platform [3], the authors suggest that UAVs could be
components as well as real-time communication and synchro- used as Fog nodes and then communicate and interact with IoT
nization with the edge devices. Drones in the field can then devices. MODCAP differs in such that we design a dynamic
quickly react to new information about the environment. By system in which drones can interact both as Fog nodes and
synchronizing different camera feeds through Fog Computing, as edge devices. Similarly to MODCAP, the authors mention
faster searches after natural disasters is enabled. application scenarios for UAVFog in disaster control such as
In this paper we present MODCAP, a Multi Operational earthquake, volcanic eruptions, bush fires, floods or terrorist
Drone Collaboration Platform, based on a Fog architecture. attacks. We extend the application scenarios by avalanches and
MODCAP manages multiple drones for areas affected by a landslides. MODCAP can also be used in order to detect area
disaster. MODCAP also keeps all involved search parties syn- changes at an early stage and thus avoid further disasters.
chronized about the search status. It partitions the search areas UAVFog has been prototypically implemented. However, [3]
into different sections and assigns each of them to a different does not state that the UAVFogs actually flew. The components
drone. MODCAP supports different types of missions, such relevant to the authors were implemented while the remaining
1 Source: https://www.openfogconsortium.org/fog-computing-fog- was simulated. The system was subjected to a quantitative
networking-crucial-to-commercial-drones/ analysis of the response times. This highlights interesting
AvioSE 2019: 1st Workshop on Avionics Systems and Software Engineering @ SE19, Stuttgart, Germany 63
findings but gives no indication of the actual applicability in
the described scenarios.
In [4], Yang and his co-authors emphasize the importance
of IoT technologies on emergency response operations. The
authors claim that IoT technology has many positive effects
on the various phases of rescue operations. Not only does it
promote cooperation between the various participating orga- Fig. 1. Different possible search paths. Adapted from [8]
nizations, it also improves situational awareness and allows
full visibility of the emergency forces and their remaining re- calculated path to be obstructed. For example, an avalanche or
sources, making operations faster, more efficient and effective. moved tree may cause the drone to have to change its altitude,
We confirm this and believe that UAVs and Fog Computing or make it impossible to continue on the predicted path. In this
offer more opportunities for effective collaboration. situation, the MODCAP system should be informed, and a new
Mayer et al. describe in [5] a social sensing service that is path should be calculated.
based on a Fog Computing architecture and therefore the avail- New paths may also need to be calculated if the identi-
ability in harsh environments where no Internet connection is fication of an object needs more precise data. While MOD-
available. They describe an architecture for the interplay of CAP should be able to identify any object relevant for the
sensors, Fog and cloud components and their interfaces, and search operation, an important classification are humans, both
mention drones as possible Fog nodes, similar to MODCAP’s survivors and victims of the natural disaster. In a search and
approach. However, it is not clear from the paper whether the rescue scenario UAVs collect images from above and far away,
system has been implemented and tested in this way. making the recognition harder than those possible with modern
In [6], Wang et al. bring together drones and Fog Computing deep learning techniques.
to record sports events from different perspectives with several To achieve these requirements, the flexibility and availability
camera quipped drones. They are particularly interested in advantages of commercial drones and the Fogxy architecture
orchestrating multiple drones in real-time to capture fast and [9] are leveraged. The biggest advantage that drones offer for
dynamic sports scenes. The foundations of the work in [6] search and rescue scenarios is their mobility. By combining
were published in [7]. The authors focus on adaptive video several drones, a network of flexible field devices can be
streaming algorithms and not yet on Fog Computing as an created. This combination is easily enabled by the hierarchical
architectural solution to the problem. structure of the Fogxy architecture, which allows different
roles for the drones. It enables the fast integration of additional
III. MODCAP S YSTEM drones that become available during the search operations.
A. UAVs and Fog for Search and Rescue By assigning different search missions to available drones,
the MODCAP system allows the parallel execution of search
The MODCAP system has three requirements:
efforts. As soon as any of the drones detects a relevant
1) Dynamically calculate individual search patterns based object or potential survivors in their search areas, the human
on available drones operator who initialized the mission is informed. Additionally,
2) Search path adaptation based on geometry of the area depending on the location of the victim, the same or other
3) Human operators should be informed of the search status UAVs could be faster to provide emergency support equipment
It is important for the search path calculation to be dynamic, than teams on the ground. This different mission assignment
due to the possibility that new UAVs register themselves with between drones and humans is the main reason why the system
the system after the start of the operation. The new drone described is highly cooperative.
would then receive a search area not yet covered by the ones
already in operation, and the previous UAVs would reduce B. Architecture
their search areas, and thus be faster. The MODCAP architecture offers several advantages. The
When calculating these paths, it is important to note that components in each layer is shown in Figure 2.
different flight patterns are possible (see Figure 1). Whenever By having a Fog layer directly on the affected area, not
a single UAV is available, MODCAP uses a creeping line only do we ensure that the UAVs do not need any internet
pattern, as it will search for the entire area. However, if several connection, but also, new UAVs that become available after the
drones are available, we use a sector search pattern, for it is start of the mission can be integrated to the ongoing operation.
easier to divide the area. This is done with the orchestration drone, who can commu-
While Figure 1 shows the search paths in a 2D-view, the nicate directly with other mission drones in the field, and at
vertical component is critical in high slopes areas, such as the same time is close enough to the orchestration gateway,
alpine montains where this system was tested. To take this into to ensure data synchronisation both to the cloud server, and
consideration, MODCAP uses Geographic Information System to the data aggregator, who informs the orchestration drone if
(GIS) data of the inspected area. any geographic data changes are captured. Smart devices such
Furthermore, these search paths may need to be dynamically as smart watches or phones used by victims can also help the
adjusted if a change caused by the natural disaster causes the MODCAP system locate any survivors.
AvioSE 2019: 1st Workshop on Avionics Systems and Software Engineering @ SE19, Stuttgart, Germany 64
Fig. 2. Top-Level Design of MODCAP
The hierarchical nature of Fog Computing allows us to meet geographic information, or transport equipment, depending on
the challenges and enable real-time video processing through the drone’s capabilities. Combined, this forms a registration
drone orchestration. We applied the Fogxy pattern [9], [10] to protocol that allows drones to register even after the search
realize MODCAP and distribute the components over 3 layers: operation has already begun. Figure 3 shows the system
Cloud Layer: for computational intensive aspects of interface, with the search path assigned to a drone, and its
the system. This includes rendering aerial imagery into a current location (seen in Figure 3 by the red dot).
geographic information system data, search path calculation
and distribution, as well as the use of a pre-trained machine IV. C ASE S TUDY
learning model to identify people in the area. This layer does
initially require an internet connection for downloading any
previously available geographic information data, as well as
for machine learning training. This data is then transmitted
to the Fog Layer through the synchronisation link to the
orchestration gateway.
Fog Layer: serves as an intermediary between the cloud
and the field. This layer is responsible for data and task
synchronization between all the components of the system.
It ensures that each registered drone gets a specific and
different search area, as well as that any detected people get
immediately communicated to the emergency operation center.
As an affected disaster area may suffer internet connection
Fig. 3. User interface showing the search path and location for an individual
limitations, this layer only requires a connection to the server drone. The slanted line in the beginning shows the drone is directed to fly in
on the cloud layer, and not to any external online services. a direction away from the other drones in the area.
Field Layer: takes place in the physical field affected by
the natural disaster. It is responsible for following the orders The MODCAP system was implemented and tested
from the Fog layer, such as searching for any survivors or with different types of personal drones: The DJI Phan-
collecting data about the impact of the disaster. tom 4 Pro, Phantom 3 Standard, and Matrice Series
Upon notification of a disaster by an operator, the system (https://www.dji.com/de/products/drones). Each drone was
opens the possibility for any drone in the area to register connected to the Fog layer through a local Wi-Fi network. The
itself in the search rescues. Depending on the number of Fog layer is responsible for the different mission calculation
registrations, it calculates different areas and missions for each and assignments, explained below. The server providers in the
of the respective UAVs. Each UAV receives an individual flight Fog layer were composed of not only tablets which allowed for
path, with a mission to scan for objects and survivors, gather fast and synchronized drone communication, but also a data
AvioSE 2019: 1st Workshop on Avionics Systems and Software Engineering @ SE19, Stuttgart, Germany 65
aggregation server, responsible for processing the geographic Important aspects of the MODCAP system rely on the Fog
data incoming from the UAVs and calculating the difference architecture. It connects MODACP to any available UAVs, as
caused by the natural disaster. well as calculate and assign missions. Beyond area searching,
To efficiently integrate these different components and be these include emergency equipment delivery. Modular com-
able to iteratively test each part of the system, agile method- ponents which can be attached to the drones allow different
ologies were used. While these have often been adopted equipment to be handled accordingly to the target area. These
for software engineering, the process poses a challenge for components are important for the application of MODCAP to
hardware components, where late changes are harder and more larger emergency and search and rescue operations.
complicated. One technique that can make agile hardware The prototype implemented in our case study shows promis-
development easier, is the use of modular components [11]. ing results as far as mission distribution and search operations
In a context where each disaster is unique, modular 3D- go, however in practice, hardware limitations such as energy
printed components offer a fast mechanism to integrate new supply restricting flight time could play a key role in the
possibilities. For example if a sample collection of the area applicability of the system. Other possible technical limitations
of the natural disaster is needed, a modular grabber can be include personal UAVs not possessing cameras with a resolu-
quickly printed and simply connected to the common interface. tion high enough to detect people on the ground. While this
Furthermore, these modular and agile aspects were relevant could be overcome by reducing the altitude of the drone, this
when considering possible drones. While most commercial may not always be possible. Furthermore, certain operations
UAVs have on board cameras needed for assisted people aspect of MODCAP can greatly vary from one country to the
location, the same can not be said about payload capabilities. other due to drones and resources availability. Assessing these
One advantage of 3D-printed plastic components is their circumstances is the next step for our suggested system.
comparable light weight, which makes them easier to transport While these may not have been overcome yet, the integra-
on board of commercial UAVs. These drones, on the other tion of different search phases, geographic data analysis and
hand, usually require flat surfaces to land on, and attaching rescue missions illustrates how UAVs and human can increase
something underneath them can imply making it harder to take their capabilities by collaborating.
off and land. As a workaround for this problem, 3D-printed
R EFERENCES
components were only attached to the UAVs while these were
already in flight or hover mode and removed before landing. [1] O. Consortium et al., “Openfog reference architecture for fog comput-
ing,” Architecture Working Group, 2017.
While this solution worked on a prototype level, a mature [2] B. J. Bulušek, “Coverage path planning in non-convex polygon areas for
system could surely improve this aspect. orthophotomap creation using uavs,” in Conference on. IEEE, vol. 111,
p. 117, 2015.
One advantage of MODCAP is the ability to distribute [3] N. Mohamed, J. Al-Jaroodi, I. Jawhar, H. Noura, and S. Mahmoud,
several UAVs among different search paths, which decreases “Uavfog: A uav-based fog computing for internet of things,” in 2017
the search duration. The search algorithm chosen by the IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced &
Trusted Computed, Scalable Computing & Communications, Cloud &
system depends on the total area to be overflown and the Big Data Computing, Internet of People and Smart City Innovation
number of drones available for the complete mission. (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), IEEE, 2017.
In addition to the mission distribution, another important [4] L. Yang, S.-H. Yang, and L. Plotnick, “How the internet of things
technology enhances emergency response operations,” Technological
aspect for MODCAP is the human interaction between system, Forecasting and Social Change, vol. 80, no. 9, pp. 1854–1867, 2013.
drones and people serving the area. Traditionally, search and [5] R. Mayer, H. Gupta, E. Saurez, and U. Ramachandran, “The fog
rescue missions have employed transceivers on the ground to makes sense: Enabling social sensing services with limited internet
connectivity,” in Proceedings of the 2Nd International Workshop on
locate any survivors. The interference caused between these Social Sensing, SocialSens’17, (New York, NY, USA), pp. 61–66, ACM,
devices and the electronic on board of most UAVs makes it 2017.
hard to pair these two technologies, but in no way diminishes [6] X. Wang, A. Chowdhery, and M. Chiang, “Networked drone cameras
for sports streaming,” in 2017 IEEE 37th International Conference on
the value offered by the radio devices. By having on ground Distributed Computing Systems (ICDCS), pp. 308–318, June 2017.
search teams not only operate such transceivers, but also limit [7] X. Wang, A. Chowdhery, and M. Chiang, “Skyeyes: Adaptive video
the total search area, quickly communicate between affected streaming from uavs,” in Proceedings of the 3rd Workshop on Hot Topics
in Wireless, HotWireless ’16, (New York, NY, USA), pp. 2–6, ACM,
locations, access the rescue operations risk and approach, the 2016.
MODCAP system offers and requires a high collaboration [8] M. Eldridge, J. Harvey, T. Sandercock, and A. Smith, “Design and build
between UAVs and on ground search teams. a search and rescue uav,” Univ. Adelaide, Adelaide, Australia, 2009.
[9] A. Seitz, F. Thiele, and B. Bruegge, “Fogxy - An Architecural Pattern
for Fog Computing,” in Proceedings of the 23Nd European Conference
V. C ONCLUSION on Pattern Languages of Programs, EuroPLoP ’18, (New York, NY,
USA), ACM, 2018.
In conclusion, the MODCAP system offers several advan- [10] A. Seitz, F. Thiele, and B. Bruegge, “Focus group: Patterns for fog
computing,” in Proceedings of the 22Nd European Conference on
tages compared to traditional search and rescue operations. Pattern Languages of Programs, EuroPLoP ’17, (New York, NY, USA),
Not only does the leverage of multiple drones allow for a pp. 37:1–37:2, ACM, 2017.
bigger search area which gets covered faster, but also these [11] L. Boskovski and M. Avezum, “Combining hardware and software
development: A case study on interdisciplinary teaching projects,” in
commercial drones enable access to aerial cameras, which ISEE@ SE, 2018.
provide a higher flexibility in observing hard-to-reach areas.
AvioSE 2019: 1st Workshop on Avionics Systems and Software Engineering @ SE19, Stuttgart, Germany 66