=Paper=
{{Paper
|id=None
|storemode=property
|title=Disaster Management Tool (DMT) - Usability Engineering, System Architecture and Field Experiments
|pdfUrl=https://ceur-ws.org/Vol-953/paper3.pdf
|volume=Vol-953
}}
==Disaster Management Tool (DMT) - Usability Engineering, System Architecture and Field Experiments==
https://ceur-ws.org/Vol-953/paper3.pdf
Disaster Management Tool (DMT) –
Usability Engineering, System Architecture
and Field Experiments
Martin Frassl, Michael Lichtenstern, and Michael Angermann
Institute of Communications and Navigation,
German Aerospace Center (DLR)
82234 Wessling, Germany
{martin.frassl,m.lichtenstern,michael.angermann}@dlr.de
Abstract. The Disaster Management Tool (DMT) supports informa-
tion management during crises. It has been designed to support field
workers, on-site coordination centers and headquarters by facilitating an
efficient flow of information between them. In this paper we describe
the functionality and architecture of the DMT and give insight into our
development process over the last four years. The DMT has undergone
extensive field experiments during a series of Assessment Mission Courses
(AMCs) for experts in coordination and assessment within the European
Civil Protection Mechanism. Results and lessons learned from these ex-
periments are presented.
1 Introduction
Today’s international response to large scale crises is amazingly rapid and effec-
tive. To a large extend this is owed to institutions such as the European Com-
mission’s Monitoring and Information Center (MIC) based in Brussels or the
United Nations’ Office for the Coordination of Humanitarian Affairs (OCHA)
based in Geneva, which are important information hubs and help to coordinate
the international response of many governmental and non-governmental relief
organizations. International cooperation does not only increase the amount of
available resources, but also requires a significant amount of coordination and
communication by relief experts in the field. These experts have a proven track
record that they are able to cope with complex and uncertain information, even
with basic communication means, such as voice communication and basic of-
fice computing software or even pen and paper. Nevertheless, several research
strands, such as ad hoc and sensor networks, social computing, pervasive com-
puting or combinations as in ambient intelligence are motivated to investigate
the disaster management domain by the hope that their particular contributions
could improve relief efforts. We are inspired by the skills of today’s disaster man-
agement experts and the potential of the aforementioned technologies to combine
them in a holistic fashion that builds on existing workflows and organizational
structures. While we embrace the capabilities we may gain from mobile and em-
bedded sensors and computational power, ubiquitous internet connectivity and
2 Frassl et al.
Fig. 1. DMT hardware and user interface.
vast amounts of information and cognitive resources from crowdsourcing and
social networks, we are also concerned that exactly these assets are likely to be
affected and potentially unavailable in disaster situations. Hence, our research
focusses on how to use these technologies without critically relying on them. In
previous work we have investigated the specific requirements for a tool to assist
disaster management [5]. In the following paper we report on our work towards
a software prototype that helps to study how experts use such a tool under field
conditions. We briefly describe the application domain the system is intended
to be used in. We describe the functionality and the system architecture of the
DMT. Finally, we present and discuss evaluation feedback of users who worked
with the DMT during several training missions.
1.1 Application Domain Background
Europe has established the European Civil Protection Mechanism (EUCP mech-
anism), a process of cooperation during emergencies. This mechanism can be
activated by participating states for missions inside and outside of Europe. In
such a case the participating countries join their efforts to share resources and
increase efficiency [1]. Cooperation between organizations from several countries
and a central information and coordination center in Brussels requires a common
picture of the situation and thus information sharing across organizational and
geographical borders. Prerequisite for a successful mission is a rapid assessment
of the specific needs for the disaster response. Typically, several partners, both
from the local emergency management agencies as well as international assess-
ment and coordination experts, perform the assessment of a situation. Fast and
reliable collection and exchange of findings are important to select the best-
suited assets for relief. Assessment experts have already a variety of technical
tools available: GPS navigation devices, satellite communication terminals, elec-
tronic maps or web sites filled with information about the situation before the
disaster. Working with these tools requires experience and time, with especially
the latter being a scarce resource during a mission. Time pressure and other
stressors tend to lower the frustration thresholds of users. To support disaster
Disaster Management Tool (DMT) 3
management experts in the field, the UN system and the EUCP system have
introduced dedicated support units, which cover information and communica-
tion technology and a portfolio of additional tasks such as transportation, camp
building, subsistence and administration. In UNDAC (United Nations Disaster
Assessment and Coordination) missions, this role is frequently assigned to the
International Humanitarian Partnership (IHP), an association of organizations
from mainly Scandinavian countries. In EUCP missions TAST teams (Technical
Assistance and Support) are available in the form of EUCP modules. Further-
more, several non-governmental organizations (NGOs) provide assistance for a
specific field, like Mapaction for the in situ production of maps or Ericsson Re-
sponse for communication services. Nevertheless, basic knowledge like navigation
with a GPS device or setting up a BGAN satellite terminal is expected from a
coordination and assessment expert.
1.2 Related Work
The difficulty of the challenges in the disaster management domain have at-
tracted a growing number of researchers that contribute towards several of the
involved problems. Meissner et al. have investigated a range of requirements and
design challenges for an integrated disaster management communication and
information system [7]. Furthermore, Meissner et al. drafted high-level archi-
tectures for the communications and personal task scheduling subsystems. The
need for rapid configuration of deployed network components has been recog-
nized early and several groups have proposed to use rapidly deployable wire-
less networks for disaster response to fill the gap of potentially disaster-affected
communication infrastructures. Based on basic connectivity, autonomous peer-
to-peer data exchange is an important step towards decentralization and robust-
ness [3]. Some research groups work on transferring today’s workflows in disaster
management to the digital domain [6]. Others strive to use new technologies and
adapt them to the use in disaster management. A prominent example is the
Ushahidi project, which aims at employing Web 2.0 technologies [9]. The work
of the Sahana foundation on the application layer has achieved significant im-
pact by applying and customizing available software components to the specific
needs during a disaster [4].
2 Development Process
As described in a previous work-in-progress paper [5], we followed a primarily
user oriented development paradigm. During the entire development process,
prospective users have been involved at several stages to increase the usefulness
and acceptance of the system, and to provide us with feedback and their wishes
for features. User-centered development does not mean to only translate the
user’s exiting processes identically to a digital version. Additionally, we strive
to introduce new ideas and to adapt these to the user’s needs. During the last
four years, the evolving prototypes of the system have been tested and evaluated
4 Frassl et al.
by users, and their feedback has been reviewed and directly included into new
developments. Details about our requirement analysis and development process,
i.e. the Adaptive Frequency Spiral Model (AFSM), a modified version of Boehm’s
well-known spiral model, specifically tailored to the disaster management domain
can be found in [5]. During the last years, we had the chance to work with
different groups of end users, mainly assessment experts and TAST members.
Up to now the DMT has been presented to and used by 89 participants of the
Assessment Mission Course (AMC) and 13 participants of the Staff Management
Course (SMC) - both courses are part of the European Civil Protection Training
Program - plus approximately 40 participants of the TAST training courses of
the German Federal Agency for Technical Relief (THW), and 18 participants of
an international training in the context of the EU LIMES project.
2.1 Timeline
In the early phase of the DMT’s development, we emphasized the collection of
requirements and the analysis of the processes in disaster management opera-
tions. The temporal evolution of the added functionality can be seen in Fig. 2.
The participation at the operations on the G8 Summit in Heiligendamm in June
2007 and the INSARAG certification of a THW Heavy Urban Search and Res-
cue (USAR) team in August 2007 in Hoya were important steps to get a basic
understanding of tasks and operational procedures. We observed workflows and
conducted many informal interviews about information management in disaster
relief operations. At the end of a first cycle of requirements analysis, we par-
ticipated at the second AMC in the 5th training cycle in November 2007 (i.e.
5AMC2), where an initial set of functional and non-functional requirements for
a Disaster Management Tool evolved [5]. Based on these requirements a first
G8 Hoya 5AMC2 6AMC1 7AMC1 7AMC2 7AMC3 8AMC1 8AMC2 8AMC3 9AMC1
2007 2008 2009 2010 2011
Point Of Interest (POI) form factor software architecture usability engineering multi mission coordinate converter
map management 3D graphics UI redesign shape mangement remodel gateway remodel network
distributed network bookmarks (views)
stability review architecture
Fig. 2. Timeline of the DMT development
prototype had been developed and implemented by the time of 6AMC1 in June
2008. Most of the basic concepts which are still valid in the current DMT ver-
sion, such as the distributed network synchronization of the data or the spatial
data aggregation in a Point Of Interest (POI) have been used here for the first
time. In this early stage of development, the hardware composition, i.e., a box
containing a small computer, a touch screen, several sensors, satellite terminal,
rechargeable batteries, chargers etc. added up to 25 kilograms – too heavy for
Disaster Management Tool (DMT) 5
mobile operations. In addition, a proprietary development of a 3D globe visual-
ization turned out to be slow and unstable. Nevertheless we received generally
good user feedback which motivated us to develop a completely new system,
including a redesigned user interface in which we replaced the initial 3D globe
visualization with NASA World Wind Technology [2]. We reduced the form fac-
tor by using smaller boxes and replacing the computer and the separate touch
screen with an off-the-shelf laptop. Additionally, the functionality was extended
by adding several new features like placemarks, for the next system iteration in
June 2009, at the 7AMC1. Beginning from this stage, the user experience was
satisfactory, but the underlying software architecture became more and more
cluttered. For the next AFSM cycle we concentrated on a review of the over-
all system architecture. Furthermore, we modified the user interface (UI) for
increased usability and redesigned the underlying distributed network for data
synchronization. This version has been presented and tested by the course par-
ticipants at the subsequent AMC (7AMC2) in November 2009. More features
have been included and evaluated in every iteration. To obtain quantitative user
feedback we deployed a usability engineering process based on questionnaires
for the 7AMC3. Initial results revealed a lack of stability. Due to the fact that
there were only two months to the 8AMC1 in June 2010, we concentrated on
this issue. For the AMC in November 2010, we again extended the function-
ality by implementing a multi-mission capability, which enables the system to
concurrently handle multiple missions in parallel.
2.2 Usability Engineering
Our usability engineering process is based on the following methods: participat-
ing user observation, informal interviews and questionnaire based evaluation [8]
[11]. We use the method of participating user observation, i.e. to join in perform-
ing the users’ tasks. In the beginning we used this method to gather initial system
requirements. Now it serves as a feedback channel to study the acceptance of im-
plemented functionality, and to obtain novel ideas and demands for the DMT.
Observing the user in the field (at least during exercises and trainings) gives
insights that are difficult to obtain in simulated environments (e.g. a usability
laboratory). Informal interviews help to constantly improve our understanding
of the users and their experiences with the DMT. This informal feedback chan-
nel revealed many subconscious requirements and weaknesses of the system. To
obtain quantitative user feedback we developed a questionnaire-based evaluation
process to identify strong and weak points of the system and to revise require-
ments. Revising requirements includes the derivation of new requirements and
points out functionality which has not been proven to be particularly useful,
and therefore needs to be redesigned or even removed from the system. The
questionnaire is divided into three parts. The first part is about the background
of the user, including gender, age, expertise as well as computer and mission
experiences. In the second part the user has the possibility to rate experiences
with the DMT on a 5-step scale (strongly disagree, disagree, neutral, agree and
strongly agree). The nine questions presented to the user are:
6 Frassl et al.
1. In my opinion the Disaster Management Tool (DMT) is easy to use.
2. I think the provided services (e.g. Points Of Interest, Map handling, etc.) fit
the requirements for disaster management.
3. The way data is entered into the system is appropriate and efficient.
4. The software provides me with valuable information to fulfill my tasks.
5. I can find the needed functionality, and do not have to consult the trainer.
6. The system performance is adequate and does not slow my work.
7. The system is supporting the relief work and does not distract or limit me
doing my work during relief operations.
8. The Disaster Management Tool increases the situation awareness and there-
fore supports better coordination of relief operations.
9. I would use the DMT-System for my work.
In the third section the user writes free text to suggest missing or unnecessary
functionality and what he or she likes or dislikes about the DMT. Results of the
usability engineering process are included in Section 4.
3 Technical Prototype and Core Functionality
Functional and non-functional requirements for an information management sys-
tem in disaster management have been reported in [5] and have driven the def-
inition of the DMT’s core functionality. The purpose of the DMT is to assist
information management during disaster relief operations. The system’s visual
core component is a dynamic situation map, based on a 3D globe on which
geospatial information of various types are displayed. Examples are vector data
like points of interest, augmented with specific text or imagery information,
polygons to mark a certain area, or rasterized information such as satellite maps
based on images taken before or after the mission or digital elevation models.
Tools to handle the input, output and management of the data are offered. Sev-
eral sensors, such as position and attitude sensors, can be attached and processed
for different purposes such as showing the own position on the map or sending
it to other users. Additionally all other relevant data items in the system are
shared among all connected instances of the DMT, resulting in a distributed, de-
centralized and disruption-tolerant system. Depending on the available network
infrastructure, the best connections are chosen to transmit the information, be
it an ad hoc, infrastructure or satellite connection.
3.1 Modular Architecture
In order to maintain a stable and extendable software architecture the function-
ality of the DMT is partitioned into five modules:
– User Interface for visualization and user input
– Data Hub / Synchronization for managing data objects
– Persistence for storing of data objects
– Network for providing transparent communication
– Sensors (and the affiliated sensor fusion) to manage external hardware
Disaster Management Tool (DMT) 7
The cornerstone of the DMT software architecture is the Data Hub. All informa-
tion, independent from its origin (data storage, network, user input), is passed
through this component. When the user enters information via the UI, data is
received via the network module, or a sensor transmits a new measurement, the
Data Hub decides what to do with it. The data is analyzed and accordingly for-
warded to other modules. The Data Hub module is responsible for ensuring that
new information is synchronized with other DMT instances via the distributed
network. If the user enters new data via the UI, the Data Hub informs the Per-
sistence component and sends a notification via the Network component. If the
network component receives a notification about new data on the other side, the
data is requested and upon reception forwarded to the Persistence and the UI
component. The User Interface is designed under the paradigm of keeping its
complexity to a minimum, offering necessary, but avoiding all nonessential “ex-
pert” functionality. There are mainly two reasons for this approach. Firstly, the
DMT system is generally used by users, who are not working with the system in
their daily work. Secondly, the users use the system in a stressful environment.
Therefore the main interface is condensed to a minimalistic on-screen menu with
the possibility to manage the most important data types and system settings
(Points Of Interest, Shapes, Maps, Bookmarks, Units, System Settings). Beside
the menu, the NASA World Wind globe is the central visualization element,
where all spatial data is shown (see Fig. 1). In the Persistence component, two
main tasks are encapsulated, the reliable storage of all data and the guarantee
of data integrity. After a restart of the DMT system, the stored information is
read from a persistent storage and loaded into the system. The current imple-
mentation is based on the operating system’s file system, which is reliable and
has no further installation requirements. Through the modularized architecture,
encapsulation of the functionality to other modules and clear interfaces, a re-
placement of the underlying information storage technology by other solutions
like a database can be carried out with minimal effort. The Network module
offers an interface for a reliable and efficient data exchange. This module is re-
sponsible to handle network-related tasks, such as finding neighboring hosts and
starting the initial connection procedure, or selecting the appropriate commu-
nication channel (TCP, UDP via Wi-Fi, satellite network, etc.) to an already
known host. Network connections are chosen based on their availability and a
cost function, depending on the data characteristics and the current status of
the system. The Sensors component represents a layer of abstraction for binding
external sensors, such as GPS receivers or a 3D compass to the system. Sensor
input is preprocessed and fused within the Sensors module. Sensor fusion offers
the possibility to combine several sensor inputs to improve the quality of the
output, such as a more precise position by combining several Global Navigation
Satellite Systems (GNSS) and/or acceleration sensors [10]. In order to provide
access to a sensor’s status and measurements or to set parameters for a sensor,
this module has a direct interface with the UI.
8 Frassl et al.
4 Field Experiments and Results
It was very insightful to observe the users using the DMT in the field. Several
problems and gaps have been discovered. Main issues were hardware and stability
problems, environment specific problems, such as direct sunlight exposure and
reduced interaction possibilities (e.g. no mouse) in the field. Some new features
have been implemented after observing the users having problems or wasting
time, for example the need for extended export functionalities, as users still
use their well known software tools and have to follow the predefined reporting
chain, or a coordinate conversion tool to rapidly access different formats of a
coordinate. In general, the users were very motivated to give direct feedback to
the observing developers and many new ideas have been collected this way.
4.1 User Feedback and Empirical Findings
In this paper we analyze the rated second part of our questionnaire (see Section
2.2), which provides quantitative measures of user experiences with the DMT.
We collected data during five AMCs: 7AMC3, 8AMC1, 8AMC2, 8AMC3 and
9AMC1. Each AMC has room for up to 20 participants, who are grouped into
four teams. Thus, each team consists of up to five team members. On each AMC
5
4
3 7AMC3
8AMC1
8AMC2
2
8AMC3
9AMC1
1
0
DMT is easy to use DMT fits the data entering is soŌware provides info trainer needs to be performance adequate increases awarness -> DMT is supporƟng not would use DMT for
requirements appropriate to fulfill task consulted not slowing beƩer coordinaƟon distracƟng work
Fig. 3. Evaluation results: mean values of the user experiences with the DMT
we took part in, we started by giving a general briefing on the DMT to all
participants. Subsequently, we gave a more detailed training to a subset of these
participants (initially one team, in later AMCs up to three teams) on the software
and supported them in using it for the assessments during the three course days.
While we usually gave close support in the first day of the course, we reduced
the support over the following days. The teams typically used the system on
their own on the third day. Within the 7AMC3 the Disaster Management Tool
was used by one team of five participants and by a team of four participants on
8AMC1. In the 8AMC2 three assessments teams used the DMT software during
the training course, which resulted in 13 valid questionnaires. As a result of the
difficulties of monitoring more than one team in the field we evaluated again
one team at the 8AMC3 with five participants and four participants during
Disaster Management Tool (DMT) 9
the 9AMC3. The overall result from the questionnaires indicate encouraging
acceptance by our users. Only two aspects score below 4 for all surveys. These
are question 3, ”The way data is entered into the system is appropriate and
efficient.” and question 5, ”I can find the needed functionality, and do not have
to consult the trainer”. To analyze the reason for the relatively low score on
data entering, we asked the users in informal interviews why they think that
this issue is not ideally solved in the DMT. The result was, that the users are
used to enter text with standard office software (Microsoft Word) and therefore
miss functionality like tagging text by putting bold, italic or underline in the
DMT. Also the possibility of structuring lists with bullet points or indenting
paragraphs is an important feature for them. Currently, the data entering box is
a textfield which does not offer formatting possibilities of text and therefore does
not sufficiently meet this requirement. The score for question 5 can be ascribed
to the fact that the users on the AMC get only 30 minutes of training on the
system and afterwards they have two DMT trainers joining and supporting them
during the assessments. Directly supporting the user in the field increases the
users’ awareness of the DMT’s features, but on the other hand reduces their self-
confidence of using the system without instructions, resulting in the consistently
suboptimal score. At the moment we assume that a change in the training and
support balance as well as compact documentation (”cheat sheets”) on how to
perform specific tasks with the DMT should have a positive effect on this issue.
5 Conclusions and Outlook
The DMT has reached a level of stability that allows its operation by users other
than its developers. Its current set of functionality supports coordination and
assessment experts in their mission-related tasks. This encompasses the efficient
collection, comprehensive displaying and automatic sharing of information. A
range of additional helping functionalities, such as automatic conversion between
coordinate systems or exporting of its data to feed into reports. Our observations
of users working with the tool, informal feedback, as well as formalized feedback
in the form of questionnaires have driven the addition and sometimes removal
of functionalities. While robustness and consolidation of its functionality remain
our foremost priority, we will continue to integrate novel concepts into the DMT.
Many new ideas have been proposed by our users and have been captured in our
usability engineering process. A particularly interesting concept is to leverage
social networks by motivating their users to offer their “cognitive surplus”, to
remotely assist in missions. Experts in specific fields, such as structural engi-
neering, language and cultural expertise could contribute without actually being
present in the field. Organized online communities could accomplish time con-
suming tasks, such as spotting specific features in aerial images or tracing road
networks in a parallel and rapid fashion, literally from their living rooms. This
would take off workload from relief workers and empower the general public to
contribute to disaster relief. When these concepts will mature they will find their
way into the mission-approved version of the DMT.
10 Frassl et al.
6 Acknowledgements
This work is partially funded by the Helmholtz Foundation and the SOCI-
ETIES (Self Orchestrating CommunIty ambiEnT IntelligEnce Spaces) project,
co-funded by the European Commission within FP7. We thank the NASA World
Wind Project, in particular Patrick Hogan and Tom Gaskins for providing the
outstanding World Wind technology. We thank Dr. Susanne Wacht (THW) for
the chance to regularly teach at the THW’s TAST courses and are deeply in-
debted to Claus Höllein (THW), Harm Bastian Harms (Johanniter International
Assistance) and Wolfgang Krajic (synergies) and all participants and trainers for
their feedback, support and the possibility to join the Assessment Mission Course
(AMC), which was and is essential for the development of the DMT.
References
1. 2010/481/EC, Euratom. Commission decision of 29 july 2010 amending decision
2004/277/ec, euratom as regards rules for the implementation of council decision
2007/779/ec, euratom establishing a community civil protection mechanism. Brus-
sels, Belgium, 2008.
2. D. Bell, F. Kuehnel, C. Maxwell, R. Kim, K. Kasraie, T. Gaskins, P. Hogan, and
J. Coughlan. NASA World Wind: Opensource GIS for mission operations. In
Aerospace Conference, 2007 IEEE, pages 1–9, march 2007.
3. A. Capata, A. Marrella, R. Russo, M. Bortenschlager, and H. Rieser. A geo-
based application for the management of mobile actors during crisis situations. In
Proceedings of the 5th ISCRAM, Washington, DC, USA, 2008.
4. M. Careem, C. De Silva, R. De Silva, Raschid, and Weerawarana. Sahana: Overview
of a disaster management system. In Information and Automation, 2006. ICIA
2006. International Conference on, pages 361–366, 2006.
5. M. Frassl, M. Lichtenstern, M. Khider, and M. Angermann. Developing a system
for information management in disaster relief - methodology and requirements.
In Proceedings of the International Community on information systems for crisis
response and management (ISCRAM 2010), Seattle, Washington, USA, 2010.
6. M. Khalilbeigi, D. Bradler, I. Schweizer, F. Probst, and J. Steimle. Towards com-
puter support of paper workflows in emergency management. Proceedings of the
7th International ISCRAM Conference–Seattle, USA, 2010.
7. A. Meissner, T. Luckenbach, T. Risse, T. Kirste, and H. Kirchner. Design chal-
lenges for an integrated disaster management communication and information sys-
tem. In Information System, DIREN 2002 (co-located with IEEE INFOCOM 2002,
2002.
8. J. Nielsen. Usability Engineering. Morgan Kaufmann, 1st edition, Sept. 1994.
9. O. Okolloh. Ushahidi, or ’testimony’: Web 2.0 tools for crowdsourcing crisis infor-
mation. Participatory Learning and Action, 59(1):65–70, 2009.
10. P. Robertson, M. Angermann, and B. Krach. Simultaneous localization and map-
ping for pedestrians using only foot-mounted inertial sensors. In Proceedings of the
11th international conference on Ubiquitous computing, Ubicomp ’09, pages 93–96,
New York, NY, USA, 2009. ACM.
11. B. Shneiderman and C. Plaisant. Designing the user interface : strategies for
effective human-computer interaction. Addison-Wesley, 2010.