Disaster Management Tool (DMT) { Usability Engineering, System Architecture and Field Experiments

Abstract

The Disaster Management Tool (DMT) supports information management during crises. It has been designed to support eld workers, on-site coordination centers and headquarters by facilitating an e cient ow of information between them. In this paper we describe the functionality and architecture of the DMT and give insight into our development process over the last four years. The DMT has undergone extensive eld experiments during a series of Assessment Mission Courses (AMCs) for experts in coordination and assessment within the European Civil Protection Mechanism. Results and lessons learned from these experiments are presented.

Introduction

Today's international response to large scale crises is amazingly rapid and e ective. To a large extend this is owed to institutions such as the European Commission's Monitoring and Information Center (MIC) based in Brussels or the United Nations' O ce for the Coordination of Humanitarian A airs (OCHA) based in Geneva, which are important information hubs and help to coordinate the international response of many governmental and non-governmental relief organizations. International cooperation does not only increase the amount of available resources, but also requires a signi cant amount of coordination and communication by relief experts in the eld. These experts have a proven track record that they are able to cope with complex and uncertain information, even with basic communication means, such as voice communication and basic ofce computing software or even pen and paper. Nevertheless, several research strands, such as ad hoc and sensor networks, social computing, pervasive computing or combinations as in ambient intelligence are motivated to investigate the disaster management domain by the hope that their particular contributions could improve relief e orts. We are inspired by the skills of today's disaster management experts and the potential of the aforementioned technologies to combine them in a holistic fashion that builds on existing work ows and organizational structures. While we embrace the capabilities we may gain from mobile and embedded sensors and computational power, ubiquitous internet connectivity and vast amounts of information and cognitive resources from crowdsourcing and social networks, we are also concerned that exactly these assets are likely to be a ected and potentially unavailable in disaster situations. Hence, our research focusses on how to use these technologies without critically relying on them. In previous work we have investigated the speci c requirements for a tool to assist disaster management [ 5 ]. In the following paper we report on our work towards a software prototype that helps to study how experts use such a tool under eld conditions. We brie y describe the application domain the system is intended to be used in. We describe the functionality and the system architecture of the DMT. Finally, we present and discuss evaluation feedback of users who worked with the DMT during several training missions. 1.1

Application Domain Background Europe has established the European Civil Protection Mechanism (EUCP mechanism), a process of cooperation during emergencies. This mechanism can be activated by participating states for missions inside and outside of Europe. In such a case the participating countries join their e orts to share resources and increase e ciency [ 1 ]. Cooperation between organizations from several countries and a central information and coordination center in Brussels requires a common picture of the situation and thus information sharing across organizational and geographical borders. Prerequisite for a successful mission is a rapid assessment of the speci c needs for the disaster response. Typically, several partners, both from the local emergency management agencies as well as international assessment and coordination experts, perform the assessment of a situation. Fast and reliable collection and exchange of ndings are important to select the bestsuited assets for relief. Assessment experts have already a variety of technical tools available: GPS navigation devices, satellite communication terminals, electronic maps or web sites lled with information about the situation before the disaster. Working with these tools requires experience and time, with especially the latter being a scarce resource during a mission. Time pressure and other stressors tend to lower the frustration thresholds of users. To support disaster management experts in the eld, the UN system and the EUCP system have introduced dedicated support units, which cover information and communication technology and a portfolio of additional tasks such as transportation, camp building, subsistence and administration. In UNDAC (United Nations Disaster Assessment and Coordination) missions, this role is frequently assigned to the International Humanitarian Partnership (IHP), an association of organizations from mainly Scandinavian countries. In EUCP missions TAST teams (Technical Assistance and Support) are available in the form of EUCP modules. Furthermore, several non-governmental organizations (NGOs) provide assistance for a speci c eld, like Mapaction for the in situ production of maps or Ericsson Response for communication services. Nevertheless, basic knowledge like navigation with a GPS device or setting up a BGAN satellite terminal is expected from a coordination and assessment expert. 1.2

Related Work The di culty of the challenges in the disaster management domain have attracted a growing number of researchers that contribute towards several of the involved problems. Meissner et al. have investigated a range of requirements and design challenges for an integrated disaster management communication and information system [ 7 ]. Furthermore, Meissner et al. drafted high-level architectures for the communications and personal task scheduling subsystems. The need for rapid con guration of deployed network components has been recognized early and several groups have proposed to use rapidly deployable wireless networks for disaster response to ll the gap of potentially disaster-a ected communication infrastructures. Based on basic connectivity, autonomous peerto-peer data exchange is an important step towards decentralization and robustness [ 3 ]. Some research groups work on transferring today's work ows in disaster management to the digital domain [ 6 ]. Others strive to use new technologies and adapt them to the use in disaster management. A prominent example is the Ushahidi project, which aims at employing Web 2.0 technologies [ 9 ]. The work of the Sahana foundation on the application layer has achieved signi cant impact by applying and customizing available software components to the speci c needs during a disaster [ 4 ]. 2

Development Process

As described in a previous work-in-progress paper [ 5 ], we followed a primarily user oriented development paradigm. During the entire development process, prospective users have been involved at several stages to increase the usefulness and acceptance of the system, and to provide us with feedback and their wishes for features. User-centered development does not mean to only translate the user's exiting processes identically to a digital version. Additionally, we strive to introduce new ideas and to adapt these to the user's needs. During the last four years, the evolving prototypes of the system have been tested and evaluated by users, and their feedback has been reviewed and directly included into new developments. Details about our requirement analysis and development process, i.e. the Adaptive Frequency Spiral Model (AFSM), a modi ed version of Boehm's well-known spiral model, speci cally tailored to the disaster management domain can be found in [ 5 ]. During the last years, we had the chance to work with di erent groups of end users, mainly assessment experts and TAST members. Up to now the DMT has been presented to and used by 89 participants of the Assessment Mission Course (AMC) and 13 participants of the Sta Management Course (SMC) - both courses are part of the European Civil Protection Training Program - plus approximately 40 participants of the TAST training courses of the German Federal Agency for Technical Relief (THW), and 18 participants of an international training in the context of the EU LIMES project. 2.1

Timeline In the early phase of the DMT's development, we emphasized the collection of requirements and the analysis of the processes in disaster management operations. The temporal evolution of the added functionality can be seen in Fig. 2. The participation at the operations on the G8 Summit in Heiligendamm in June 2007 and the INSARAG certi cation of a THW Heavy Urban Search and Rescue (USAR) team in August 2007 in Hoya were important steps to get a basic understanding of tasks and operational procedures. We observed work ows and conducted many informal interviews about information management in disaster relief operations. At the end of a rst cycle of requirements analysis, we participated at the second AMC in the 5th training cycle in November 2007 (i.e. 5AMC2), where an initial set of functional and non-functional requirements for a Disaster Management Tool evolved [ 5 ]. Based on these requirements a rst G8 Hoya 5AMC2 6AMC1 7AMC1 7AMC2 7AMC3 8AMC1 8AMC2 8AMC3 9AMC1 2007 2008

2009 Point Of Interest (POI) map management distributed network

2010 2011 boo3fkoDmrmgarrakfapsch(tviocierswss)oftwUaIrereadrecshigitnectureusshaabpileitymeannggsitneaembeierliintnytg remmoudlteilmgiassteiownay coroermdoinrdeaevtlieenwceotanwrvcoehrrkitteercture prototype had been developed and implemented by the time of 6AMC1 in June 2008. Most of the basic concepts which are still valid in the current DMT version, such as the distributed network synchronization of the data or the spatial data aggregation in a Point Of Interest (POI) have been used here for the rst time. In this early stage of development, the hardware composition, i.e., a box containing a small computer, a touch screen, several sensors, satellite terminal, rechargeable batteries, chargers etc. added up to 25 kilograms { too heavy for mobile operations. In addition, a proprietary development of a 3D globe visualization turned out to be slow and unstable. Nevertheless we received generally good user feedback which motivated us to develop a completely new system, including a redesigned user interface in which we replaced the initial 3D globe visualization with NASA World Wind Technology [ 2 ]. We reduced the form factor by using smaller boxes and replacing the computer and the separate touch screen with an o -the-shelf laptop. Additionally, the functionality was extended by adding several new features like placemarks, for the next system iteration in June 2009, at the 7AMC1. Beginning from this stage, the user experience was satisfactory, but the underlying software architecture became more and more cluttered. For the next AFSM cycle we concentrated on a review of the overall system architecture. Furthermore, we modi ed the user interface (UI) for increased usability and redesigned the underlying distributed network for data synchronization. This version has been presented and tested by the course participants at the subsequent AMC (7AMC2) in November 2009. More features have been included and evaluated in every iteration. To obtain quantitative user feedback we deployed a usability engineering process based on questionnaires for the 7AMC3. Initial results revealed a lack of stability. Due to the fact that there were only two months to the 8AMC1 in June 2010, we concentrated on this issue. For the AMC in November 2010, we again extended the functionality by implementing a multi-mission capability, which enables the system to concurrently handle multiple missions in parallel. 2.2

Usability Engineering Our usability engineering process is based on the following methods: participating user observation, informal interviews and questionnaire based evaluation [ 8 ] [ 11 ]. We use the method of participating user observation, i.e. to join in performing the users' tasks. In the beginning we used this method to gather initial system requirements. Now it serves as a feedback channel to study the acceptance of implemented functionality, and to obtain novel ideas and demands for the DMT. Observing the user in the eld (at least during exercises and trainings) gives insights that are di cult to obtain in simulated environments (e.g. a usability laboratory). Informal interviews help to constantly improve our understanding of the users and their experiences with the DMT. This informal feedback channel revealed many subconscious requirements and weaknesses of the system. To obtain quantitative user feedback we developed a questionnaire-based evaluation process to identify strong and weak points of the system and to revise requirements. Revising requirements includes the derivation of new requirements and points out functionality which has not been proven to be particularly useful, and therefore needs to be redesigned or even removed from the system. The questionnaire is divided into three parts. The rst part is about the background of the user, including gender, age, expertise as well as computer and mission experiences. In the second part the user has the possibility to rate experiences with the DMT on a 5-step scale (strongly disagree, disagree, neutral, agree and strongly agree). The nine questions presented to the user are: 1. In my opinion the Disaster Management Tool (DMT) is easy to use. 2. I think the provided services (e.g. Points Of Interest, Map handling, etc.) t the requirements for disaster management. 3. The way data is entered into the system is appropriate and e cient. 4. The software provides me with valuable information to ful ll my tasks. 5. I can nd the needed functionality, and do not have to consult the trainer. 6. The system performance is adequate and does not slow my work. 7. The system is supporting the relief work and does not distract or limit me doing my work during relief operations. 8. The Disaster Management Tool increases the situation awareness and therefore supports better coordination of relief operations. 9. I would use the DMT-System for my work.

In the third section the user writes free text to suggest missing or unnecessary functionality and what he or she likes or dislikes about the DMT. Results of the usability engineering process are included in Section 4. 3

Technical Prototype and Core Functionality

Functional and non-functional requirements for an information management system in disaster management have been reported in [ 5 ] and have driven the definition of the DMT's core functionality. The purpose of the DMT is to assist information management during disaster relief operations. The system's visual core component is a dynamic situation map, based on a 3D globe on which geospatial information of various types are displayed. Examples are vector data like points of interest, augmented with speci c text or imagery information, polygons to mark a certain area, or rasterized information such as satellite maps based on images taken before or after the mission or digital elevation models. Tools to handle the input, output and management of the data are o ered. Several sensors, such as position and attitude sensors, can be attached and processed for di erent purposes such as showing the own position on the map or sending it to other users. Additionally all other relevant data items in the system are shared among all connected instances of the DMT, resulting in a distributed, decentralized and disruption-tolerant system. Depending on the available network infrastructure, the best connections are chosen to transmit the information, be it an ad hoc, infrastructure or satellite connection. 3.1

Modular Architecture In order to maintain a stable and extendable software architecture the functionality of the DMT is partitioned into ve modules: { User Interface for visualization and user input { Data Hub / Synchronization for managing data objects { Persistence for storing of data objects { Network for providing transparent communication { Sensors (and the a liated sensor fusion) to manage external hardware The cornerstone of the DMT software architecture is the Data Hub. All information, independent from its origin (data storage, network, user input), is passed through this component. When the user enters information via the UI, data is received via the network module, or a sensor transmits a new measurement, the Data Hub decides what to do with it. The data is analyzed and accordingly forwarded to other modules. The Data Hub module is responsible for ensuring that new information is synchronized with other DMT instances via the distributed network. If the user enters new data via the UI, the Data Hub informs the Persistence component and sends a noti cation via the Network component. If the network component receives a noti cation about new data on the other side, the data is requested and upon reception forwarded to the Persistence and the UI component. The User Interface is designed under the paradigm of keeping its complexity to a minimum, o ering necessary, but avoiding all nonessential \expert" functionality. There are mainly two reasons for this approach. Firstly, the DMT system is generally used by users, who are not working with the system in their daily work. Secondly, the users use the system in a stressful environment. Therefore the main interface is condensed to a minimalistic on-screen menu with the possibility to manage the most important data types and system settings (Points Of Interest, Shapes, Maps, Bookmarks, Units, System Settings). Beside the menu, the NASA World Wind globe is the central visualization element, where all spatial data is shown (see Fig. 1). In the Persistence component, two main tasks are encapsulated, the reliable storage of all data and the guarantee of data integrity. After a restart of the DMT system, the stored information is read from a persistent storage and loaded into the system. The current implementation is based on the operating system's le system, which is reliable and has no further installation requirements. Through the modularized architecture, encapsulation of the functionality to other modules and clear interfaces, a replacement of the underlying information storage technology by other solutions like a database can be carried out with minimal e ort. The Network module o ers an interface for a reliable and e cient data exchange. This module is responsible to handle network-related tasks, such as nding neighboring hosts and starting the initial connection procedure, or selecting the appropriate communication channel (TCP, UDP via Wi-Fi, satellite network, etc.) to an already known host. Network connections are chosen based on their availability and a cost function, depending on the data characteristics and the current status of the system. The Sensors component represents a layer of abstraction for binding external sensors, such as GPS receivers or a 3D compass to the system. Sensor input is preprocessed and fused within the Sensors module. Sensor fusion o ers the possibility to combine several sensor inputs to improve the quality of the output, such as a more precise position by combining several Global Navigation Satellite Systems (GNSS) and/or acceleration sensors [ 10 ]. In order to provide access to a sensor's status and measurements or to set parameters for a sensor, this module has a direct interface with the UI.

Field Experiments and Results

It was very insightful to observe the users using the DMT in the eld. Several problems and gaps have been discovered. Main issues were hardware and stability problems, environment speci c problems, such as direct sunlight exposure and reduced interaction possibilities (e.g. no mouse) in the eld. Some new features have been implemented after observing the users having problems or wasting time, for example the need for extended export functionalities, as users still use their well known software tools and have to follow the prede ned reporting chain, or a coordinate conversion tool to rapidly access di erent formats of a coordinate. In general, the users were very motivated to give direct feedback to the observing developers and many new ideas have been collected this way. 4.1

User Feedback and Empirical Findings In this paper we analyze the rated second part of our questionnaire (see Section 2.2), which provides quantitative measures of user experiences with the DMT. We collected data during ve AMCs: 7AMC3, 8AMC1, 8AMC2, 8AMC3 and 9AMC1. Each AMC has room for up to 20 participants, who are grouped into four teams. Thus, each team consists of up to ve team members. On each AMC 7AMC3 8AMC1 8AMC2 8AMC3 9AMC1 5 4 3 2 1 0 DMTiseasytouse

DMTfitsthe requirements dataenteringis soŌ ware providesinfo trainer needsto be performanceadequate increasesawarness-> DMTissupporƟ ngnot would useDMTfor appropriate tofulfil task consulted notslowing beƩ er coordinaƟ on distracƟ ng work we took part in, we started by giving a general brie ng on the DMT to all participants. Subsequently, we gave a more detailed training to a subset of these participants (initially one team, in later AMCs up to three teams) on the software and supported them in using it for the assessments during the three course days. While we usually gave close support in the rst day of the course, we reduced the support over the following days. The teams typically used the system on their own on the third day. Within the 7AMC3 the Disaster Management Tool was used by one team of ve participants and by a team of four participants on 8AMC1. In the 8AMC2 three assessments teams used the DMT software during the training course, which resulted in 13 valid questionnaires. As a result of the di culties of monitoring more than one team in the eld we evaluated again one team at the 8AMC3 with ve participants and four participants during the 9AMC3. The overall result from the questionnaires indicate encouraging acceptance by our users. Only two aspects score below 4 for all surveys. These are question 3, "The way data is entered into the system is appropriate and e cient." and question 5, "I can nd the needed functionality, and do not have to consult the trainer". To analyze the reason for the relatively low score on data entering, we asked the users in informal interviews why they think that this issue is not ideally solved in the DMT. The result was, that the users are used to enter text with standard o ce software (Microsoft Word) and therefore miss functionality like tagging text by putting bold, italic or underline in the DMT. Also the possibility of structuring lists with bullet points or indenting paragraphs is an important feature for them. Currently, the data entering box is a text eld which does not o er formatting possibilities of text and therefore does not su ciently meet this requirement. The score for question 5 can be ascribed to the fact that the users on the AMC get only 30 minutes of training on the system and afterwards they have two DMT trainers joining and supporting them during the assessments. Directly supporting the user in the eld increases the users' awareness of the DMT's features, but on the other hand reduces their selfcon dence of using the system without instructions, resulting in the consistently suboptimal score. At the moment we assume that a change in the training and support balance as well as compact documentation ("cheat sheets") on how to perform speci c tasks with the DMT should have a positive e ect on this issue. 5

Conclusions and Outlook

The DMT has reached a level of stability that allows its operation by users other than its developers. Its current set of functionality supports coordination and assessment experts in their mission-related tasks. This encompasses the e cient collection, comprehensive displaying and automatic sharing of information. A range of additional helping functionalities, such as automatic conversion between coordinate systems or exporting of its data to feed into reports. Our observations of users working with the tool, informal feedback, as well as formalized feedback in the form of questionnaires have driven the addition and sometimes removal of functionalities. While robustness and consolidation of its functionality remain our foremost priority, we will continue to integrate novel concepts into the DMT. Many new ideas have been proposed by our users and have been captured in our usability engineering process. A particularly interesting concept is to leverage social networks by motivating their users to o er their \cognitive surplus", to remotely assist in missions. Experts in speci c elds, such as structural engineering, language and cultural expertise could contribute without actually being present in the eld. Organized online communities could accomplish time consuming tasks, such as spotting speci c features in aerial images or tracing road networks in a parallel and rapid fashion, literally from their living rooms. This would take o workload from relief workers and empower the general public to contribute to disaster relief. When these concepts will mature they will nd their way into the mission-approved version of the DMT.

Acknowledgements

This work is partially funded by the Helmholtz Foundation and the SOCIETIES (Self Orchestrating CommunIty ambiEnT IntelligEnce Spaces) project, co-funded by the European Commission within FP7. We thank the NASA World Wind Project, in particular Patrick Hogan and Tom Gaskins for providing the outstanding World Wind technology. We thank Dr. Susanne Wacht (THW) for the chance to regularly teach at the THW's TAST courses and are deeply indebted to Claus Hollein (THW), Harm Bastian Harms (Johanniter International Assistance) and Wolfgang Krajic (synergies) and all participants and trainers for their feedback, support and the possibility to join the Assessment Mission Course (AMC), which was and is essential for the development of the DMT.

References

1. 2010 /481/EC, Euratom. Commission decision of 29 july 2010 amending decision 2004/277/ec, euratom as regards rules for the implementation of council decision 2007/779/ec, euratom establishing a community civil protection mechanism . Brussels, Belgium, 2008 .

2. D. Bell , F. Kuehnel , C. Maxwell , R. Kim , K. Kasraie , T. Gaskins , P. Hogan , and J. Coughlan. NASA World Wind: Opensource GIS for mission operations . In Aerospace Conference , 2007 IEEE, pages 1 {9, march 2007 .

3. A. Capata , A. Marrella , R. Russo , M. Bortenschlager , and H. Rieser . A geobased application for the management of mobile actors during crisis situations . In Proceedings of the 5th ISCRAM , Washington, DC, USA, 2008 .

4. M. Careem , C. De Silva , R. De Silva, Raschid, and Weerawarana . Sahana: Overview of a disaster management system . In Information and Automation , 2006 . ICIA 2006. International Conference on, pages 361 { 366 , 2006 .

5. M. Frassl , M. Lichtenstern , M. Khider , and M. Angermann . Developing a system for information management in disaster relief - methodology and requirements . In Proceedings of the International Community on information systems for crisis response and management (ISCRAM 2010 ), Seattle, Washington, USA, 2010 .

6. M. Khalilbeigi , D. Bradler , I. Schweizer , F. Probst , and J. Steimle . Towards computer support of paper work ows in emergency management . Proceedings of the 7th International ISCRAM Conference{Seattle, USA , 2010 .

7. A. Meissner , T. Luckenbach , T. Risse , T. Kirste , and H. Kirchner . Design challenges for an integrated disaster management communication and information system . In Information System , DIREN 2002 (co-located with IEEE INFOCOM 2002 , 2002 .

8. J. Nielsen . Usability Engineering. Morgan Kaufmann, 1st edition, Sept. 1994 .

9. O. Okolloh . Ushahidi, or 'testimony': Web 2.0 tools for crowdsourcing crisis information . Participatory Learning and Action , 59 ( 1 ): 65 { 70 , 2009 .

10. P. Robertson , M. Angermann , and B. Krach . Simultaneous localization and mapping for pedestrians using only foot-mounted inertial sensors . In Proceedings of the 11th international conference on Ubiquitous computing, Ubicomp '09 , pages 93 { 96 , New York, NY, USA, 2009 . ACM.

11. B. Shneiderman and C. Plaisant . Designing the user interface : strategies for e ective human-computer interaction . Addison-Wesley , 2010 .