Disaster Management Tool (DMT) – Usability Engineering, System Architecture and Field Experiments Martin Frassl, Michael Lichtenstern, and Michael Angermann Institute of Communications and Navigation, German Aerospace Center (DLR) 82234 Wessling, Germany {martin.frassl,m.lichtenstern,michael.angermann}@dlr.de Abstract. The Disaster Management Tool (DMT) supports informa- tion management during crises. It has been designed to support field workers, on-site coordination centers and headquarters by facilitating an efficient flow of information between them. In this paper we describe the functionality and architecture of the DMT and give insight into our development process over the last four years. The DMT has undergone extensive field experiments during a series of Assessment Mission Courses (AMCs) for experts in coordination and assessment within the European Civil Protection Mechanism. Results and lessons learned from these ex- periments are presented. 1 Introduction Today’s international response to large scale crises is amazingly rapid and effec- tive. To a large extend this is owed to institutions such as the European Com- mission’s Monitoring and Information Center (MIC) based in Brussels or the United Nations’ Office for the Coordination of Humanitarian Affairs (OCHA) based in Geneva, which are important information hubs and help to coordinate the international response of many governmental and non-governmental relief organizations. International cooperation does not only increase the amount of available resources, but also requires a significant amount of coordination and communication by relief experts in the field. These experts have a proven track record that they are able to cope with complex and uncertain information, even with basic communication means, such as voice communication and basic of- fice computing software or even pen and paper. Nevertheless, several research strands, such as ad hoc and sensor networks, social computing, pervasive com- puting or combinations as in ambient intelligence are motivated to investigate the disaster management domain by the hope that their particular contributions could improve relief efforts. We are inspired by the skills of today’s disaster man- agement experts and the potential of the aforementioned technologies to combine them in a holistic fashion that builds on existing workflows and organizational structures. While we embrace the capabilities we may gain from mobile and em- bedded sensors and computational power, ubiquitous internet connectivity and 2 Frassl et al. Fig. 1. DMT hardware and user interface. vast amounts of information and cognitive resources from crowdsourcing and social networks, we are also concerned that exactly these assets are likely to be affected and potentially unavailable in disaster situations. Hence, our research focusses on how to use these technologies without critically relying on them. In previous work we have investigated the specific requirements for a tool to assist disaster management [5]. In the following paper we report on our work towards a software prototype that helps to study how experts use such a tool under field conditions. We briefly describe the application domain the system is intended to be used in. We describe the functionality and the system architecture of the DMT. Finally, we present and discuss evaluation feedback of users who worked with the DMT during several training missions. 1.1 Application Domain Background Europe has established the European Civil Protection Mechanism (EUCP mech- anism), a process of cooperation during emergencies. This mechanism can be activated by participating states for missions inside and outside of Europe. In such a case the participating countries join their efforts to share resources and increase efficiency [1]. Cooperation between organizations from several countries and a central information and coordination center in Brussels requires a common picture of the situation and thus information sharing across organizational and geographical borders. Prerequisite for a successful mission is a rapid assessment of the specific needs for the disaster response. Typically, several partners, both from the local emergency management agencies as well as international assess- ment and coordination experts, perform the assessment of a situation. Fast and reliable collection and exchange of findings are important to select the best- suited assets for relief. Assessment experts have already a variety of technical tools available: GPS navigation devices, satellite communication terminals, elec- tronic maps or web sites filled with information about the situation before the disaster. Working with these tools requires experience and time, with especially the latter being a scarce resource during a mission. Time pressure and other stressors tend to lower the frustration thresholds of users. To support disaster Disaster Management Tool (DMT) 3 management experts in the field, the UN system and the EUCP system have introduced dedicated support units, which cover information and communica- tion technology and a portfolio of additional tasks such as transportation, camp building, subsistence and administration. In UNDAC (United Nations Disaster Assessment and Coordination) missions, this role is frequently assigned to the International Humanitarian Partnership (IHP), an association of organizations from mainly Scandinavian countries. In EUCP missions TAST teams (Technical Assistance and Support) are available in the form of EUCP modules. Further- more, several non-governmental organizations (NGOs) provide assistance for a specific field, like Mapaction for the in situ production of maps or Ericsson Re- sponse for communication services. Nevertheless, basic knowledge like navigation with a GPS device or setting up a BGAN satellite terminal is expected from a coordination and assessment expert. 1.2 Related Work The difficulty of the challenges in the disaster management domain have at- tracted a growing number of researchers that contribute towards several of the involved problems. Meissner et al. have investigated a range of requirements and design challenges for an integrated disaster management communication and information system [7]. Furthermore, Meissner et al. drafted high-level archi- tectures for the communications and personal task scheduling subsystems. The need for rapid configuration of deployed network components has been recog- nized early and several groups have proposed to use rapidly deployable wire- less networks for disaster response to fill the gap of potentially disaster-affected communication infrastructures. Based on basic connectivity, autonomous peer- to-peer data exchange is an important step towards decentralization and robust- ness [3]. Some research groups work on transferring today’s workflows in disaster management to the digital domain [6]. Others strive to use new technologies and adapt them to the use in disaster management. A prominent example is the Ushahidi project, which aims at employing Web 2.0 technologies [9]. The work of the Sahana foundation on the application layer has achieved significant im- pact by applying and customizing available software components to the specific needs during a disaster [4]. 2 Development Process As described in a previous work-in-progress paper [5], we followed a primarily user oriented development paradigm. During the entire development process, prospective users have been involved at several stages to increase the usefulness and acceptance of the system, and to provide us with feedback and their wishes for features. User-centered development does not mean to only translate the user’s exiting processes identically to a digital version. Additionally, we strive to introduce new ideas and to adapt these to the user’s needs. During the last four years, the evolving prototypes of the system have been tested and evaluated 4 Frassl et al. by users, and their feedback has been reviewed and directly included into new developments. Details about our requirement analysis and development process, i.e. the Adaptive Frequency Spiral Model (AFSM), a modified version of Boehm’s well-known spiral model, specifically tailored to the disaster management domain can be found in [5]. During the last years, we had the chance to work with different groups of end users, mainly assessment experts and TAST members. Up to now the DMT has been presented to and used by 89 participants of the Assessment Mission Course (AMC) and 13 participants of the Staff Management Course (SMC) - both courses are part of the European Civil Protection Training Program - plus approximately 40 participants of the TAST training courses of the German Federal Agency for Technical Relief (THW), and 18 participants of an international training in the context of the EU LIMES project. 2.1 Timeline In the early phase of the DMT’s development, we emphasized the collection of requirements and the analysis of the processes in disaster management opera- tions. The temporal evolution of the added functionality can be seen in Fig. 2. The participation at the operations on the G8 Summit in Heiligendamm in June 2007 and the INSARAG certification of a THW Heavy Urban Search and Res- cue (USAR) team in August 2007 in Hoya were important steps to get a basic understanding of tasks and operational procedures. We observed workflows and conducted many informal interviews about information management in disaster relief operations. At the end of a first cycle of requirements analysis, we par- ticipated at the second AMC in the 5th training cycle in November 2007 (i.e. 5AMC2), where an initial set of functional and non-functional requirements for a Disaster Management Tool evolved [5]. Based on these requirements a first G8 Hoya 5AMC2 6AMC1 7AMC1 7AMC2 7AMC3 8AMC1 8AMC2 8AMC3 9AMC1 2007 2008 2009 2010 2011 Point Of Interest (POI) form factor software architecture usability engineering multi mission coordinate converter map management 3D graphics UI redesign shape mangement remodel gateway remodel network distributed network bookmarks (views) stability review architecture Fig. 2. Timeline of the DMT development prototype had been developed and implemented by the time of 6AMC1 in June 2008. Most of the basic concepts which are still valid in the current DMT ver- sion, such as the distributed network synchronization of the data or the spatial data aggregation in a Point Of Interest (POI) have been used here for the first time. In this early stage of development, the hardware composition, i.e., a box containing a small computer, a touch screen, several sensors, satellite terminal, rechargeable batteries, chargers etc. added up to 25 kilograms – too heavy for Disaster Management Tool (DMT) 5 mobile operations. In addition, a proprietary development of a 3D globe visual- ization turned out to be slow and unstable. Nevertheless we received generally good user feedback which motivated us to develop a completely new system, including a redesigned user interface in which we replaced the initial 3D globe visualization with NASA World Wind Technology [2]. We reduced the form fac- tor by using smaller boxes and replacing the computer and the separate touch screen with an off-the-shelf laptop. Additionally, the functionality was extended by adding several new features like placemarks, for the next system iteration in June 2009, at the 7AMC1. Beginning from this stage, the user experience was satisfactory, but the underlying software architecture became more and more cluttered. For the next AFSM cycle we concentrated on a review of the over- all system architecture. Furthermore, we modified the user interface (UI) for increased usability and redesigned the underlying distributed network for data synchronization. This version has been presented and tested by the course par- ticipants at the subsequent AMC (7AMC2) in November 2009. More features have been included and evaluated in every iteration. To obtain quantitative user feedback we deployed a usability engineering process based on questionnaires for the 7AMC3. Initial results revealed a lack of stability. Due to the fact that there were only two months to the 8AMC1 in June 2010, we concentrated on this issue. For the AMC in November 2010, we again extended the function- ality by implementing a multi-mission capability, which enables the system to concurrently handle multiple missions in parallel. 2.2 Usability Engineering Our usability engineering process is based on the following methods: participat- ing user observation, informal interviews and questionnaire based evaluation [8] [11]. We use the method of participating user observation, i.e. to join in perform- ing the users’ tasks. In the beginning we used this method to gather initial system requirements. Now it serves as a feedback channel to study the acceptance of im- plemented functionality, and to obtain novel ideas and demands for the DMT. Observing the user in the field (at least during exercises and trainings) gives insights that are difficult to obtain in simulated environments (e.g. a usability laboratory). Informal interviews help to constantly improve our understanding of the users and their experiences with the DMT. This informal feedback chan- nel revealed many subconscious requirements and weaknesses of the system. To obtain quantitative user feedback we developed a questionnaire-based evaluation process to identify strong and weak points of the system and to revise require- ments. Revising requirements includes the derivation of new requirements and points out functionality which has not been proven to be particularly useful, and therefore needs to be redesigned or even removed from the system. The questionnaire is divided into three parts. The first part is about the background of the user, including gender, age, expertise as well as computer and mission experiences. In the second part the user has the possibility to rate experiences with the DMT on a 5-step scale (strongly disagree, disagree, neutral, agree and strongly agree). The nine questions presented to the user are: 6 Frassl et al. 1. In my opinion the Disaster Management Tool (DMT) is easy to use. 2. I think the provided services (e.g. Points Of Interest, Map handling, etc.) fit the requirements for disaster management. 3. The way data is entered into the system is appropriate and efficient. 4. The software provides me with valuable information to fulfill my tasks. 5. I can find the needed functionality, and do not have to consult the trainer. 6. The system performance is adequate and does not slow my work. 7. The system is supporting the relief work and does not distract or limit me doing my work during relief operations. 8. The Disaster Management Tool increases the situation awareness and there- fore supports better coordination of relief operations. 9. I would use the DMT-System for my work. In the third section the user writes free text to suggest missing or unnecessary functionality and what he or she likes or dislikes about the DMT. Results of the usability engineering process are included in Section 4. 3 Technical Prototype and Core Functionality Functional and non-functional requirements for an information management sys- tem in disaster management have been reported in [5] and have driven the def- inition of the DMT’s core functionality. The purpose of the DMT is to assist information management during disaster relief operations. The system’s visual core component is a dynamic situation map, based on a 3D globe on which geospatial information of various types are displayed. Examples are vector data like points of interest, augmented with specific text or imagery information, polygons to mark a certain area, or rasterized information such as satellite maps based on images taken before or after the mission or digital elevation models. Tools to handle the input, output and management of the data are offered. Sev- eral sensors, such as position and attitude sensors, can be attached and processed for different purposes such as showing the own position on the map or sending it to other users. Additionally all other relevant data items in the system are shared among all connected instances of the DMT, resulting in a distributed, de- centralized and disruption-tolerant system. Depending on the available network infrastructure, the best connections are chosen to transmit the information, be it an ad hoc, infrastructure or satellite connection. 3.1 Modular Architecture In order to maintain a stable and extendable software architecture the function- ality of the DMT is partitioned into five modules: – User Interface for visualization and user input – Data Hub / Synchronization for managing data objects – Persistence for storing of data objects – Network for providing transparent communication – Sensors (and the affiliated sensor fusion) to manage external hardware Disaster Management Tool (DMT) 7 The cornerstone of the DMT software architecture is the Data Hub. All informa- tion, independent from its origin (data storage, network, user input), is passed through this component. When the user enters information via the UI, data is received via the network module, or a sensor transmits a new measurement, the Data Hub decides what to do with it. The data is analyzed and accordingly for- warded to other modules. The Data Hub module is responsible for ensuring that new information is synchronized with other DMT instances via the distributed network. If the user enters new data via the UI, the Data Hub informs the Per- sistence component and sends a notification via the Network component. If the network component receives a notification about new data on the other side, the data is requested and upon reception forwarded to the Persistence and the UI component. The User Interface is designed under the paradigm of keeping its complexity to a minimum, offering necessary, but avoiding all nonessential “ex- pert” functionality. There are mainly two reasons for this approach. Firstly, the DMT system is generally used by users, who are not working with the system in their daily work. Secondly, the users use the system in a stressful environment. Therefore the main interface is condensed to a minimalistic on-screen menu with the possibility to manage the most important data types and system settings (Points Of Interest, Shapes, Maps, Bookmarks, Units, System Settings). Beside the menu, the NASA World Wind globe is the central visualization element, where all spatial data is shown (see Fig. 1). In the Persistence component, two main tasks are encapsulated, the reliable storage of all data and the guarantee of data integrity. After a restart of the DMT system, the stored information is read from a persistent storage and loaded into the system. The current imple- mentation is based on the operating system’s file system, which is reliable and has no further installation requirements. Through the modularized architecture, encapsulation of the functionality to other modules and clear interfaces, a re- placement of the underlying information storage technology by other solutions like a database can be carried out with minimal effort. The Network module offers an interface for a reliable and efficient data exchange. This module is re- sponsible to handle network-related tasks, such as finding neighboring hosts and starting the initial connection procedure, or selecting the appropriate commu- nication channel (TCP, UDP via Wi-Fi, satellite network, etc.) to an already known host. Network connections are chosen based on their availability and a cost function, depending on the data characteristics and the current status of the system. The Sensors component represents a layer of abstraction for binding external sensors, such as GPS receivers or a 3D compass to the system. Sensor input is preprocessed and fused within the Sensors module. Sensor fusion offers the possibility to combine several sensor inputs to improve the quality of the output, such as a more precise position by combining several Global Navigation Satellite Systems (GNSS) and/or acceleration sensors [10]. In order to provide access to a sensor’s status and measurements or to set parameters for a sensor, this module has a direct interface with the UI. 8 Frassl et al. 4 Field Experiments and Results It was very insightful to observe the users using the DMT in the field. Several problems and gaps have been discovered. Main issues were hardware and stability problems, environment specific problems, such as direct sunlight exposure and reduced interaction possibilities (e.g. no mouse) in the field. Some new features have been implemented after observing the users having problems or wasting time, for example the need for extended export functionalities, as users still use their well known software tools and have to follow the predefined reporting chain, or a coordinate conversion tool to rapidly access different formats of a coordinate. In general, the users were very motivated to give direct feedback to the observing developers and many new ideas have been collected this way. 4.1 User Feedback and Empirical Findings In this paper we analyze the rated second part of our questionnaire (see Section 2.2), which provides quantitative measures of user experiences with the DMT. We collected data during five AMCs: 7AMC3, 8AMC1, 8AMC2, 8AMC3 and 9AMC1. Each AMC has room for up to 20 participants, who are grouped into four teams. Thus, each team consists of up to five team members. On each AMC 5 4 3 7AMC3 8AMC1 8AMC2 2 8AMC3 9AMC1 1 0 DMT is easy to use DMT fits the data entering is soŌware provides info trainer needs to be performance adequate increases awarness -> DMT is supporƟng not would use DMT for requirements appropriate to fulfill task consulted not slowing beƩer coordinaƟon distracƟng work Fig. 3. Evaluation results: mean values of the user experiences with the DMT we took part in, we started by giving a general briefing on the DMT to all participants. Subsequently, we gave a more detailed training to a subset of these participants (initially one team, in later AMCs up to three teams) on the software and supported them in using it for the assessments during the three course days. While we usually gave close support in the first day of the course, we reduced the support over the following days. The teams typically used the system on their own on the third day. Within the 7AMC3 the Disaster Management Tool was used by one team of five participants and by a team of four participants on 8AMC1. In the 8AMC2 three assessments teams used the DMT software during the training course, which resulted in 13 valid questionnaires. As a result of the difficulties of monitoring more than one team in the field we evaluated again one team at the 8AMC3 with five participants and four participants during Disaster Management Tool (DMT) 9 the 9AMC3. The overall result from the questionnaires indicate encouraging acceptance by our users. Only two aspects score below 4 for all surveys. These are question 3, ”The way data is entered into the system is appropriate and efficient.” and question 5, ”I can find the needed functionality, and do not have to consult the trainer”. To analyze the reason for the relatively low score on data entering, we asked the users in informal interviews why they think that this issue is not ideally solved in the DMT. The result was, that the users are used to enter text with standard office software (Microsoft Word) and therefore miss functionality like tagging text by putting bold, italic or underline in the DMT. Also the possibility of structuring lists with bullet points or indenting paragraphs is an important feature for them. Currently, the data entering box is a textfield which does not offer formatting possibilities of text and therefore does not sufficiently meet this requirement. The score for question 5 can be ascribed to the fact that the users on the AMC get only 30 minutes of training on the system and afterwards they have two DMT trainers joining and supporting them during the assessments. Directly supporting the user in the field increases the users’ awareness of the DMT’s features, but on the other hand reduces their self- confidence of using the system without instructions, resulting in the consistently suboptimal score. At the moment we assume that a change in the training and support balance as well as compact documentation (”cheat sheets”) on how to perform specific tasks with the DMT should have a positive effect on this issue. 5 Conclusions and Outlook The DMT has reached a level of stability that allows its operation by users other than its developers. Its current set of functionality supports coordination and assessment experts in their mission-related tasks. This encompasses the efficient collection, comprehensive displaying and automatic sharing of information. A range of additional helping functionalities, such as automatic conversion between coordinate systems or exporting of its data to feed into reports. Our observations of users working with the tool, informal feedback, as well as formalized feedback in the form of questionnaires have driven the addition and sometimes removal of functionalities. While robustness and consolidation of its functionality remain our foremost priority, we will continue to integrate novel concepts into the DMT. Many new ideas have been proposed by our users and have been captured in our usability engineering process. A particularly interesting concept is to leverage social networks by motivating their users to offer their “cognitive surplus”, to remotely assist in missions. Experts in specific fields, such as structural engi- neering, language and cultural expertise could contribute without actually being present in the field. Organized online communities could accomplish time con- suming tasks, such as spotting specific features in aerial images or tracing road networks in a parallel and rapid fashion, literally from their living rooms. This would take off workload from relief workers and empower the general public to contribute to disaster relief. When these concepts will mature they will find their way into the mission-approved version of the DMT. 10 Frassl et al. 6 Acknowledgements This work is partially funded by the Helmholtz Foundation and the SOCI- ETIES (Self Orchestrating CommunIty ambiEnT IntelligEnce Spaces) project, co-funded by the European Commission within FP7. We thank the NASA World Wind Project, in particular Patrick Hogan and Tom Gaskins for providing the outstanding World Wind technology. We thank Dr. Susanne Wacht (THW) for the chance to regularly teach at the THW’s TAST courses and are deeply in- debted to Claus Höllein (THW), Harm Bastian Harms (Johanniter International Assistance) and Wolfgang Krajic (synergies) and all participants and trainers for their feedback, support and the possibility to join the Assessment Mission Course (AMC), which was and is essential for the development of the DMT. References 1. 2010/481/EC, Euratom. Commission decision of 29 july 2010 amending decision 2004/277/ec, euratom as regards rules for the implementation of council decision 2007/779/ec, euratom establishing a community civil protection mechanism. Brus- sels, Belgium, 2008. 2. D. Bell, F. Kuehnel, C. Maxwell, R. Kim, K. Kasraie, T. Gaskins, P. Hogan, and J. Coughlan. NASA World Wind: Opensource GIS for mission operations. In Aerospace Conference, 2007 IEEE, pages 1–9, march 2007. 3. A. Capata, A. Marrella, R. Russo, M. Bortenschlager, and H. Rieser. A geo- based application for the management of mobile actors during crisis situations. In Proceedings of the 5th ISCRAM, Washington, DC, USA, 2008. 4. M. Careem, C. De Silva, R. De Silva, Raschid, and Weerawarana. Sahana: Overview of a disaster management system. In Information and Automation, 2006. ICIA 2006. International Conference on, pages 361–366, 2006. 5. M. Frassl, M. Lichtenstern, M. Khider, and M. Angermann. Developing a system for information management in disaster relief - methodology and requirements. In Proceedings of the International Community on information systems for crisis response and management (ISCRAM 2010), Seattle, Washington, USA, 2010. 6. M. Khalilbeigi, D. Bradler, I. Schweizer, F. Probst, and J. Steimle. Towards com- puter support of paper workflows in emergency management. Proceedings of the 7th International ISCRAM Conference–Seattle, USA, 2010. 7. A. Meissner, T. Luckenbach, T. Risse, T. Kirste, and H. Kirchner. Design chal- lenges for an integrated disaster management communication and information sys- tem. In Information System, DIREN 2002 (co-located with IEEE INFOCOM 2002, 2002. 8. J. Nielsen. Usability Engineering. Morgan Kaufmann, 1st edition, Sept. 1994. 9. O. Okolloh. Ushahidi, or ’testimony’: Web 2.0 tools for crowdsourcing crisis infor- mation. Participatory Learning and Action, 59(1):65–70, 2009. 10. P. Robertson, M. Angermann, and B. Krach. Simultaneous localization and map- ping for pedestrians using only foot-mounted inertial sensors. In Proceedings of the 11th international conference on Ubiquitous computing, Ubicomp ’09, pages 93–96, New York, NY, USA, 2009. ACM. 11. B. Shneiderman and C. Plaisant. Designing the user interface : strategies for effective human-computer interaction. Addison-Wesley, 2010.