Ontology-based Reasoning for Critical Incidents in Public Events Dimos Ntioudis, Angelos Chatzimichail, Georgios Meditskos, Stefanos Vrochidis, and Ioannis Kompatsiaris Information Technologies Institute, Centre for Research and Technology - Hellas, {ntdimos,angechat,gmeditsk,stefanos,ikom}@iti.gr Abstract. One of the most critical challenges during critical incidents in big public events are the absence of real time information as well as the lack of filtering this information to meaningful knowledge in order to provide crisis decision support. Towards addressing those challenges, se- mantic technologies can provide a semantically data representation of the critical events, along with superior capabilities in reasoning to support the decision making. In this paper we describe the semantic reasoning framework of the DESMOS project, that could constitute the backbone of the decision support systems in public events. Keywords: Ontologies · Reasoning · Security · Internet of Things. 1 Introduction The era of the Internet of Things (IoT) is upon us, with a huge number of IoT devices already in everybody lives. A large number of applications in Smart Cities, E-health, Security and other domains are exploiting the IoT technologies, like sensors, smartphones and actuators. One of the most significant domains of IoT is the safety and security, which IoT is used to save lives and mitigate dangerous incidents. However, several challenges arose with the existing IoT technologies regarding the interoperability of those different IoT technologies since their data is based on proprietary formats and they do not use common formats or a vocabulary to describe the interoperable data. The basic structure of the IoT is the Machine-to- Machine (M2M) communication. For example, the measurements of sensors are required to be distributed and analyzed by other devices or sensors and not being human readable without any kind of processing. Therefore, those measurements should be understandable from one machine to another. In this paper, we describe the methodology and procedure that underpin the development of reasoning rules in the DESMOS ontology. DESMOS is a novel framework for the intelligent interconnection of smart infrastructures, mobile and wearable devices and apps for the provision of a secure environment for citizens, especially for visitors and tourists. The platform aims to promote the collab- oration between people and devices for protecting tourists, supporting timely Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2 D. Ntioudis et al. reporting of incidents, adaptation of the interconnected environments in case of emergency, and the provision of assistance by empowering local authorities and volunteers. The main research contribution of the paper is the combination of the data processing modules (localization) with the semantic knowledge under a common reasoning framework. The rest of the paper is organized as follows. In Section 2, we describe related work. In Section 3 the Desmos framework and the main technical components. Section 4 describes the Data Integration. rule-based semantic reasoning. In Sec- tion 5 the Rule-based Reasoning is presented. Finally, Section 6 concludes the paper. 2 Related Work The Internet of Things (IoT) is becoming more and more popular and its appli- cations are facing an enormous proliferation resulting in a new digital ecosystem. IoT is based on a wide range of different heterogeneous technologies and devices, there is not a uniform vocabulary for representation and processing of data. This has led to a large number of incompatible IoT platforms. Through this way, it is very difficult for data scientists to extract knowledge from the enormous number of data producing every second through the IoT applications. Semantic web technologies tries to overcome such challenges. Semantic web leverage web standards and semantic technologies to interconnect all types of devices by transforming low-level sensor data into high-level knowledge that is comprehensible to humans and machines. Semantic modelling produces a definite description of the data meaning in a structured way by combining application knowledge and context-relevant information with sensor data. The ontology - based development, which is a domain of semantic modelling, of IoT frameworks can lead to universal IoT solutions multiplying the benefits of IoT. W3C (World Wide Web Consortium) suggested an ontology, the Seman- tic Sensor Network (SSN), as a human and machine-readable specification that covers networks of sensors and their deployment on top of sensors and observa- tions [1]. The target of this project was to face the problems that arose from the heterogeneous data from different devices. However, there are limited on- tologies that annotate the time–space correlation between the sensor data and the resources. In order to overcome that, the authors in [2] deployed the Sensor, Observation, Sample, and Actuator (SOSA) ontology providing a formal but lightweight general-purpose specification for modelling the interaction between the entities involved in the acts of observation, actuation, and sampling. One of the most important ontologies for event annotation is Event [3]. This ontology is centered around the notion of event, with cognitive agents classify ar- bitrary time/space regions. The agent class was derived from the FOAF (Friend of a Friend) ontology, a core ontology for the social relationships [4]. In [5] presented an ontology for situation awareness (SAW). These ontologies anno- tated events with general situations and could be expanded with supplementary ontologies. In [6], researchers developed a platform to assist citizens in report- Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Ontology-based Reasoning for Critical Incidents in Public Events 3 ing security threats together with their severity and location. The information about the threats were stored in a knowledge base of the system that allowed for lightweight reasoning with the gathered facts. There are many research studies for semantic reasoning on critical event response and management. In [7], developed a system based on semantic tech- nologies for monitoring social media for earthquake reports and weather alerts, and for notifying the public in case of an emergency. Also, in [8], researchers presented a novel approach to emergency response applications integrating a formal semantic domain model into an event-based decision support system, which supports reasoning on this model. A semantic web-based disaster trail management ontology that covered all vital facets of disasters like disaster type, disaster location, disaster time and so forth is presented in [9]. The ontology was assessed efficaciously via competency questions, externally by the domain ex- perts and internally with the help of SPARQL queries. Finally, in [10] Semantic Web Technologies applied for Decision Support in Climate-Related Crisis Man- agement, supporting crisis management systems in the domain of situational awareness. 3 DESMOS Framework In this section we are presenting the framework for encoding, aggregating and se- mantically analysing information relevant to the DESMOS application domain. The framework is presented in Figure 1. It consists of a) a Communication Layer that allows the interactions between the system components, b) an Ontol- ogy, that is used for semantically representing notions pertinent to the project, c) a Knowledge Base Repository (KBR) that hosts the domain ontology, d) a Knowledge Base Service that acts as the main interface to the ontology, re- sponsible for populating the ontology with incoming data as well as performing rule-based reasoning over the knowledge base, e) Mobile Apps, Wearable Devices and Sensors that feed the system with real-time data and finally f) a Localiza- tion Module that retrieves the received signal strength indicator (RSSI) values of the wearable devices from the KBR in order to calculate the distance based on data analysis algorithms. 3.1 Use Cases and User Requirements The basis for the development of an intelligent interconnected framework for public security and protection are the needs and requirements of the domain experts. The purpose of the requirements collection is to understand the needs of the users and the problems they try to solve. A common methodology for defining the use cases and requirements has been used, starting from the identification of the most common security and protection incidents in the pilot location, the ways of dealing with them up to now, and the difficulties encountered in managing them. Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 4 D. Ntioudis et al. KNOWLEDGE BASE REPOSITORY Mobile Apps Wearables Locati on GraphDB SPARQL RDF4J COMMUNICATION LAYER KNOWLEDGE BASE SERVICE RSSI ANALYSIS S ERVICE Knowledge Base Population Distance Localization Module Events Crawler RSSI Fig. 1. DESMOS framework. Table 1. Use Cases. UC# Use Case Name Use Case Description UC 1 MylosKarpa & City It refers to CPR-certified users and chronic patients and Karpa in particular people suffering from a heart disease. UC 2 MylosKidFinder It refers to families with children aged 3 years to 10 who visit the pilot location and there is a possibility that gets lost in the crowd. The use cases show in Table 1 were identified and studied by running several focus group sessions between stakeholders concerned with integrated risk man- agement (municipal authorities, volunteers etc.) focusing on their needs. In ad- dition, requirements were identified and prioritized through interview questions and focus group sessions between domain experts, end-users, lawyers and soft- ware engineers. Table 2 contains an indicative subset of the user requirements. These user requirements are catalogued as [UR xzz ], where x is the identifier of the use case scenario in which the requirement originated (see Table 1), and zz is the serial number of the requirement. 4 Data Integration The Knowledge Base Service (KBS) in Figure 1 is a central component of the system architecture since it is the main interface to the DESMOS ontology also referred to as the Knowledge Base (KB). The ontology is the knowledge represen- tation model for semantically representing notions pertinent to the project: (a) Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Ontology-based Reasoning for Critical Incidents in Public Events 5 Table 2. Subset of the User Requirements. UR# User Requirement Name Requirement Description UR 102 Locate chronic patients Display to volunteers and CPR-certified personnel (i.e. through their mobile phones) the location of chronic patients and in particular people suffering from a heart disease through their mobile phones when requested. UR 103 Activate closest AED Detect and activate the beacon of the AED that beacon is closest to the respective health emergency situa- tion. UR 204 Locate lost children Display to volunteers (i.e. through their mobile phones) the location of a lost children once they are declared as such from their parents or guardians. location of the personnel, (b) location of automated external defibrillators (AED) (c) the RSSI observations made by mobile devices or fixed nodes (i.e. hubs) both listening for broadcasted addresses from Bluetooth Low Energy (BLE) devices d) the results of the RSSI analysis service (i.e. Localization module), e) visitor alert requests and f) personnel assignments to critical incidents. Volunteer Notification Trigger Visitor Hub subclassOf Beacon Notification Human subclassOf takesAction subclassOf Agent Category Action category Coordinates subclassOf owns Point hasRole coordinates requires subclassOf location Loss Location Health Device Role Event Sensor Issue subclassOf contains subclassOf triggers Missing subclassOf subclassOf Person Sensor Volunteer subclassOf Visitor Data Is result of contains Transmitt BLE er rssi mac Observation DESMOS GEO RSSI MAC SSN/SOSA FOAF Fig. 2. Abstract representation of existing ontologies with the DESMOS ontology. The KB is hosted by GraphDB1 that acts as the Knowledge Base Repository. A part of the developed ontology is depicted in Figure 2. For the development of the ontology existing ontologies like SSN, SOSA, Geo (i.e. WGS84) and FOAF 1 https://www.ontotext.com/products/graphdb/ Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 6 D. Ntioudis et al. have been reused. In addition, KBS features subscribe/publish capabilities to a dedicated communication bus (MSB) allowing interactions of the KB with other system components. In a sense, KBS accepts input from other components and semantically integrates it into the ontology. It also handles the output of the semantic reasoning process running on top of the KB and forwards the inferred, high-level knowledge back to other interested system components. As shown in Figure 1, KBS consists of two sub-components, namely the Knowledge Base Population (KBP) component, responsible for integrating data into the KB, and the Events Crawler that is activated as soon as an event is triggered. The interactions of the KBS with the KBR is achieved through SPARQL queries or by using the RDF4J2 library tools. More specifically, the communication layer allows the exchange of messages in JSON format. These messages are analyzed by the KBS and are semanti- cally integrated into the appropriate classes considering the domain ontology. The ontology itself as well as the data collected from the various system com- ponents (e.g. sensors, mobile apps etc.) will be stored in the KB and the KBP component is responsible for the data manipulation. Some indicative examples of representing notions of the domain are given below (i.e. in RDF Triples). The first example depicts the representation of a Volunteer along with his location consisting of the Volunteer’s latest coordinates (i.e. latitude and longitude). Listing 1.1. RDF representation of a Volunteer in the KB. rdf:type desmos:Human ; desmos:hasRole desmos:Volunteer ; desmos:id "867325031117405"^^xsd:string ; desmos:name "Volunteer_1."@en ; sosa:hosts ; rdfs:label "Volunteer 1"@en . geo:hasGeometry [geo:asWKT POINT(22.9954257 40.5654389)] Moreover, mobile devices belonging to Volunteers send regular messages about registered wearable devices that are within their range. These messages are also characterized by a unique identifier of the sending device that will allow the information on the wearable device to be matched with the mobile device that sent the message. Below is the representation of an rssi observation in the on- tology. Note how the last property in the representation, isObservedBy, matches the corresponding observation with the mobile device. Listing 1.2. RDF representation of an RSSI Observation. rdf:type sosa:Observation; sosa:observedProperty ; sosa:hasFeatureOfInterest ; sosa:madeBySensor ; 2 https://rdf4j.org/ Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Ontology-based Reasoning for Critical Incidents in Public Events 7 sosa:resultTime "2019-05-29T12:52:01Z "^^xsd:dateTime; sosa:usedProcedure ; desmos:signal_strength "-51"^^xsd:double; sosa:isObservedBy . In addition, Visitors of the park may use the application that they have previously installed in their mobile devices and send a request that will trigger an Event. This request may refer either to a Health Issue or a Missing Person situation. The following example depicts the representation of a Health Issue event in the ontology. Listing 1.3. RDF representation of a Health Issue Event. rdf:type desmos:HealthIssue; desmos:id "0a990cf5-8c3f-4484"^^xsd:string ; geo:hasGeometry [geo:asWKT POINT(22.9879934 40.57327)] It is worth mentioning that there is an initialization phase where the ontology is populated with the volunteer profiles together with the necessary information about their mobile devices, the exact location of the fixed nodes that listen for broadcasted addresses from BLE devices as well as the location of the AED systems that are available in the area. Once this data are populated in the ontology, the KBS can perform reasoning to infer new knowledge, and then update the interested components. 5 Rule-based Reasoning In addition, in order to search for incoming messages and handling their se- mantic representation and integration into the KB, KBS also incorporates a reasoning mechanism to infer underlying knowledge and discover connections between ontology entities during various events occurring in space, such as the disappearance of a person, or various medical incidents. This mechanism is rule- based and consists of a set of SPARQL queries. These queries were constructed by following the approach that is described in the following subsection. 5.1 Competency Questions The user requirements in Table 2 were first mapped to ontology’s Competency Questions (CQs). A competency question is a natural language sentence that expresses a pattern for a type of question people expect an ontology to answer [11]. The answerability of CQs hence becomes a functional requirement of the ontology. Based on the list of user requirements above, the ontology is able to respond to several CQs, such as providing the location of a specific person (e.g. a patient, a lost child), or detect the closest AED. Following the methodology pro- posed in [12], we translated the list of CQs into respective SPARQL queries [13] and evaluated the retrieved results. The following tables represent an indicative set of CQs along with their SPARQL translation. Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 8 D. Ntioudis et al. Given a Visitor’s request request that is related to UC 1 (i.e. request for medical help) and consists of a request id and the coordinates of the user re- questing for help, the following queries (Queries 1.4, 1.5) calculate the nearest Volunteer and the nearest AED respectively. Listing 1.4. (UC 1): Who is the nearest Volunteer? SELECT ?imei WHERE { ?not rdf:type desmos:VisitorNotification ; geo:hasGeometry [ geo:asWKT ?point1 ] ; desmos:id ?not_id. ?volunteer desmos:hasRole desmos:Volunteer ; geo:hasGeometry [ geo:asWKT ?point2 ] ; desmos:imei ?imei. BIND((geof:distance(?point1, ?point2, uom:metre)) as ?dist) . FILTER(?not_id = STR(not_id)) . } ORDER BY ASC(?dist) LIMIT 1 Listing 1.5. (UC 1): What is the nearest AED? SELECT ?aed_id WHERE { ?not rdf:type desmos:VisitorNotification ; geo:hasGeometry [ geo:asWKT ?point1 ] ; desmos:id ?not_id. ?aed rdf:type desmos:AED ; geo:hasGeometry [ geo:asWKT ?point2 ] ; desmos:id ?aed_id. BIND((geof:distance(?point1, ?point2, uom:metre)) as ?dist) . FILTER(?not_id = STR(not_id)) . } ORDER BY ASC(?dist) LIMIT 1 Similarly, given a Visitor’s request that is related to UC 2 (i.e. request help for lost child) and consists of the request id and the mac address of the child’s wearable device, the following queries (Queries 1.6, 1.7) are calculating the lat- est observers of the missing child (i.e. Volunteers) and their distance from the lost child respectively. More specifically, query 1.6 finds the 3 nearest observers while query 1.7 uses the latter information as input (i.e. through FILTER) and calculates the respective distance of each observer. Note that the distance between the Volunteers and the lost child is calcu- lated based on RSSI-based localization techniques that are performed by the Localization Module. More specifically in UC 2 children are wearing wearable equipment emitting a Radio Frequency (RF) signal with an RSSI that is re- ceived from the the volunteers’ devices as well as from several fixed points (i.e. hubs) scattered around the pilot area. The RSSI values are then used to cal- culate the respective distance in meters (in the Localization module) and this information is integrated into the KB and enriches the existing knowledge. Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Ontology-based Reasoning for Critical Incidents in Public Events 9 The localization module analyzes the RSSI values based on a kalman filtering and a regression method that transforms the RSSI values to distances. After that, given the coordinates of the closest Volunteers and Hubs from the KB, namely A, B, C with coordinates (x1 , y1 ), (x2 , y2 ), (x3 , y3 ) respectively, whose distance from the point of interest (i.e. lost child) is d1 , d2 , d3 respectively which was estimated before, we can then calculate the coordinates of the point of interest (x, y) using the following equations: (x − xi )2 + (y − yi )2 = d2i , i = 1, 2, 3 Listing 1.6. (UC 2): Who are the child’s latest observers? SELECT ?imei (MAX(?time) as ?instanceTime) { ?s rdf:type sosa:Observation ; sosa:isObservedBy ?o ; desmos:isDataOf ?p ; sosa:resultTime ?time . ?vol sosa:hosts ?o ; desmos:imei ?imei . ?p desmos:mac ?mac . FILTER(?mac = STR(mac)) } GROUP BY ?imei ORDER BY DESC(?instanceTime) LIMIT 3 Listing 1.7. (UC 2): What is the location of the lost child’s observers and their re- spective distance from the lost child? SELECT ?imei ?lat ?long ?distance { ?s rdf:type sosa:Observation ; sosa:isObservedBy ?o ; desmos:isDataOf ?p ; sosa:resultTime ?time ; desmos:distance ?distance . ?vol sosa:hosts ?o ; desmos:imei ?imei ; desmos:lat ?lat ; desmos:long ?long . ?p desmos:mac ?mac . FILTER(?imei = STR(imei) && ?time = time) } 6 Conclusion In this paper, we have discussed and demonstrated an ontology-based reasoning framework for the management of critical incidents in public events. We have Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 10 D. Ntioudis et al. presented the main components of the DESMOS semantic reasoning framework and how this has been incorporated in the DESMOS main framework. For future work, the framework will be tested and validated in two pilots in two different public places which protection and security challenges are higher. 7 Acknowledgements This research has been co-financed by the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH-CREATE-INNOVATE (project code: T1EDK-03487). References 1. Compton, M. The SSN ontology of the W3C semantic sensor network incubator group, J. Web Semant. 17, 25–32, 2012 2. Janowicz, K.; Haller, A.; Cox, S.; Le Phuoc, D.; Lefrançois, M. SOSA: A lightweight ontology for sensors, observations, samples, and actuators, J. Web Semant., 17, 1– 10, 2018 3. Raimond, Y. Event Ontology, Motools.sourceforge.net. 2019, Available online: http://motools.sourceforge.net/event/event.html (accessed on 31 January 2019) 4. Brickley, D.; Miller, L. FOAF Vocabulary Specification, Motools, Available online: http://xmlns.com/foaf/spec/ (accessed on 31 January 2019) 5. Matheus, C.; Kokar, M.; Baclawski, K. A core ontology for situation awareness, In Proceedings of the Sixth International Conference of Information Fusion, Cairns, Australia, 8–11, July 2003 6. Bobek, S.; Nalepa, G. Uncertain context data management in dynamic mobile en- vironments, In Future Generation Computer Systems; Elsevier: Amsterdam, The Netherlands, , pp. 110–124, (2017) 7. Pandey, Y., Bansal, S. K.,A Semantic Safety Check System for Emergency Manage- ment. Open Journal of Semantic Web (OJSW), 4(1), 35-50, (2017) 8. Hristoskova, A., Ongenae, F., De Turck, F. Semantic reasoning for intelligent emer- gency response applications, In 2013 11th IEEE International Conference on Indus- trial Informatics (INDIN) (pp. 547-554), IEEE, (2013) 9. Ahmad, A., Othman, R., Fauzan, M., & Ilyas, Q. M. A Semantic Ontology for Disaster Trail Management System, 10. Kontopoulos, E., Mitzias, P., Dasiopoulou, S., Moßgraber, J., Mille, S., Hertweck, P., ... & Kompatsiaris, I. Applying Semantic Web Technologies for Decision Support in Climate-Related Crisis Management 11. Uschold, M., & Gruninger, M. Ontologies: Principles, methods and applications, The knowledge engineering review, 11 (2), 93-136, (1996) 12. Zemmouchi-Ghomari, L., & Ghomari, A. R. Translating natural language com- petency questions into SPARQLQueries: a case study. In The First International Conference on Building and Exploring Web Based Environments (pp. 81-86), (2013) 13. Harris, S., Seaborne, A., & Prud’hommeaux, E. SPARQL 1.1 query language, W3C recommendation, 21 (10), 778, (2013) Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).