The continuous evolution of serious and organized crime and terrorism poses significant challenges to Law Enforcement Agencies (LEAs) that need to be equipped with effective and efficient tools and solutions for the detection, prediction, investigation, and ultimately prevention of such activities. One particular challenge highlighted by the recent terrorist attacks in many European cities 1 is the growing role that online channels (e.g., Surface, Deep, and Dark Web and social media platforms) play towards enabling and facilitating terrorist causes, including radicalization, recruitment, and training, as well as the planning and coordination of terrorist attacks [14]. Thus, there is a need for LEAs to monitor online sources so as to be in a position to detect, as early as possible, criminal and/or terrorist activities, in order to prevent and mitigate potential threats.
Moreover, during specific missions and operations undertaken by LEAs, it is of high importance for police officers (particularly those on the field) to be constantly aware of the overall situation, and in particular of any potential threats. For instance, when policing large-scale events, such as music festivals, it would be extremely important for LEA personnel to be aware of any suspicious activities taking place, since such information would allow them to act faster and possibly prevent a potential terrorist action. For instance, the monitoring and analysis of data produced from Internet of Things (IoT) devices, such as wearable and fixed sensors and cameras, could also assist towards this direction.
Therefore, the abundance of information produced from both IoT devices and online sources necessitates the development of methods and tools for efficiently representing processed and analyzed data based on a uniform modeling process. To this end, semantic structures (i.e., ontologies) can be utilized, which permit the semantic representation of all pertinent knowledge derived from the already analyzed data. Overall, ontologies permit the understanding, sharing, and reuse of knowledge across different systems, while also supporting various semantic operations, such as semantic integration, enrichment and reasoning, to either further enhance the available information or to infer logical conclusions out of the already semantically annotated information.
In this paper, we discuss how Semantic Web technologies can be used to effectively capture, interpret, and reason about information that revolves around criminal and terrorist activities, including evidence obtained both from online environments and also from sensors. The proposed approach has been developed within the context of the CONNEXIONs EU-funded project (https://www. connexions-project.eu/) and consists of an ontology and a range of semanticbased functionalities. The overall objective is to provide intelligence to LEAs that would allow them to improve their ability to analyze evidence and investigate crime and terrorism with the maximum possible effectiveness and efficiency. The next step constitutes the integration and correlation of such information through advanced methods (e.g., through ontologies and their supported functionalities) to finally deliver pertinent information in an interactive manner to a variety of end users, such as field officers, investigators, and analysts. To this end, immersive environments (such as virtual and augmented reality) are also considered for the delivery of such information in order to improve situational awareness, investigation, and training capabilities.
It is evident from the above that the establishment of a well-defined data representation framework is of high importance, since it permits the capturing of several modalities and thus enables the subsequent application of a variety of analytics tools. The following sections describe the knowledge structures of the CONNEXIONs ontology and the reasoning services that allow the deduction of logical inferences out of the already semantically annotated information (i.e., facts and relationships).
Ontologies are the starting point and the key component of Semantic Web technologies as they model the pertinent knowledge of any studied system. One definition states "an ontology is a formal, explicit specification of a shared conceptualization that is characterized by high semantic expressiveness required for increased complexity" [4]. In short, they provide the mold for the available data in order to build a homogeneous data repository that can effectively utilize Semantic Web technologies. In fact, such technologies are powerful tools towards enhancing decision support and reasoning capabilities towards fighting crime and terrorism. This section first discusses related work (Section 3.1), then outlines the approach adopted by the CONNEXIONs Ontology (Section 3.2), and finally presents the conceptualization of the proposed ontology (Section 3.3).
Turner [12] proposed the Adversary-Intent-Target (AIT) ontology, namely a model for semantically representing adversary groups and their intentions, a classification of their weapons and attack types, and the relationship between the outcomes of an attack and the various recognized intentions of the adversary group. Another prominent approach is the work by Mannes and Golbeck [8,9], which proposes an ontology for representing terrorist activity and addresses the key issues the authors encountered during the development of the ontology, mostly revolving around how sequences of events can be described, and also representing the social networks that underpin terrorist organizations.
Moreover, Benahmed et al. [1] proposed an ontology for automating the characterization and the classification of terrorist threats at early stages, aiming at a more efficient threat mitigation, while Galjano et al. [5] focused more on monitoring subjects and objects (targets) of potential interest in an effort to monitor terrorist threats. In addition, an ontology for uncovering terrorism-related hidden semantic associations, which was the result of a knowledge fusion from several existing ontologies and open knowledge systems on the Web, was proposed by Chmielewski et al. [3].
Other relevant approaches include the one by Inyaem et al. [7], where a fuzzy ontology (whereby relationships have degrees of membership) for representing terrorism events was proposed. Also, Najgebauer et al. [10] developed a terrorism ontology for representing terrorist threat indications and facilitating the early detection of terrorist action preparation activities, constituting the backbone of an early warning system. Finally, Veerasamy et al. [13] presented an ontology specifically developed for cyberterrorism, which is aimed at identifying whether a cyber-event can be classified as a cyberterrorist attack or a support activity, and provides a rich semantic representation of underlying relationships, interactions, and influencing factors.
The main limitation of the ontologies presented above lies in their narrow focus, which typically revolves around criminal/terrorist groups and their intentions, attacks, and impacts. Though absolutely essential, these aspects constitute only a fragment of the knowledge required during the detection, prediction, and investigation of criminal and terrorist activities. The CONNEXIONs ontology, on the other hand, aims to provide a more holistic semantic model, covering various additional aspects besides the aforementioned ones, such as online behaviors, knowledge derived from the analysis of sensors-based data, as well as spatiotemporal information.
The CONNEXIONs ontology aims to provide the means to model the relevant knowledge regarding the studied system, in order to enhance the decision support and reasoning capabilities for fighting crime and terrorism.
One important feature of the ontologies is their interoperability which is ensured by using common ontologies (and vocabularies) or by semantically mapping entities between different ontologies. Interoperability enables sharing knowledge from different repositories that are expressed using the same semantics. In practice, it is beneficial to reuse ontologies and vocabularies that are established provided they meet the particular needs.
Specifically, the CONNEXIONs ontology reuses the following resources about spatial information and multimedia modeling:
1. The Basic Geo (WGS84 lat/long) Vocabulary [2] is a lightweight ontology (formalized in OWL 2) that follows the specification of world geodesic system (revision 1984) for the representation of coordinates and altitude. This vocabulary directly supports only points in space that are defined by their coordinates. However, it can be extended to manipulate aggregations of points and subsequently form complex shapes. 2. The SIMMO (Socially Interconnected MultiMedia-enriched Objects) [11] is a model for expressing multimodal data in a social context with rich context.
In particular, it focuses on their structure, interconnections and provenance
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). which renders it highly useful and versatile in expressing online user activity and media. In order for the model to be imported in the CONNEXIONs ontology it was first ported as an OWL 2 ontology.
The CONNEXIONs ontology is formalized in OWL [6], which is recommended by W3C and is designed to model complex knowledge systems. OWL has rich expressivity and can capture intricate relationships between entities. Moreover, OWL is logic-based language and supports implicit knowledge extraction and consistency verification.
The specification of the is the first step towards exploiting semantic web technologies towards a reasoning framework that will aid the LEAs on crime event situations. In the following section the most important classes and properties of the CONNEXIONs ontology are presented (see also Figure 1). The class Event represents a situation that takes place and is of relevance to the system; e.g., a social event, a crime or even an online event. This situation typically involves multiple entities that have varying roles and affect differently the event. Such entities with active role within the frame of the event are called Agents and their main characteristic is that they have the ability to affect the Event. Agents may be living beings (e.g., Persons) or web agents, namely, software entities that act on the Web (e.g., User Account). Additionally, Agents can perform activities that carry some significance for the event (the Activity class). Finally, agents might possess Physical Objects, such as sensors (Sensing Device). Physical objects can be carried or be stationary, where in both cases, the specification of their location is important (has location property).
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Beyond the agents and the physical objects present at an event, online activity can provide a more complete view over the event. To this end, the ontology adopts the extended version of SIMMO model to express such online activities. In particular, the classes User Account and Web Domain are used to identify relevant online activities. In order to represent the activities and the artefacts posted online, the class simmo:Document is used in order to model multimodal and complex entities. Such documents can range from posts that contain a single text passage to web pages that contain multiple images and videos. Moreover, documents can reference (simmo:reference) other documents, or can be associated with other entities by a shared topic. User accounts are one of the more important contributors to the creation of documents and they can have relationships between them, e.g., a user account can be affiliated with others. Relationships between user accounts and documents are mainly expressed via the property simmo:contribution to express the creation of the document by the user.
Spatial information is crucial towards a complete understanding of a situation, especially in case of real world applications. For purely spatial information, the class Location is used that represents spatial information by reusing the Basic Geo Vocabulary that represents the locations as points in space. However, in time sensitive situations the spatial configurations are often fast changing, thus, a static spatial information is not sufficient. For this reason, we proposed the combination of temporal and spatial information using the Spatio-Temporal Context class (Figure 2), which is responsible for keeping a history of locations and their correspondent times.
Ontologies define a set of entities and the relationships between them at a theoretical level. However, in order to have practical applications these need to be emanated as a data structure that adheres to the semantic model defined by the Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). ontology. In the context of the CONNEXIONs project, the data structure that holds the semantic information is an RDF database and comprises a Knowledge Base (KB) with an interface that utilizes the SPARQL (SPARQL Protocol and RDF Query Language) query language in order to insert and retrieve data. SPARQL is a semantic query language that operates on RDF data (W3C standard) and one of key technologies of the Semantic Web. At a higher logical level, the operations implemented with SPARQL are semantic integration, enrichment and reasoning. The first two are responsible for inserting knowledge into the KB, while the third for inferring hidden knowledge from the available knowledge.
Semantic integration is the process of consuming structured data that can be translated to semantic information and consequently be populated to the KB. Typically, the information is provided in the form of JSON (JavaScript Object Notation) data and the relevant information is extracted from it. In order for the information to be inserted to the KB it is transformed to SPARQL queries that update the contents of the KB. E.g., if face recognition was to performed on video footage obtained from some event, then the resulting information could include the following:
-Event ID, i.e., the ID of the social event where the person was recognized, e.g., "1234". -Person ID, i.e., the ID of the recognized person, e.g., "person 5262".
-Timestamp, i.e., the moment when the person was detected, e.g., "2019-05-17T14:00:00Z". -The coordinates of the person, e.g., "latitude: 40.62" and "longitude: 22.94".
In this case the corresponding SPARQL query that could be used to populate the new information to the KB would be similar to the presented in Listing 1.1.
Semantic enrichment is the process of enhancing the context of individual entities within the KB using a variety of sources. This process resembles semantic integration in that they both aim to populate the ontology. However, semantic enrichment typically refers to external sources, such as DBpedia (https:// wiki.dbpedia.org/) and other public Knowledge Bases (e.g., criminal records).
In particular, let us examine the case of reporting a past violent activity against a VIP. This particular enrichment example ties with the integration discussed above and enriches the already known information. Data from the criminal records would include:
-Person (determined by their ID) who committed the violent activity, e.g., "5262". -Target of the activity (ID), i.e., the entity that was intended to be harmed via the activity, e.g., "7382". -Timestamp, the moment when the person was detected, e.g., "2017-04-12T14:00:00Z". -The coordinates of the person, e.g., "latitude: 40.65" and "longitude: 23.01".
The SPARQL code that is presented in Listing 1.2 is produced given the previous information. The code is executed and the resulted changes are made to the KB. In this case, the entities that correspond to the event and the VIP participant were added in advance via integration methods, similar to the previous example.
Semantic reasoning is the automated process of inferring implicit information from the knowledge available in the KB. Semantic reasoning is performed as a set of SPARQL rules that evaluate the necessary conditions for the inference to
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). be valid. The inferences that result from the semantic reasoning are stored in the KB and can be forwarded to other tools if needed.
Semantic reasoning is performed automatically whenever the KB is updated via semantic integration and enrichment operations. In those cases, the semantic content changes and reasoning is applied to include the new information. Additionally, the semantic reasoning could be itself an enrichment process, by enriching the KB with new inferred information. In practice, semantic reasoning is triggered mainly by new analysis results becoming available. Moreover, a second method that triggers semantic reasoning is the use of specific reasoning requests from other tools or the end users. If this method is deployed as a second way to initiate reasoning, such reasoning request must be well defined. It is possible to allocate some reasoning methods to each of the two triggering mechanisms.
An example of semantic reasoning could examine whether a person that has performed violent actions against a VIP in the past attends the same social event as the VIP, then the person can be characterized as a potential threat. The SPARQL query (Listing 1.3) implements such a reasoning rule, and it is validated when the rule conditions are reached. Additionally, if the rule is realized, the appropriate updates are made to the KB.
? event : hasP articip ant ? person . ? event : h a s V I P P a r t i c i p a n t ? vip . ? person : performs ? activity . ? activity rdf : type : V i ol en tA c ti vi t y . ? activity : against ? vip . } Listing 1.3. SPARQL query for semantic reasoning
This paper presented the Semantic Web technologies that are used in the context of the CONNEXIONs project towards an enriched reasoning framework for the support of LEAs in their fight against crime and terrorism. First, the ontology that acts as the basis for the semantic representation of multimodal heterogeneous and complex data, that are pertinent to the studied system, was presented. Next, the available semantic operations that aim to manipulate effectively the available data and provide useful insights about implicit knowledge that derives from the available data were illustrated. Overall, the presented framework is currently being successfully applied within the CONNEXIONs project and we aim to incorporate more complex semantic reasoning rules in the future.
Acknowledgments This work has received funding from the European Union's H2020 research and innovation programme as part of the CONNEXIONs (GA No 786731) project.