Ontologies play a key role in the Semantic Web, providing the machine-interpretable semantic vocabulary and serving as the knowledge representation and exchange vehicle. On top of ontologies, various functionalities can be supported, such as semantic integration, enrichment and reasoning to either further enhance or enrich them with additional information, or to deduct implicit knowledge out of the already annotated information. This paper presents a holistic semantic model employed within the CONNEXIONs EU-funded project that is aimed at semantically representing and reasoning about all pertinent notions derived from the analysis of high volumes of heterogeneous data with the goal to ultimately improve the capabilities of Law Enforcement Agencies in their ght against crime and terrorism. The proposed model enables the compound of various important aspects that resolve around several sources of information considered important in this context, including online sources that are often adversely exploited, as well information produced by Internet of Thing devices, such as sensors and cameras.
The continuous evolution of serious and organized crime and terrorism poses
signi cant challenges to Law Enforcement Agencies (LEAs) that need to be
equipped with e ective and e cient tools and solutions for the detection,
prediction, investigation, and ultimately prevention of such activities. One particular
challenge highlighted by the recent terrorist attacks in many European cities1 is
the growing role that online channels (e.g., Surface, Deep, and Dark Web and
social media platforms) play towards enabling and facilitating terrorist causes,
including radicalization, recruitment, and training, as well as the planning and
coordination of terrorist attacks [
Commons License Attribution 4.0 International (CC BY 4.0). online sources so as to be in a position to detect, as early as possible, criminal and/or terrorist activities, in order to prevent and mitigate potential threats.
Moreover, during speci c missions and operations undertaken by LEAs, it is of high importance for police o cers (particularly those on the eld) to be constantly aware of the overall situation, and in particular of any potential threats. For instance, when policing large-scale events, such as music festivals, it would be extremely important for LEA personnel to be aware of any suspicious activities taking place, since such information would allow them to act faster and possibly prevent a potential terrorist action. For instance, the monitoring and analysis of data produced from Internet of Things (IoT) devices, such as wearable and xed sensors and cameras, could also assist towards this direction.
Therefore, the abundance of information produced from both IoT devices and online sources necessitates the development of methods and tools for e ciently representing processed and analyzed data based on a uniform modeling process. To this end, semantic structures (i.e., ontologies) can be utilized, which permit the semantic representation of all pertinent knowledge derived from the already analyzed data. Overall, ontologies permit the understanding, sharing, and reuse of knowledge across di erent systems, while also supporting various semantic operations, such as semantic integration, enrichment and reasoning, to either further enhance the available information or to infer logical conclusions out of the already semantically annotated information.
In this paper, we discuss how Semantic Web technologies can be used to e ectively capture, interpret, and reason about information that revolves around criminal and terrorist activities, including evidence obtained both from online environments and also from sensors. The proposed approach has been developed within the context of the CONNEXIONs EU-funded project (https://www. connexions-project.eu/) and consists of an ontology and a range of semanticbased functionalities. The overall objective is to provide intelligence to LEAs that would allow them to improve their ability to analyze evidence and investigate crime and terrorism with the maximum possible e ectiveness and e ciency. 2
CONNEXIONs aims to develop a next-generation platform that will assist LEAs to improve their capabilities to gather intelligence, analyze evidence, and thus investigate crime and terrorism in an e ective and e cient manner.
The rst step towards investigating crime and terrorism e ectively and efciently is to collect data related to a speci c case. CONNEXIONs considers data obtained from various online sources, including the Surface, Deep, and Dark Web, and social media platforms, as well as data obtained from other sources, such as IoT devices and police reports; in addition, it can also consider digital evidence obtained from seized devices. This allows LEAs to acquire a more comprehensive view of potentially criminal and terrorism activities, thus enabling them to increase their chances to ultimately prevent them in time.
The next step constitutes the integration and correlation of such information through advanced methods (e.g., through ontologies and their supported functionalities) to nally deliver pertinent information in an interactive manner to a variety of end users, such as eld o cers, investigators, and analysts. To this end, immersive environments (such as virtual and augmented reality) are also considered for the delivery of such information in order to improve situational awareness, investigation, and training capabilities.
It is evident from the above that the establishment of a well-de ned data representation framework is of high importance, since it permits the capturing of several modalities and thus enables the subsequent application of a variety of analytics tools. The following sections describe the knowledge structures of the CONNEXIONs ontology and the reasoning services that allow the deduction of logical inferences out of the already semantically annotated information (i.e., facts and relationships). 3
Ontologies are the starting point and the key component of Semantic Web
technologies as they model the pertinent knowledge of any studied system. One
de nition states \an ontology is a formal, explicit speci cation of a shared
conceptualization that is characterized by high semantic expressiveness required for
increased complexity " [
Turner [
Moreover, Benahmed et al. [
Other relevant approaches include the one by Inyaem et al. [
The main limitation of the ontologies presented above lies in their narrow focus, which typically revolves around criminal/terrorist groups and their intentions, attacks, and impacts. Though absolutely essential, these aspects constitute only a fragment of the knowledge required during the detection, prediction, and investigation of criminal and terrorist activities. The CONNEXIONs ontology, on the other hand, aims to provide a more holistic semantic model, covering various additional aspects besides the aforementioned ones, such as online behaviors, knowledge derived from the analysis of sensors-based data, as well as spatiotemporal information. 3.2
The CONNEXIONs ontology aims to provide the means to model the relevant knowledge regarding the studied system, in order to enhance the decision support and reasoning capabilities for ghting crime and terrorism.
One important feature of the ontologies is their interoperability which is ensured by using common ontologies (and vocabularies) or by semantically mapping entities between di erent ontologies. Interoperability enables sharing knowledge from di erent repositories that are expressed using the same semantics. In practice, it is bene cial to reuse ontologies and vocabularies that are established provided they meet the particular needs.
Speci cally, the CONNEXIONs ontology reuses the following resources about
spatial information and multimedia modeling:
1. The Basic Geo (WGS84 lat/long) Vocabulary [
The CONNEXIONs ontology is formalized in OWL [
The speci cation of the ontology is the rst step towards exploiting semantic web technologies towards a reasoning framework that will aid the LEAs on crime event situations. In the following section the most important classes and properties of the CONNEXIONs ontology are presented (see also Figure 1).
The class Event represents a situation that takes place and is of relevance to the system; e.g., a social event, a crime or even an online event. This situation typically involves multiple entities that have varying roles and a ect di erently the event. Such entities with active role within the frame of the event are called Agents and their main characteristic is that they have the ability to a ect the Event. Agents may be living beings (e.g., Persons) or web agents, namely, software entities that act on the Web (e.g., User Account ). Additionally, Agents can perform activities that carry some signi cance for the event (the Activity class). Finally, agents might possess Physical Objects, such as sensors (Sensing Device). Physical objects can be carried or be stationary, where in both cases, the speci cation of their location is important (has location property).
Beyond the agents and the physical objects present at an event, online activity can provide a more complete view over the event. To this end, the ontology adopts the extended version of SIMMO model to express such online activities. In particular, the classes User Account and Web Domain are used to identify relevant online activities. In order to represent the activities and the artefacts posted online, the class simmo:Document is used in order to model multimodal and complex entities. Such documents can range from posts that contain a single text passage to web pages that contain multiple images and videos. Moreover, documents can reference (simmo:reference) other documents, or can be associated with other entities by a shared topic. User accounts are one of the more important contributors to the creation of documents and they can have relationships between them, e.g., a user account can be a liated with others. Relationships between user accounts and documents are mainly expressed via the property simmo:contribution to express the creation of the document by the user.
Spatial information is crucial towards a complete understanding of a situation, especially in case of real world applications. For purely spatial information, the class Location is used that represents spatial information by reusing the Basic Geo Vocabulary that represents the locations as points in space. However, in time sensitive situations the spatial con gurations are often fast changing, thus, a static spatial information is not su cient. For this reason, we proposed the combination of temporal and spatial information using the Spatio-Temporal Context class (Figure 2), which is responsible for keeping a history of locations and their correspondent times. ontology. In the context of the CONNEXIONs project, the data structure that holds the semantic information is an RDF database and comprises a Knowledge Base (KB) with an interface that utilizes the SPARQL (SPARQL Protocol and RDF Query Language) query language in order to insert and retrieve data. SPARQL is a semantic query language that operates on RDF data (W3C standard) and one of key technologies of the Semantic Web. At a higher logical level, the operations implemented with SPARQL are semantic integration, enrichment and reasoning. The rst two are responsible for inserting knowledge into the KB, while the third for inferring hidden knowledge from the available knowledge. 4.1
Semantic integration is the process of consuming structured data that can be translated to semantic information and consequently be populated to the KB. Typically, the information is provided in the form of JSON (JavaScript Object Notation) data and the relevant information is extracted from it. In order for the information to be inserted to the KB it is transformed to SPARQL queries that update the contents of the KB. E.g., if face recognition was to performed on video footage obtained from some event, then the resulting information could include the following: { Event ID, i.e., the ID of the social event where the person was recognized, e.g., \1234". { Person ID, i.e., the ID of the recognized person, e.g., \person 5262". { Timestamp, i.e., the moment when the person was detected, e.g., \2019-0517T14:00:00Z". { The coordinates of the person, e.g., \latitude: 40.62" and \longitude: 22.94".
In this case the corresponding SPARQL query that could be used to populate the new information to the KB would be similar to the presented in Listing 1.1. INSERT { : person_5262 rdf : type : Person . : person_5262 rdf : type owl : NamedIndividual . : person_5262 : connID " 5262 ". : loc_5262 rdf : type : Location . : loc_5262 geo : lat " 40.62 " ^^ xsd : double . : loc_5262 geo : long " 22.94 " ^^ xsd : double . : stc_5262 rdf : type : Spatio - TemporalContext . : stc_5262 rdf : type owl : NamedIndividual . : stc_5262 : hasLocation : loc_5262 . : stc_5262 : occurredAt " 2019 -05 -17 T14 :00:00 Z" ^^ xsd : dateTime . : person_5262 : hasSpatio - TemporalContext : stc_5262 . ? event : hasParticipant : person_5262 . ? event_loc : containsLocation : loc_5262 . ? event : connID " 1234 ". ? event : hasSpatio - TemporalContext ? event_stc .
? event_stc : hasLocation ? event_loc . 4.2
Semantic enrichment is the process of enhancing the context of individual entities within the KB using a variety of sources. This process resembles semantic integration in that they both aim to populate the ontology. However, semantic enrichment typically refers to external sources, such as DBpedia (https:// wiki.dbpedia.org/) and other public Knowledge Bases (e.g., criminal records).
In particular, let us examine the case of reporting a past violent activity against a VIP. This particular enrichment example ties with the integration discussed above and enriches the already known information. Data from the criminal records would include: { Person (determined by their ID) who committed the violent activity, e.g., \5262". { Target of the activity (ID), i.e., the entity that was intended to be harmed via the activity, e.g., \7382". { Timestamp, the moment when the person was detected, e.g., \2017-0412T14:00:00Z". { The coordinates of the person, e.g., \latitude: 40.65" and \longitude: 23.01".
The SPARQL code that is presented in Listing 1.2 is produced given the previous information. The code is executed and the resulted changes are made to the KB. In this case, the entities that correspond to the event and the VIP participant were added in advance via integration methods, similar to the previous example.
INSERT { : act_0291 rdf : type : ViolentActivity . : act_0291 rdf : type owl : NamedIndividual . : loc_0291 rdf : type : Location . : loc_0291 rdf : type owl : NamedIndividual . : loc_0291 geo : lat " 40.65 " ^^ xsd : double . : loc_0291 geo : long " 23.01 " ^^ xsd : double . : stc_0291 rdf : type : Spatio - TemporalContext . : stc_0291 rdf : type owl : NamedIndividual . : stc_0291 : hasLocation : loc_0291 . : stc_0291 : occurredAt " 2019 -04 -12 T14 :00:00 Z" ^^ xsd : dateTime . : act_0291 : hasSpatio - TemporalContext : stc_0291 . : act_0291 : against ? vip .
? person : performs : act_0291 . Semantic reasoning is the automated process of inferring implicit information from the knowledge available in the KB. Semantic reasoning is performed as a set of SPARQL rules that evaluate the necessary conditions for the inference to be valid. The inferences that result from the semantic reasoning are stored in the KB and can be forwarded to other tools if needed.
Semantic reasoning is performed automatically whenever the KB is updated via semantic integration and enrichment operations. In those cases, the semantic content changes and reasoning is applied to include the new information. Additionally, the semantic reasoning could be itself an enrichment process, by enriching the KB with new inferred information. In practice, semantic reasoning is triggered mainly by new analysis results becoming available. Moreover, a second method that triggers semantic reasoning is the use of speci c reasoning requests from other tools or the end users. If this method is deployed as a second way to initiate reasoning, such reasoning request must be well de ned. It is possible to allocate some reasoning methods to each of the two triggering mechanisms.
An example of semantic reasoning could examine whether a person that has performed violent actions against a VIP in the past attends the same social event as the VIP, then the person can be characterized as a potential threat. The SPARQL query (Listing 1.3) implements such a reasoning rule, and it is validated when the rule conditions are reached. Additionally, if the rule is realized, the appropriate updates are made to the KB.
SELECT ? person ? vip WHERE { ? event : hasParticipant ? person . ? event : hasVIPParticipant ? vip . ? person : performs ? activity . ? activity rdf : type : ViolentActivity .
? activity : against ? vip . } 5
Listing 1.3. SPARQL query for semantic reasoning
This paper presented the Semantic Web technologies that are used in the context of the CONNEXIONs project towards an enriched reasoning framework for the support of LEAs in their ght against crime and terrorism. First, the ontology that acts as the basis for the semantic representation of multimodal heterogeneous and complex data, that are pertinent to the studied system, was presented. Next, the available semantic operations that aim to manipulate e ectively the available data and provide useful insights about implicit knowledge that derives from the available data were illustrated. Overall, the presented framework is currently being successfully applied within the CONNEXIONs project and we aim to incorporate more complex semantic reasoning rules in the future. Acknowledgments This work has received funding from the European Union's H2020 research and innovation programme as part of the CONNEXIONs (GA No 786731) project.