The detection and representation of events is a critical element in automated surveillance systems. We present here an ontology for representing complex semantic events to assist video surveillance-based vandalism detection. The ontology contains the de nition of a rich and articulated event vocabulary that is aimed at aiding forensic analysis to objectively identify and represent complex events. Our ontology has then been applied in the context of London Riots, which took place in 2011. We report also on the experiments conducted to support the classi cation of complex criminal events from video data.
In the context of vandalism and terrorist activities, video surveillance forms an integral part of any incident investigation and, thus, there is a critical need for developing an \automated video surveillance system" with the capability of detecting complex events to aid the forensic investigators in solving the criminal cases. As an example, in the aftermath of the London riots in August 2011 police had to scour through more than 200,000 hours of CCTV videos to identify suspects. Around 5,000 o enders were found by trawling through the footage, after a process that took more than ve months.
With the aim to develop an open and expandable video analysis
framework equipped with tools for analysing, recognising, extracting and classifying
events in video, which can be used for searching during investigations with
unpredictable characteristics, or exploring normative (or abnormal) behaviours,
several e orts for standardising event representation from surveillance footage
have been made [
While various approaches have relied on o ering foundational support for the domain ontology extension, to the best of our knowledge, a systematic ontology for standardising the event vocabulary for forensic analysis and an application of it has not been presented in the literature so far.
In this paper, we present an OWL 2 [
Our work signi cantly extends the preliminary works [
The remainder of the paper is organised as follows. Related work is addressed in Section 2. Section 3 presents a detailed description of the forensic ontology about complex criminal events. In Section 4 we discuss how to use the ontology to assist video surveillance-based vandalism detection. In Section 5 we conduct some experiments with our ontology based on CCTV footage of London riots from 2011, and nally, Section 6 concludes. 2
In [
The Event Model E [37] has been developed based on an analysis and
abstraction of events in various domains such as research publications, personal
media [
In [
While the above-mentioned approaches essentially provide frameworks for the representation of events, none of them address the problem of formalising forensic events in terms of a standard representation language such as OWL 23 and, importantly, none have been applied and tested so far in a real use case, which are the topics of the following sections. 3
In the following, we present an OWL 2 ontology to support to some extent the semantic retrieval of complex events to aid automatic or semi-automatic video surveillance-based vandalism detection. The idea is to develop an ontology that not only conveys a shared vocabulary, but some inferences based on it may assist a human being to support the video analysis by hinting to videos that may be more relevant than others in the detection of criminal events. 3.1
To facilitate the elimination of the terminological ambiguity and the
understanding and interoperability among people and machines [
Our complex event classes extend DOLCE's Perdurant class. To assign the action
classes into respective categories, we follow a four-way classi cation of
actionverbs: namely, into State, Process, Achievement and Accomplishment using event
properties such as telic, stage and cumulative (see [
Forensic Perdurant Entities. Perdurant entities extend in time by accumulating di erent temporal parts and some of their proper temporal parts may be not present. To this end, Perdurant entities are divided into the classes Event and Stative, classi ed according to their temporal characteristics.
The axiom sets below provide a subset of our formal extension of the Perdurant vocabulary.
Event
Stative Achievement
Accomplishment Saying Seeing
Physical Aggression
Process
Action Gesture
State MataLevel
Event Psycological Aggression An excerpt of the forensic ontology is shown in Figure 1.
The concept State o ers representation for MetaLevelEvent which encompasses abstract human events such as Accusing, Believing and Liking among others. As previously stated, the concept State represents a collection of events which are exhibited by a human that is time-consuming, non-dynamic, cumulative and homogenous. The other sub-class of State is PsychologicalAggression which characterises the human actions such as Blaming, Decrying, Harassing and so forth. The concept Process includes several human action categories that represent dynamic events which can be split into several intermediate stages for analysis. For the purposes of clarity, the concept Process o ers three sub-concepts namely Action, Gesture and PhysicalAggression. The Action class incorporates di erent event such as Dancing, Greeting, Hugging among other concepts de ned. The concept Gesture formalises the di erent interest points related to human gestures. In order to eliminate the ambiguity traditionally present in human gestures across cross-cultural impact, the action performed during the gesture is captured and represented in the ontology and, thus, enabling the removal of subjectivity from the concept de nition. The nal sub-class of the Process class includes the concept PhysicalAggression and formalises human con icting actions.
By and large, the human actions categorised into State and Process represent the microscopic movements of humans.
From the automatic surveillance viewpoint, these microscopic events may be extracted from media items. In contrast, the event representations formalised by means of the concepts Achievement and Accomplishment o er a rich combination of human events that allow for the construction of complex events with or without the combination of microscopic features. For instance, the concept Unlawful Entry
Attempted
ForcibleEntry Forcible Entry Entering Property Cyber stalking Cyber mobbing
TheftOf
Information TheftOf Identity
TheftOf Password Vandalism Molotov Throwing
Gun Shot Damage Structure Damage
Apartment Cyber Bullying Cyber Crime
Phishing
Blackmail Cyber Threat
Botnet Malware
Hacking hierarchy for Vandalism is illustrated in Figure 2, while the concept hierarchy for CyberCrime is shown in Figure 3 instead.
Forensic Endurant Entities. DOLCE is based on fundamental distinction
among Endurant and Perdurant entities. The di erence between Endurant and
Perdurant entities is related to their behaviour in time. Endurant are wholly
present at any time they are present. Philosophers believe that endurant are
entities that are in time while lacking, however, temporal parts [
Material Artifact
Axiom set (1) describes a subset formalization of the Endurant vocabulary and an excerpt of the forensic extension of the ontology structure shown in Figure 4.
participantIn = participant
(1) 4
We next show how the so far developed ontology is expected to be used to assist video surveillance-based vandalism detection. 4.1
Given surveillance videos and any media in general, we need a method to annotate them by using the terminology provided by our ontology. This gives rise to a set of facts that, together with the inferred facts, may support a more e ective automatic or, more likely, semi-automatic retrieval of relevant information, such as e.g. vandalic acts. Speci cally, the inferred information may suggest a user look at some e.g. video sequences or video still images, rather than to others rst.
The general model we are inspired on is based on [
For instance, stating that an image object o is about a DamageVehicle can be represented conceptually via the DL expression
(9isAbout:DamageVehicle)(o) :
As speci ed in [
We recall that Resources (and Sources) are modelled as follows:
u9hasCameraId:string u9hasLatitude:string u9hasLongitute:string u9hasLocationName:string
has = isFrom has has v has : Note that in the last role inclusion axiom, is role composition and, thus, has has v has dictates that the property has is transitive, while with has = isFrom we say that isFrom is the inverse of has. Therefore, isFrom is transitive as well.
The following example illustrates the mechanism of image of annotation together with a meaningful inference.
Example 1. Consider the following DL axioms resulting from annotating images of a video (video6) registered by a camera (cameraC004): participateIn(personA; throwing5) ; Throwing(throwing5)
isFrom(throwing5; endurant6) ; Resource(endurant6) hasVideoId(endurant6; video6) ; Source(endurant7) hasCameraId(endurant7; cameraC004) ; has(endurant7; endurant6) : Now, as isFrom is transitive, we may infer: Then, it is not di cult to see that we nally infer isFrom(throwing5; endurant7) :
which can be read as: \A person (PersonA) participated in a physical aggression that has been registered by camera C004". 4.2
As we are focusing on forensic domain and dealing with variety of concepts aiming at aiding forensic analysis, to objectively identify and represent complex events, we next show that a (manually build) General Concept Inclusion (GCI) axiom may help to classify high-level events in terms of a composition of some lower level events. The following are such GCI examples:
\If an event involves a vehicle that is subject of a breaking door or breaking windows then the event is about a damaged vehicle" (see Figure 6).
\If an event involves a structure that is subject of kicking, then the event is about a damaged structure" (see Figure 6).
The following example illustrates the use of such GCIs.
Example 2. Suppose we have an image classi er that is able to provide us with the following facts. Speci cally, assume it is able to identify vehicles and breaking windows:
participant(Perdurant2; Endurant1); Vehicle(Endurant1) ; BreakingWindows(Perdurant2) : From these facts and the GCI about DamageVehicle, we may infer that the image is about a damaged vehicle, i.e. we may infer
DamageVehicle(Perdurant2) : The following set of GCIs illustrates instead how one may have multiple GCIs to classify a single event, such as those for Vandalism (see, e.g. Figure 7).8 Note that in the example above, we assume that events (perdurant) may be complex in the sense that they may compose by multiple sub-events (parts). So, e.g. in the last GCI, we roughly state 8 Recall that all these GCIs provide su cient conditions to be an instance of Vandalism, but no necessary condition.
\If a (complex) event involves both throwing and an explosion (two subevents) then the event is about vandalism".
Following our previous examples, we next are going to formulate another kind of background knowledge. Our main focus in this example is on recognizing highlevel events, which occur in the same location (same street in our modelling). In order to model this scenario, we may use the Semantic Web Rule Language (SWRL) to model the locatedSameAs role and then use it in GCIs. The SWRL rule is: \Two perdurants that occur in the same street occur in the same place.
Perdurant(?p1); Perdurant(?p2); hasLocationName(?p1; ?l1);
hasLocationName(?p2; ?l2); SameAs(?l1; ?l2) ! locatedSameAs(?p1; ?p2) :
The following axioms illustrate how to use the previously de ned relation (few examples captured from our data set by these rules are illustrated in Figure 8).
5
We conducted two experiments with our ontology, which we are going to describe in the following.9
In the rst case, we evaluated the classi cation e ectiveness of manually built GCIs to identify crime events, while in the second case we drop the manualbuilt GCIs and, try to learn such GCIs instead automatically from examples and compare their e ectiveness with respect to the manually built ones. 9 The ontologies used in the experiments and experimental results can be found at http://www.umbertostraccia.it/cs/ftp/ForensicOntology.zip. 5.1
Roughly, we have considered a number of crime videos, annotated them manually and then checked whether the manually built GCIs, as described in Section 4.2, were able to determine crime events correctly.
Setup. Speci cally, we considered our ontology and around 3.07 TB of video data about the London riot 2011,10 of which 929 (GB) is in a non-proprietary format. We considered 140 videos (however, the videos cannot be made publicly available). Within these videos, all the available CCTV cameras (35 CCTV) along with their features such as latitude, longitude, start time, end time and street name, have been annotated manually according to our methodology described in Section 4 and included into our ontology. We have also calculated all the geographic distances between each camera. The resulting ontology contains 1800 created individuals of which, e.g. 106 are of type Event.
Vandalism (13; 57) Riot (4; 21) AbnormalBehavior (2; 80) Crowding (1; 64) DamageStructure (3; 9) DamageVehicle (3; 16)
Throwing (1; 30) Then, we considered criminal events occurring in the videos (speci cally, we focused on vandalic events). For each class of events, we manually built one or more GCIs, as illustrated in Section 4.2. The list of crime events considered is reported in Table 2. In it, the rst number in parenthesis reports the number of GCIs we built for each of them, while the second number indicates the number of event instances (individuals) we created during the manual video annotation process. So, for instance, for the event DamageStructure we have built 3 classi cation GCIs and we have created 9 instances of DamageStructure during the manual video annotation process. For further clari cation, the 3 GCIs for DamageStructure are
while, e.g., an instance of DamageStructure is the individual Kicking1, whose related information excerpt is:
Kicking(Kicking1); isFrom(Kicking1; 2bdf); Resource(2bdf); isFrom(2bdf; C004); has(2bdf; pr11); part(pr11; Kicking1); part(pr11; BreackingWindows3);
BreackingWindows(BreackingWindows3); : : : As a matter of general information, the global metric statistics of the so built ontology is reported in Table 3. 10 These are part of the EU funded project LASIE \Large Scale Information Exploitation of Forensic Data", http://www.lasie-project.eu. Evaluation. Let O be the built ontology from which we drop axioms stating explicitly that an individual is an instance of a crime event listed in Table 2. Please note that without the GCIs none of the crime events instances in O can be inferred to be instances of the crime events in Table 2.11 Now, on O we run an OWL 2 reasoner that determines the instances of all crime event classes in the ontology.
To determine the classi cation e ectiveness of the GCIs, we compute the so-called micro/macro averages of precision, recall and F1-score w.r.t. inferred data. The evaluation result of the rst test is shown in Table 4. In the second experiment, we apply a concept learning approach to replace the manually built GCIs describing the crime events listed in Table 2. To this end, 11 Roughly, crime events are subclasses of the Event class, while crime event instances are instances of the class Stative (see Figure 3). the DL-Learner12 system was used to learn descriptions of the criminal events in Table 2, based on existing instances of these classes.
Setup. Let now O be the ontology as in Section 5.1, but from which we
also drop additionally the manually created GCIs for the crime event listed in
Table 2. On it we used the CELOE algorithm [
Speci cally, we used a K-fold cross style validation method [
i=1 Ci) n Ci and is denoted as T rainseti. Then, for each Ci we run CELOE on the training set T rainseti and generated at most 10 class expressions of the form Dj v Ci, out of which we have chosen the best solution (denoted DCi v Ci or GCIi). If the best solution is not unique, we select the rst listed one.
The best-selected GCIs found by CELOE for each of the target classes in Table 2 are:
With the help of a reasoner, we then infer all instances in O, that are not in T rainseti, that are instances of the selected DCi and consider them as our result set (denoted Resultseti).
Evaluation. To determine the classi cation e ectiveness of the learned GCIs, i.e. of GCIi, average precision, recall and F1 measures across the folds are computed. The evaluation results of the second test are shown in Table 5. Discussion. The results are generally promising. In the manually built GCI case, precision and F1 are reasonably good, though in one case (Riot) the recall and, thus, F1 is not satisfactory. For the learned GCI case, the individual measures are generally comparable to the manual ones. 12 http://dl-learner.org/
Given that the learned GCIs are completely di erent than the manually built ones, it is surprising that both sets perform more or less the same. However, please note that DL-Learner was neither able to learn a GCI for Vandalism nor for Riot. This fact is re ected in the generally worse micro/macro precision, recall and F1 measures.
Eventually, we also merged the manually built GCIs and the learned ones together and tested them as in Section 5.1. The results in Table 6 show, however, that globally their e ectiveness is as for the manual case (and does not improve). In this work, we have proposed an extensive ontology for representing complex criminal events. The proposed ontology focuses on events that are often required by forensic analysts. In this context, the Perdurant, as de ned in the DOLCE ontology as an occurrence in time, and the Endurant, de ned in the DOLCE ontology as contentious in time, have both been extended to represent all forensics entities together with meaningful entities for video surveillance-based vandalism detection. The aim of the built ontology is to support the interoperability of the automated surveillance system.
To classify high-level events in terms of the composition of lower level events we focused on both manually built and automatically learned GCIs and have compared the evaluation results of both experiments. The results are generally promising and the e ectiveness of machine derived de nitions for high-level crime events is encouraging though needs further development.
In the future, we intend to deal with vague or imprecise knowledge and we
would like to work on the problem of automatically learn fuzzy concept
description [
ACM, Salamanca, Spain (2015) 35. Vendler, Z.: Verbs and times. The Philosophical Review 62(2), 143{160 (1957) 36. Vendler, Z. (ed.): Linguistics in Philosophy. G - Reference, Information and Interdisciplinary Subjects Series, Cornell University Press (1967), https://books. google.co.uk/books?id=OR1DAAAAIAAJ 37. Westermann, U., Jain, R.: Toward a common event model for multimedia applications. IEEE Multimedia 14(1) (2007)