An approach for knowledge assisted semantic analysis and annotation of video content, based on an ontology infrastructure is presented. Semantic concepts in the context of the examined domain are defined in an ontology, enriched with qualitative attributes of the semantic objects (e.g. color homogeneity), multimedia processing methods (color clustering, respectively), and numerical data or low-level features generated via training (e.g. color models, also defined in the ontology). Semantic Web technologies are used for knowledge representation in RDF/RDFS language. Rules in F-logic are defined to describe how tools for multimedia analysis should be applied according to different object attributes and low-level features, aiming at the detection of video objects corresponding to the semantic concepts defined in the ontology. This supports flexible and managed execution of various application and domain independent multimedia analysis tasks. This ontology-based approach provides the means of generating semantic metadata and as a consequence Semantic Web services and applications have a greater chance of discovering and exploiting the information and knowledge in multimedia data. The proposed approach is demonstrated in the Formula One and Football domains and shows promising results.
As a result of recent progress in hardware and telecommunication technologies,
multimedia has become a major source of content on the World Wide Web, used
in a wide range of applications in areas such as content production and
distribution, telemedicine, digital libraries, distance learning, tourism, distributed
CAD/CAM, GIS, etc. The usefulness of all these applications is largely
determined by their accessibility and portability and as such, multimedia data sets
present a great challenge in terms of storing, querying, indexing and retrieval. In
addition, the rapid increase of the available amount of multimedia information
has revealed an urgent need for developing intelligent methods for
understanding and managing the conveyed information. To face such challenges
developing faster hardware or more sophisticated algorithms has become insufficient.
Rather, a deeper understanding of the information at the semantic level is
required [
Although new multimedia standards, such as MPEG-4 and MPEG-7 [
This is due to the difficulty, often mentioned as the semantic gap, in
capturing concepts mapped into a set of image and/or spatiotemporal features that
can be automatically extracted from video data without human intervention
[
In this paper, an approach for knowledge assisted semantic content analysis and annotation, based on a multimedia ontology infrastructure, is presented. Content-based analysis of multimedia requires methods which will automatically segment video sequences and key frames into image areas corresponding to salient objects, track these objects in time, and provide a flexible framework for object recognition, indexing, retrieval and for further analysis of their relative Domain Knowledge Base
RDFS ontology F-logic Rules Algorithm Repository
Main Processing Module
Multimedia MCuoltnitmenetdia Content Semantic Multimedia
Description
motion and interactions. This problem can be viewed as relating symbolic terms
to visual information by utilizing syntactic and semantic structure in a manner
related to approaches in speech and language processing [
Following this approach, the multimedia analysis and annotation process largely depends on the knowledge base of the system and as a result the method can easily be applied to different domains provided that the knowledge base is enriched with the respective domain ontology. Extending the knowledge base with spatial and temporal objects interrelations would be an important step towards the detection of semantically important events for the particular domain, achieving thus a finer, high-level semantic annotation. In addition, the ontologybased approach also ensures that semantic web services and applications have a greater chance of discovering and exploiting the information and knowledge in multimedia data.
The remainder of the paper is organized as follows: section 2 a detailed description of the ontology and rules developed is given, while in section 3, its application to the Formula One domain is described. Experimental results are presented in section 4. Finally, conclusions are drawn in section 5.
In order to realize the knowledge-assisted multimedia content semantic analysis and annotation technique explained in the previous section, an analysis and a domain ontology are constructed. The multimedia analysis ontology is used to support the detection process of the corresponding domain specific objects. Knowledge about the domain under discourse is also represented in the form of an ontology, namely the domain specific ontology. The domain-independent, primitive classes comprising the analysis ontology serve as attachment points allowing the integration of the two ontologies. Practically, each domain ontology comprises a specific instantiation of the multimedia analysis ontology providing the corresponding color models, restrictions e.t.c as will be demonstrated in more detail in section 3.
Object detection in general considers the exploitation of objects characteristic features in order to apply the most appropriate detection steps for the analysis process in the form of algorithms and numerical data generated off-line by training (e.g. color models). Consequently, the development of the proposed analysis ontology deals with the following concepts (RDFS classes) and their corresponding properties, as illustrated in Fig. 2: – Class Object: the superclass of all video objects to be detected through the analysis process. Each object instance is related to appropriate feature instances by the hasFeature property and to one or more other objects through a set of appropriately defined spatial properties. – Class Feature: the superclass of multimedia low-level features associated with each object. – Class Feature Parameter which denotes the actual qualitative descriptions of each corresponding feature. It is subclassed according to the defined features, i.e. to Connectivity Feature Parameter, Homogeneity Feature Parameter e.t.c. – Class Limit: it is subclassed to Minimum and Maximum and allows the definition of value restrictions to the various feature parameters. – The Color Model and Color Component classes are used for the representation of the color information, encoded in the form of the Y, Cb, Cr components of the MPEG color space. – Class Distribution and Distribution Parameter represent information regarding the defined Feature Parameter models. – Class Motion Norm: used to represent information regarding the object motion. – Class Algorithm: the superclass of the available processing algorithms (A1, A2,. . . ,An) to be used during the analysis procedure. This class is linked to the FeatureParameter class through the usesFeatureParameter property in order to represent the potential argument list for each algorithm. – Class Detection: used to model the detection process, which in our framework consists of two stages. The CandidateRegionSelection involves finding a set of regions which are potential matches for the object to be detected, isA
Total Dependency has
Dependency hasFeature hasFeature
Parameter has
Limit isA
Partial Dependency
Size
isA Connectivity
Texture isA isA Homogeneity
Feature Standard isA Deviation Mean Value isA
Limit isA Maximum
isA Minimum
Object
hasDetection isA
Object_2 isA Object_1
Feature
Parameter isA Connectivity Feature Parameter isA
Texture isA Feature Parameter Homogeneity
Feature Parameter Motion Norm
Color Model Distribution Parameter has ParameterValue hasDistribution Parameter "Integer"
Distribution
isA Gaussian
isA Candidate Region
Selection uses Feature Parameter
isA Earth's Mover
Distance- EMD hasColorComponent has Distribution hasDetection Part
Detection Detection Part
isA Final Region
Selection Algorithm has Detection
Step isA Four Connectivity Component Labelling isA K-means
Color Component
Y Cb Cr while FinalRegionSelection leads to the selection of only one region that best matches the criteria predefined for this object (e.g. size specifications). – Class Dependency: this concept addresses the possibility that the detection of one object may depend on the detection of another, due to possible spatial or temporal interrelations between the two objects. For example in the Formula One domain, the detection of the car could be assisted and improved if the more dominant and characteristic region of road is detected first. In order to differentiate between the case where the detection of object O1 requires the detection of the candidate regions of object O2 and the case where the entire final region of object O2 is required, PartialDependency and TotalDependency are introduced.
As mentioned before, the choice of algorithms employed for the detection
of each object is directly dependent on its available characteristic features. This
association is determined by a set of properly defined rules represented in F-logic.
F-logic is a language that enables both ontology representation and reasoning
about concepts, relations and instances [
The rules required for the presented approach are: rules to define the mapping between algorithms and features (which implicitly define the object detection steps), rules to determine algorithms input parameters, if any, and rules to deal with object interdependencies as explained above. The rules defined for each category have the following form: – “IF an object O has features F1 F2 . . . Fn as part of its qualitative description THEN algorithm A1 is a step for the detection of O.” – “IF an object O has feature F AND O has algorithm A as detection step
AND A uses feature F THEN A has as input the parameter values of F .” – “IF an object O1 has partial dependency on object O2 AND object O2 has as CandidateRegionSelection part the set S = {A1, A2, . . . , Am} THEN execute the set of algorithms included in S before proceeding with the detection of O1.” – IF an object O1 is totally dependent on object O2 THEN execute all detection steps for O2 before proceeding with the execution of O1 detection.” In order for the described multimedia analysis ontology to be applied, a domain specific ontology is needed. This ontology provides the vocabulary and background knowledge of the domain i.e. the semantically significant concepts and the properties among them. In the context of video understanding it maps to the important objects, their qualitative and quantitative attributes and their interrelations. 3
As previously mentioned, for the demonstration of the proposed approach the Formula One and Football domains were used. The detection of semantically significant objects, such as the road area and the cars in racing video for example, is an important step towards understanding and extracting the semantics of a temporal segment of the video by efficiently modelling the events captured in it. The set of features associated with each object comprises their definitions in terms of low-level features as used in the context of video analysis. The selection of the attributes to be included is based on their ability to act as distinctive features for the analysis to follow, i.e. the differences in their definitions indicate the different processing methods that should be employed for their identification. As a consequence, the definitions used for the Formula One domain are: – Car: a motion homogeneous (i.e. comprising elementary parts characterized by similar motion), fully connected region whose motion norm must be above a minimum value and whose size can not exceed a predefined maximum value. – Road: a color homogeneous, fully connected region, whose size has to exceed a predefined minimum value and additionally to be the largest such region in the video. – Grass: a color homogeneous, partly connected region with the requirement that each of its components has a minimum predefined size. – Sand: a color homogeneous, partly connected region with the requirement that each of its components has a size exceeding a predefined minimum.
In a similar fashion, the corresponding definitions for the Football domain include the concepts Player, Field and Spectators and their respective visual descriptions. As can be seen, the developed domain ontologies focus mainly on the representation of the object attributes and positional relations and in the current version does not include event definitions. For the same object, multiple instances of the Color Model class are supported, since the use of more than one color models for a single object may be advantageous in some cases. 3.1
Compressed-domain Video Processing and Rules
The proposed knowledge-based approach is applied to MPEG-2 compressed
streams. The information used by the proposed algorithms is extracted from
MPEG sequences during the decoding process. Specifically, the extracted color
information is restricted to the DC coefficients of the macroblocks of I-frames,
corresponding to the Y, Cb and Cr components of the MPEG color space.
Additionally, motion vectors are extracted for the P-frames and are used for
generating motion information for the I-frames via interpolation. P-frame motion
vectors are also necessary for the temporal tracking in P-frames, of the objects
detected in the I-frames [
The procedure for detecting the desired objects starts by performing a set of initial clusterings, using up to eight dominant colors in each frame to initialize a K-means algorithm. ¿From the resulting mask, which contains a number of nonconnected color-homogeneous regions, the non-connected semantic objects can be identified by color-model based selection. The application of a four connectivity component labelling algorithm results in a new mask featuring connected color-homogenous components. The color-model-based selection of an area corresponding to a color-homogeneous semantic object is performed using a suitable mask and the Earth Movers Distance (EMD). EMD computes the distance between two distributions represented as signatures and is defined as the minimum amount of work needed to change one signature into the other. Additional requirements as imposed by the models represented in the ontology, are checked to lead to the desired object detection. For motion-homogeneous objects a similar process is followed. At first, a mask containing motion-homogeneous regions is generated. Subsequently, the model- based selection depends on the information contained in the ontology (e.g. size restrictions, motion requirements).
The construction of the domain specific rules derives directly from the aforementioned video processing methodology. For example, since color clustering is the first step for the detection of any of the three objects, a rule of the first category without any feature matching condition is used to add the k-means algorithm as the first detection step to all objects. A set of different algorithms could have been used as long as the respective instantiations are defined. 4
The proposed approach was tested in two different domains: the Formula One
and the Football domain. In both cases, the exploitation of the knowledge
contained in the respective system ontology and the associated rules resulted to
the application of the appropriate analysis algorithms using suitable parameter
values, for the detection of the domain specific objects. For ontology creation the
OntoEdit ontology engineering environment [
For the Formula One domain our approach was tested on a one-hour video. As was discussed in section 3, four objects were defined for this domain. For those objects whose homogeneity attribute is described in the ontology by the Color Homogeneity class, the corresponding color models were extracted from a training set of approximately 5 minutes of manually annotated Formula One video. Since we assume the model to be a Gaussian distribution for each one of the three components of the color space, the color models were calculated from the annotated regions of the training set accordingly. Results for the Formula One domain are presented both in terms of sample segmentation masks showing the different objects detected in the corresponding frames (Fig. 3) as well as numerical evaluation of the results over a ten-minute segment of the test set (Table. 1). For the Football domain, the proposed semantic analysis and annotation framework was tested on a half-hour video, following a procedure similar to the one illustrated for the Formula One domain. Segmentation masks for this domain are shown in Fig. 4, while numerical evaluation of the results over a ten-minute segment of the test set for this domain are given in Table. 1.
For the numerical evaluation, the semantic objects appearing on each I-frame were manually annotated and compared with the results produced by the proposed system. It is important to note that the regions depicted in the generated segmentation masks correspond to semantic concepts and this mapping is defined according to the domain specific knowledge (i.e. object models) provided in the ontology. 5
In this paper we have presented an ontology-based approach for knowledge assisted domain-specific semantic video analysis. Knowledge involves qualitative object attributes, quantitative low-level features generated by training as well as multimedia processing methods. The proposed approach aims at formulating a domain specific analysis model with the additional information provided by rules, appropriately defined to address the inherent algorithmic issues.
Future work includes the enhancement of the domain ontology with more complex model representations, including spatial and temporal relationships, and the definition of semantically important events in the domain of discourse. Further exploration of low-level multimedia features (e.g. use of the MPEG-7 standardized descriptors) is expected to lead to more accurate and thus efficient representations of semantic content. The above mentioned enhancements will allow more meaningful reasoning, thus improving the efficiency of multimedia content understanding. Another possibility under consideration is the use of a more expressive language, e.g. OWL, in order to capture a more realistic model of the specific domain semantics. Object correct detections false detections missed