- In this paper, we present a methodology for ontology evolution, by focusing on the specific case of multimedia ontology evolution. In particular, we discuss the situation where the ontology needs to be enriched because it does not contain any concept that could be used to explain a new multimedia resource. The paper shows how ontology matching techniques can be used to enforce the discovery of new relevant concepts by probing external knowledge sources using both the information available in the multimedia resource and the knowledge contained in the current version of the ontology.
Ontology evolution regards the capability of managing the
modification of an ontology in a consistent way. An ontology
may change because the domain or the user needs have
changed [
In this paper, we focus on a specific case of evolution,
which is multimedia ontology evolution. In our approach, the
ontology is used to provide an interpretation for the elements
extracted from multimedia resources, such as images, textual
documents, video and audio. Our approach is defined in
the framework of the BOEMIE EU Project [
The paper is organized as follows. In Section II we describe the requirements for ontology evolution. The methodology we propose for multimedia ontology evolution is presented in Section III. A discussion of the activity of concept discovery, together with an example is provided in Section IV. Section V presents the related work in the field of ontology evolution. Conclusions and future work are discussed in Section VI.
II. REQUIREMENTS FOR THE ONTOLOGY EVOLUTION
In this paper, we address ontology evolution in a scenario
where:
• Multimedia resources are considered, which have been
already submitted to a semantic extraction process for
associating them with appropriate metadata that describe
their contents. Consequently, the ontology evolution is
triggered by incoming multimedia resources with
associated metadata information.
• An initial domain ontology is available, providing a
description of the domain of interest (in the worst case,
also a high-level, poorly detailed description). Such an
ontology is used also by the extraction process and will
be continuously evolved so that the new ontology version
is used as input for subsequent extraction and evolution
activities, according to a bootstrapping process which is
iterated until a satisfactory quality of multimedia resource
classification is reached. In the paper, we will consider the
Athletics domain and we will discuss ontology evolution
by providing some examples taken from this domain.
• An open system approach is adopted, in that when
background knowledge in the underlying ontology is not
sufficient to interpret new incoming multimedia resources,
a concept discovery activity is triggered to acquire new
knowledge from other external knowledge sources. This
way, we try to limit the human involvement of the
ontology expert in the definition of new concepts, which is
usually a manual and time consuming activity in ontology
evolution [
Before introducing our evolution methodology, we introduce some basic definition used throughout the paper: • A multimedia resource is any kind of multimedia object (e.g., image, video, sound) from which information is extracted. • Each individual extracted element from a multimedia resource is called a media object (e.g., a portion of an image). • With each media object some metadata are associated, which represent the semantic information that is extracted from the resource. • A simple concept is a concept in the domain ontology which explains a media object in terms of metadata. • An aggregate concept is a concept in the domain ontology which is defined as an aggregation of simple concepts, such as, for instance, an event. Furthermore, an aggregate concept is an explanation of the multimedia resource, that is an interpretation of the real event or object described in the resource.
To illustrate these concepts, we consider an evolution scenario in the Athletics domain characterized by the multimedia resource shown in Fig. 1. In this case, four media objects (marked in the figure) are identified during semantic extraction which have associated metadata stating that: (1) is a foam mattress, (2) is a pole, (3) is an horizontal bar, and (4) is an athlete, respectively. Moreover, with respect to the event (i.e., aggregate concept) described by the image, we can recognize that all these media objects are part of a bigger picture, which describes a pole vault event.
The metadata associated with the media objects of Fig. 1 are shown in Table I.
With respect to this example, the domain ontology
contains the aggregate concept Pole vault, while Horizontal bar,
Foam mat, Athlete and Pole are simple concepts. The
activity of classifying a new multimedia resource with respect
to the ontology is, in this context, the activity of finding an
aggregate concept in the ontology capable of explaining the
set of simple concepts detected in the multimedia resource
semantic extraction. Such an activity is called interpretation
of the multimedia resource, which can be performed by
exploiting non-standard reasoning techniques [
These four scenarios produce two main classes of evolution patterns, namely population and enrichment, respectively (see Fig. 2). In the following we briefly describe these two classes of patterns.
Evolution Pattern
Population Pattern Enrichment Pattern
Single explanation of the
multimedia resource Multiple explanation of the
multimedia resource Missing explanation of the multimedia resource with metadata information Missing explanation of the multimedia resource without
metadata information • Population patterns: the domain ontology contains the aggregate concepts as well as all the simple concepts described by the media objects metadata extracted from the multimedia resource. In this case, only ontology population has to be performed. Ontology population is the activity of introducing new instances in the ontology to describe the multimedia resource with respect to the aggregate concepts already present in the ontology. • Enrichment patterns: The ontology does not contain concepts that can be used to explain the multimedia resource. In this case, the ontology has to be enriched. Ontology enrichment is the activity of defining new concepts, properties, and/or semantic relations in the ontology. Basically, the idea behind our approach to ontology enrichment is to exploit the metadata available in the multimedia resource description in order to discover new aggregate concepts in external knowledge sources (e.g., other ontologies, web directories, lexical systems) that can be added to the domain ontology in order to explain the new multimedia resource. This activity is referred as concept discovery and it is executed by exploiting ontology matching techniques.
In this paper, we focus on the enrichment patterns and, in particular, we describe the case when all the simple concepts are already stored in the domain ontology, but the aggregate concept is missing, in order to show how the ontology can be enriched by the concept discovery activity.
III. METHODOLOGY FOR DISCOVERY- DRIVEN ONTOLOGY
EVOLUTION
In the evolution scenario discussed in the paper, the interpretation of a multimedia resource produces a description of the simple concept instances that are detected into the resource (e.g., the objects that appear into an image) but no aggregate concept (e.g., events) in the ontology can be found that explain the resource itself. In this section, we describe a methodology for ontology evolution in this scenario and the role of ontology matching techniques in supporting the identified evolution phases.
The goal of ontology evolution is to augment the background knowledge in the existing domain ontology to better classify extracted object descriptions. The ontology is enriched by adding concepts, properties, and/or semantic relations or by modifying existing ones, in order to produce a new ontology version capable of providing interpretation of the new multimedia resource.
In order to detect the ontology modifications required for explaining the new resources, we exploit the description of the media objects extracted from the resources. This means that, different from the usual evolution scenarios, in our case we know from the interpretation phase that an aggregate concept is missing in the domain ontology, but we do not know exactly which one. We have the instance of the missing concept and have to create an aggregate concept which explains it.
The evolution process is articulated in three phases, which are explained in the following.
Temporary concept definition: the incoming instance and its associated metadata information is used to define a temporary concept ct, which is matched against the domain ontology to detect semantically related concepts already present in the ontology. This internal concept detection activity is optional and aims at finding in the domain ontology some aggregate concepts which have semantic affinity with ct to be used as the starting basis in the next phase.
Concept discovery: if internal concepts have been detected in the previous phase, for each retrieved matching concept ci in the ontology (i.e., ci is semantically similar to the temporary concept ct) a probe query is created. Otherwise, the ontology expert has to define the probe query manually, by exploiting his own domain knowledge and the information available in the definition of the temporary concept ct. A probe query is a description of the properties, the constraints, and the semantic relations that should be exhibited by the (actually missing) new concept in order to consider it as a candidate for ontology enrichment. The probe query is then matched against one or more external knowledge sources to discover some external concepts that could be useful to define the new concept to be inserted in the ontology.
Ontology enrichment: this phase comprises the definition
and insertion of the new (aggregate) concept in the domain
ontology and also the validation of the resulting ontology.
The definition of the new concept is interactively performed
by the ontology expert, by exploiting the definitions of the
temporary concept ct and of the external matching concepts
retrieved during the concept discovery phase. Once the new
aggregate concept has been defined, the ontology expert inserts
it in a proper position in the ontology taxonomy. The task of
finding the most suitable placement of the new concept in
the ontology taxonomy is performed by combining ontology
matching techniques and ontology reasoning, as described
in [
As described in the previous section, ontology matching
techniques play an essential role in the ontology evolution
process. In temporary concept definition, the temporary
concept can be compared against the existing ontology to find
internal matching concepts that can help the probe query
formulation activity. In concept discovery, a probe query is
compared against external knowledge sources with the goal
of finding potentially relevant knowledge. To perform these
concept comparisons, we rely on our semantic matchmaker
H-M ATCH developed in the framework of the HELIOS project
for matching independent peer ontologies [
SA(c, c0) = WLA · LA(c, c0) + (1 − WLA) · CA(c, c0) (1)
where WLA is a weight expressing the relevance to be given
for the linguistic affinity in the semantic affinity evaluation
process. A detailed description of H-M ATCH and related
matching models is provided in [
IV. CONCEPT DISCOVERY AND ONTOLOGY ENRICHMENT }
In this section, we describe in more detail how concept discovery and ontology enrichment are performed, with respect to the jumping example of Fig. 1.
The concept discovery activity is articulated in the following steps: 1) Given a multimedia resource r and the set M of media objects that have been extracted from r, we build a temporary target concept ct. The context of ct (i.e., the set of properties, restrictions, and semantic relations featuring ct) is composed by exploiting the metadata associated with M , that is the set of simple concepts S explaining objects in M , that is,
S = {ci ∈ O | ∀mi ∈ M, mi : ci} where O denotes the ontology. 2) We use H-M ATCH to match ct against the existing domain ontology. Since ct is not featured by a name, H-M ATCH is executed by exploiting only the contextual affinity matching techniques. In fact, contextual matching considers only the contextual affinity between the concepts to be compared without considering their names. The goal of this activity is to exploit the concepts already inside the ontology which are similar to ct, and which are featured by a name, to trigger the concept discovery phase. In other terms, we know that the missing concept is similar to something that is already known in the ontology, and we exploit this similarity between the ontology (what we know) and the external knowledge sources (what is known in other sources) in order to learn new concepts to be added to the ontology. 3) For each concept ci whose affinity with ct is higher than a threshold t, we build a probe query qi. Each probe query qi is matched against external knowledge sources using H-M ATCH. The result is a set of external candidate concepts returned by the matching process whose semantic affinity is higher than or equal to a given matching threshold. Retrieved concepts are then made available to the ontology expert, who can exploit them to define the new concept to be added to the ontology. function conceptDiscovery(Extracted_Metadata){
Vector probeQueries = null; Vector candidateConcepts = null; TemporaryConcept ct = createConcept(Extracted_Metadata); foreach Concept c in Ontology{ if(hmatch.contextualMatch(ct,c) >= threshold){
probeQueries.add(c); } foreach ProbeQuery q in probeQueries{ foreach Concept e in ExternalKnowledgeSource{ if(hmatch.match(q,e) >= threshold){
candidateConcepts.add(e); }
} } } return candidateConcepts;
In order to discuss an example of concept discovery, consider the multimedia resource shown in Fig. 1 and the media objects marked in the figure. Then suppose that the domain ontology contains information about jumping events, as defined in Fig. 4. The concepts graphically represented by a white box (i.e., Sport event, Athletic event, Jumping event and High jump) represent aggregate concepts, while gray boxes (e.g., Foam mat, Pole, Horizontal bar, Athlete) represent simple concepts. Composition dependencies among simple and aggregate concepts are represented by means of dashed arrows. In the right side of the figure, the corresponding Description Logics specification is reported.
The first step of concept discovery is the creation of the temporary concept ct, starting from the incoming resource information, as shown in Fig. 5.
The definition of ct is based on the analysis of the metadata associated with the multimedia resource, that are shown in
P erson u ∃hasP rof ession.Sport SportEquipment SportEquipment SportEquipment SportEquipment Event J umping event u ∃hasP art.J umper u ∃hasP art.Horizontal bar u ∃hasP art.F oam mat ∃hasP art.J umper u ∃hasP art.Horizontal bar u ∃hasP art.F oam mat u ∃hasP art.P ole
Based on the matching concepts definitions retrieved during
the concept discovery phase, on the incoming temporary
concept ct and on the domain ontology, the ontology expert
begins the concept definition activity for ontology enrichment.
A new aggregate concept is defined using the simple concepts
available in the domain ontology. If some simple concept is
needed which is not yet described in the ontology, it is also
defined. In the example, the ontology expert chooses to add the
Pole Vault concept, on the basis of the information provided
by the two web sources and on the fact that Pole Vault is
retrieved in both the external sources. Concept definition is a
manual activity, in that the ontology expert must specify all the
properties and restrictions of the new concept. In our approach,
the amount of manual activity is reduced in that the ontology
expert works directly on the concepts specifications retrieved
during the discovery activity, by properly integrating/merging
those considered more relevant. Once the new concept is
defined, the ontology expert inserts it into the ontology, which
means that the appropriate location in the ontology hierarchy
has to be found. To this end, the ontology expert may use
H-M ATCH to locate a possible insertion point for the new
concept and a reasoner to validate the new resulting version
of the ontology [
Ontology evolution research work has mainly focused on
the problem of evaluating the impact of requirement changes
on the ontology contents [
A six-phase evolution methodology has been implemented
within the KAON [
With respect to the state of the art on ontology evolution, the original contribution of the presented approach is an evolution methodology tailored to provide a semi-automated support to the new concept detection and definition activities, which are conventionally the most human-intensive activities. In particular, we address the problem of semi-automatically discovering the new concepts required in the ontology to describe new multimedia resources by exploiting ontology matching techniques.
In this paper, we have presented a methodology for
supporting the ontology enrichment activity in the context of
multimedia ontology evolution. This problem and its peculiar
requirements, is one of the main research issues addressed
in the BOEMIE [
This work is founded by the BOEMIE Project, FP6-027538 - 6th EU Framework Programme. The authors would like to thank the partners of the BOEMIE project for providing the pictures for the examples.