Currently, ontology development has been mostly directed at the representation of domain knowledge (i.e., classes, relations and instances) and much less at the representation of corresponding text and image features. To allow for cross-media knowledge markup, a richer representation of features is needed. At present, such information is mostly missing or represented only in a very impoverished way. In this paper we propose an RDF/S-based ontology for the integrated representation of domain knowledge and text/image features.
Ontologies define the semantics for a set of objects in the world using a set of
classes, each of which may be identified by a particular symbol (either
linguistic, as image, or otherwise). In this way, ontologies cover all three sides of
the “semiotic triangle” that includes object, referent and symbol, i.e., an object
in the world is defined by its referent and represented by a symbol
Currently, ontology development and the Semantic Web effort in general have
been mostly directed at the referent side of the triangle, and much less at the
symbol side. To allow for cross-media knowledge markup, a richer
representation is needed of these symbols, i.e. of the text and image features for the
object classes that are defined by the ontology. At present, such information is
mostly missing1 or represented only in a very impoverished way, leaving the
semantic information in an ontology without a grounding to the human
cognitive and linguistic domain.
1 According to the collection of ontologies available through OntoSelect
Representation
An ontology describes a knowledge model of a particular domain of discourse at a particular point of time and is shared between two or more actors in the domain. As the ontology defines the agreed semantics of the domain, all relevant content will be marked-up with knowledge according to the ontology. The definition of the ontology in turn depends primarily2 on the content that has already been interpreted. Accordingly, content production and interpretation will drive the adaptation of the ontology infrastructure, and ontology adaptation will drive content interpretation and production. In order to arrive at such a continuous ‘hermeneutic cycle’ of content and knowledge production and interpretation, a rich representation of domain knowledge and content features is needed. Here we propose an integrated approach that organizes content and knowledge in several layers, as displayed below:
informal formal
content features feature associations ontology al al rmrm o o f f in formal informal formal inforfmormianlformal a l
…
The content layer (outermost layer) consists of cross-media data (images, video and/or mixed image and text documents).
The features layer (1st inner layer) consists of extracted features for the data in the content layer. For multilingual data, this ranges from comparatively informal feature vectors gathered by use of statistical methods to formalized descriptions of the content of text documents, typically extracted by use of natural language processing and information extraction methods. For multimedia data, this will be mostly limited to informal features as used in colour histograms and similar. 2 Aside from more generic knowledge of the physical world, time, space, etc. that will be inherited from an upper-level ontology. The feature association layer (2nd inner layer) consists of feature representations occurring in the features layer. While in the features layer features are associated with cross-media data, in the feature association layer the features are associated with classes in the semantic model.
The semantic model layer (central layer) consists of classes, with which the data in the content layer is to be interpreted (i.e., annotated) by use of the extracted and represented features in the features layer and the feature association layer.
This integrated approach allows for cross-lingual, cross-media feature extraction and representation as follows: image2text – For instance, if we know which terms express a class in English, we will be able to build a classifier for the classification of images that occur in the context of English terms for this class. text2image – For instance, if we know which images represent instances for a specific class, we will be able to extract German terms for this class from surrounding German text. text2text – For instance, if we know which terms express a class in English, and we know the context features (i.e. words) for these terms and possible translations for these words into German, we will be able to build a crosslingual classifier for recognition of unseen German terms for this class. image2class or text2class – For instance, if we know which terms express a class in English, and we know the context words for these terms, we will be able to detect a change in the semantic model for this class by monitoring any change in the context words, and similar with image feature models. 3
Towards Ontology-Based Feature Representation
The integrated ontology-based feature representation we propose is based on
ongoing work in the context of the SmartWeb project on mobile Semantic
Web access for intelligent information services in the football domain
(http://www.smartweb-project.de/). To represent terminology for concepts in
different languages we initiated an extension of RDF-based domain
knowledge representation with the meta-class ClassWithFeats.
Although there is some overlap with the SKOS
The conceptual problem we see with SKOS for the use in our scenario is that
it mixes linguistic and semantic knowledge. SKOS uses skos:broader and
skos:narrower to express “semantic” relations without clearly stating the
semantics of these relations intentionally, and defines the sub-properties
skos:broaderGeneric and skos:narrowerGeneric to have class
subsumption semantics (i.e., they inherit the rdfs:subClassOf semantics from
RDFS). We clearly keep the linguistic and semantic, ontology-based
knowledge representations apart6: the ontology is represented using the semantic
relations defined in RDFS or OWL (Full)7
The proposed feature representation is currently used in the SmartWeb ontology on sport events and related issues. Figure 2 shows the ontology with example classes and associated linguistic and image features: the ontology contains the class o:FootballPlayer with subclasses o:Defender and o:Midfielder. All these classes are instances of the meta-class feat:ClassWithFeats which allows them to use the feature-association properties feat:lingFeat and feat:imgFeat. The figure shows the linguistic features of German terms for the class o:Defender (Abwehrspieler) and o:Midfielder (Mittelfeldspieler). Note that the decomposition of Abwehrspieler contains Spieler, therefore implicitly relating the two classes in the ontology via linguistic features. Furthermore, the figure shows an image feature representation associated with the class o:Midfielder, stating that an instance of this class has a ‘human’ shape and certain color and texture features.
drdf:type en URI egproperty ...
L rdfs:Class
rdfs:subClassOf feat:ClassWithFeats feat:ClassWithFeats o:FootballPlayer
rdfs: feat:ClassWithFeats subClassOf feat:ClassWithFeats
o:Defender o:Midfielder feat:lingFeat feat:imgFeat ... feat:lingFeat lf:LingFeat lf:LingFeat lf:lang “de” lf:lang “de” lf:term “Abwehrspieler” lf:term “Mittelfeldspieler” lf:syntacticDecomp lf:syntacticDecomp lf:Lexeme lf:Lexeme lf:lexeme “Abwehr” lf:lexeme “Spieler” lf:partOfSpeech “noun” lf:partOfSpeech “noun” lf:morphDecomp lf:morphDecomp lf:hasLemma lf:hasLemma lf:Lemma lf:Lemma lf:lemma “Abwehr” lf:lemma “Spieler” meta-classes ...
classes rdfs:Class
if:ImgFeat rdfs:Class lf:LingFeat if:ImgFeat if:color “#111111” if:shape “human” lf:texture “&keypatchSet_223 lf:Lexeme lf:lexeme “Mittelfeld” lf:partOfSpeech “noun” lf:morphDecomp lf:hasLemma lf:Lemma lf:lemma “Mittelfeld” instances This research has been supported in part by the SmartWeb project, which is funded by the German Ministry of Education and Research under grant 01 IMD01 A. We acknowledge input on the representation of image-related features by Yannis Avrithis, Eric Gaussier and Yiannis Kompatsiaris.