=Paper=
{{Paper
|id=Vol-233/paper-18
|storemode=property
|title=Connecting Foundational Ontologies with MPEG-7 Ontologies for Multimodal QA
|pdfUrl=https://ceur-ws.org/Vol-233/p37.pdf
|volume=Vol-233
|dblpUrl=https://dblp.org/rec/conf/samt/RomanelliSR06
}}
==Connecting Foundational Ontologies with MPEG-7 Ontologies for Multimodal QA==
https://ceur-ws.org/Vol-233/p37.pdf
Connecting Foundational Ontologies with MPEG-7
Ontologies for Multimodal QA
Massimo Romanelli, Daniel Sonntag, and Norbert Reithinger
Abstract— In the S MART W EB project [1] we aim at developing SWI NT O (SmartWeb Integrated Ontology), is based on a up-
a context-aware, mobile, and multimodal interface to the Seman- per model ontology realized by merging well chosen concepts
tic Web. In order to reach this goal we provide a integrated from two established foundational ontologies, DOLCE [7]
ontological framework offering coverage for deep semantic con-
tent, including ontological representation of multimedia based and SUMO [8], into the S MART W EB foundational ontology
on the MPEG-7 standard1 . A discourse ontology covers concepts S MART SUMO [9]. The domain-specific knowledge like sport-
for multimodal interaction by means of an extension of the W3C event, navigation, or webcam is defined in dedicated ontolo-
standard EMMA2 . For realizing multimodal/multimedia dialog gies modeled as sub-ontologies of S MART SUMO. Semantic
applications, we link the deep semantic level with the media- integration takes place for heterogeneous information sources:
specific semantic level to operationalize multimedia information
in the system. Through the link between multimedia represen- extraction results from semi-structured data such as tabular
tation and the semantics of specific domains we approach the structures which are stored in an ontological knowledge base
Semantic Gap. [10], and hand-annotated multimedia instances such as images
Index Terms— Multimedia systems, Knowledge representation, of football teams. In addition, Semantic Web Services deliver
Multimodal ontologies, ISO standards. MPEG-7 annotated city maps with points of interest.
I. I NTRODUCTION
W ORKING with multimodal, multimedia dialog appli-
cations with question answering (QA) functionality
assumes the presence of a knowledge model that ensures ap-
propriate representation of the different levels of descriptions.
Ontologies provide instruments for the realization of a well
modeled knowledge base with specific concepts for different
domains. For related work, see e.g. [2]–[5].
Within the scope of the S MART W EB project3 we real-
ized a multi-domain ontology where a media ontology based
on MPEG-7 supports meta-data descriptions for multimedia
audio-visual content; a discourse ontology based on the W3C Fig. 1. S MART W EB’s Ontological Framework for Multimodal QA
standard EMMA covers multimodal annotation. In our ap-
proach we assign conceptual ontological labels according to
the ontological framework (figure 1) to either complete multi- II. T HE D ISC O NTO AND S MARTMEDIA O NTOLOGIES
media documents, or entities identified therein. We employ
The SWI NT O supplements QA specific knowledge in a
an abstract foundational ontology as a means to facilitate
discourse ontology (D ISC O NTO) and represents multimodal
domain ontology integration (combined integrity, modeling
information in a media ontology (S MARTMEDIA). The D IS -
consistency, and interoperability between the domain on-
C O NTO provides concepts for dialogical interaction with the
tologies). The ontological infrastructure of S MART W EB, the
user and with the Semantic Web sub-system. The D ISC O NTO
This research was funded by the German Federal Ministry for Education models multimodal dialog management using SWEMMA,
and Research under grant number 01IMD01A. the S MART W EB extention of EMMA, dialog acts, lexical
M. Romanelli, D. Sonntag and N. Reithinger are with DFKI GmbH – rules for syntactic-semantic mapping, HCI concepts (pattern
German Research Center for Artificial Intelligence, Stuhlsatzenhausweg 3,
d-66123 Saarbrücken, Germany {romanell,sonntag,bert}@dfki.de language for interaction design), and semantic question/answer
1 http://www.chiariglione.org/mpeg/standards/mpeg-7/mpeg-7.htm types. Concepts for QA functionality are realized with the
2 http://www.w3.org/TR/emma/
discourse:Query concept specifying emma:interpretation. It
3 S MART W EB aims to realize a mobile and multimodal interface to Seman-
models the user query to the system in form of a partially
tic Web Services and ontological knowledge bases [6]. The project moves
through three scenarios: handheld, car, and motorbike. In the handheld sce- filled ontology instance. The discourse:Result concept refer-
nario the user is able to pose multimodal closed- and open-domain questions ences information the user is asking for [11]. In order to
using speech and gesture. The system reacts with a concise and short answer efficiently search and browse multimedia content SWI NT O
and the possibility to browse pictures, videos, or additional text information
found on the Web or in Semantic Web sources (http://www.smartweb- specifies a multimedia sub-ontology called S MARTMEDIA.
project.de/). S MARTMEDIA is an ontology for semantic annotations based
on MPEG-7. It offers an extensive set of audio-visual de- IV. C ONCLUSION
scriptions for the semantics of multimedia [12]. Basically, the We presented the connection of our foundational ontology
S MARTMEDIA ontology uses the MPEG-7 multimedia content with an MPEG-7 ontology for multimodal QA in the context
description and multimedia content management (see [13] for of the S MART W EB project. The foundational ontology is
details on description schemes in MPEG-7) and enriches it to based on two upper model ontologies and offers coverage
account for the integration with domain-specific ontologies. for deep semantic ontologies in different domains. To capture
A relevant contribution of MPEG-7 in S MARTMEDIA is the multimedia low-level semantics we adopted an MPEG-7 based
representation of multimedia decomposition in space, time, ontology that we connected to our domain-specific concepts
and frequency as in the case of the general mpeg7:Segment- by means of relations in the top level classes of the SWI NT O
Decomposition concept. In addition we use file format and and S MARTMEDIA. This work enables the system the use of
coding parameters (mpeg7:MediaFormat, mpeg7:MediaProfile, multimedia in a multimodal context like in the case of mixed
etc.). gesture and speech interpretation, where every object that is
visible on the screen must have a comprehensive ontological
representation in order to be identified on the discourse level.
ACKNOWLEDGMENT
We would like to thank our partners in SmartWeb. The
responsibility for this article lies with the authors.
R EFERENCES
[1] Wahlster, W.: Smartweb: Mobile applications of the semantic web. In:
Fig. 2. The SWI NT O - D ISC O NTO - S MARTMEDIA Connection P. Dadam and M. Reichert, editors, GI Jahrestagung 2004, Springer,
2004.
[2] Reyle, U., Saric, J.: Ontology Driven Information Extraction, Proceed-
ings of the 19th Twente Workshop on Language Technology, 2001.
[3] Lopez, V., Motta, E.: Ontology-driven Question Answering in AquaLog
III. C LOSING THE S EMANTIC G AP In Proceedings of 9th International Conference on Applications of
Natural Language to Information Systems (NLDB), 2004.
In order to close the Semantic Gap deriving from the [4] Niekrasz, J. and Purver, M.: A multimodal discourse ontology for meet-
different levels of media representations, namely the surface ing understanding. In Bourlard, H. and Bengio, S., editors, Proceedings
of MLMI’05. LNCS.
level referring to the properties of realized media as in the [5] Nirenburg, S., Raskin, V.: Ontological Semantics, MIT Press, 2004.
S MARTMEDIA, and the deep semantic representation of these [6] Reithinger, N., Bergweiler, S.,Engel, R., Herzog, G., Pfleger, N., Ro-
objects, the smartmedia:aboutDomainInstance property with manelli, M., and Sonntag, D.: A Look Under the Hood - Design and
Development of the First SmartWeb System Demonstrator. In: Proc.
range smartdolce:entity has been added to the top level class ICMI 2005, Trento, 2005.
smartmedia:ContentOrSegment (see fig. 2). In this way the [7] Gangemi, A., Guarino, N., Masolo, C., Oltramari, A., Schcneider, L.:
link to the upper model ontology is inherited to all segments of Sweetening Ontologies with DOLCE. In Proc. of the 13th International
Conference on Knowledge Engineering and Knowledge Management
a media instance decomposition, so that we can guarantee for (EKAW02), volume 2473 of Lecture Notes in Computer Science,
deep semantic representation of the S MARTMEDIA instances Sigünza, Spain, 2002.
referencing the specific media object, or the segments making [8] Niles, I., Pease, A.: Towards a Standard Upper Ontology. In Proc. of
the 2nd International Conference on Formal Ontology in Information
up the decomposition. Through the discourse:hasMedia prop- Systems (FOIS-2001), C. Welty and B. Smith, Ogunquit, Maine, 2001.
erty with range smartmedia:ContentOrSegment located in the [9] Cimiano, P., Eberhart, A., Hitzler, P., Oberle, D., Staab, S., Studer,
smartdolce:entity top level class and inherited to each concept S.: The smartweb foundational ontology. Technical report, Institute for
Applied Informatics and Formal Description Methods (AIFB) University
in the ontology, we realize a pointer back to the S MARTMEDIA of Karlsruhe, SmartWeb Project, Karlsruhe, Germany, 2004.
ontology. [10] Buitelaar, P., Cimiano P., Racioppa S., Siegel, M.: Ontology-based
This type of representation is useful for pointing gesture Information Extraction with SOBA In Proc. of the 5th Conference on
Language Resources and Evaluation (LREC 2006).
interpretation and co-reference resolution. A map which is [11] Sonntag, D., Romanelli, M.: A Multimodal Result Ontology for In-
obtained from the Web Services to be displayed on the screen tegrated Semantic Web Dialogue Applications. In Proc. of the 5th
shows selectable objects (e.g. restaurants, hotels), and the Conference on Language Resources and Evaluation (LREC 2006).
[12] Benitez, A., Rising, H., Jorgensen, C., Leonardi, R., Bugatti, A., Hasida,
map is represented in terms of an mpeg7:StillRegion instance, K., Mehrotra, R., Tekalp, A., Ekin, A., Walker, T.: Semantics of
decomposed into different mpeg7:StillRegion instances for Multimedia in MPEG-7. In Proc. of IEEE International Conference on
each object segment of the image. The MPEG-7 instances are Image Processing (ICIP), 2002.
[13] Hunter, J.: Adding Multimedia to the Semantic Web - Building an
linked to a domain-specific instance, i.e., the deep semantic MPEG-7 Ontology. In Proc. of the Internatioyesnal Semantic Web
description of the picture (in this case the smartsumo:Map) or Working Symposium (SWWS), 2001.
the segment of picture (e. g., navigation:ChineseRestaurant).
In this way the user can refer to the restaurant by touching
on the displayed map. Hence a multimodal fusion component
can directly process the referred navigation:ChineseRestaurant
instance performing linguistic co-reference resolution: What’s
the phone number here?