Making multimedia assets on the one hand ¯rst-class objects on the Semantic Web, while keeping them on the other hand conforming to existing multimedia standards is a non-trivial task. Most proprietary media asset formats are binary, optimized for streaming or storage. However, the semantics carried by the media assets are not accessible directly. In addition, multimedia description standards lack the expressiveness to gain a semantic understanding of the media assets. There exists an array of requirements both regarding media assets, and the Semantic Web already. Based on a critical review of these requirements we investigate how ontology languages ¯t into the picture. We ¯nally analyse the usefulness of formal accounts to describe spatio-temporal aspects of multimedia assets in a practical context.
Today a huge explosion of content can be experienced on
the Web generated by, and for the home users [
Unsurprisingly there is already a portal called VideoRonk3
trying to combine other portals by providing a MetaSearch
interface, which is quite of an help as one does not want
to search on ten or more di®erent sites. However, what is
missing is the link between the contents of all these sites,
enabling distributed recommendations, cross-linking, etc.
Still, for example a cross-site search on the semantic level
is close to impossible. The most obvious reason is due to
a lack of metadata coming along with all the content. The
power of providing metadata along with content on the Web
can be seen at prospering mashups that not just combine
APIs|provided by parties as Google4| but also trying to
mashup things on a semantic level. This can be observed for
example at Joost [
As pointed out in [
Current XML-based standards [
Automating the handling of metadata for these collections and automating linkage between parts of these collections is hard as the vocabularies to describe them are mostly diverse and do not o®er facilities to attach formal descriptions. A Motivating Scenario. Imagine a person that wants to watch the recent clips similar to the ones of his favourite experimental artist. Tons of clips are potentially distributed on the Web making a search for them time consuming and laborious. Thus a central facility to search for and negotiate content is needed. This facility should allow to formulate a search goal, including the characteristics, the subject matter, a maximum price, and the preferred encoding and ¯le format of the clip. In a next step, all portal o®erings will be scanned in order to retrieve and negotiate content that matches the users' intention. Note that also parts of a video may match his intention, which means that videos need to be ¯ne granular and su±ciently well enough described. In order for this scenario to work, the descriptions of (1) the goal formulation, (2) the description of the media content by all content owners and (3) the negotiation semantics have to be compatible. Three important focal points of these semantic descriptions are: ² Expressivity for high level semantic descriptions of content as typical users are not thinking in terms of colour histograms and spatial / temporal constructs. The characteristics of the media should be described detailed enough. ² The need for rules: To e®ectively identify the part of the content that matches the users' intention, rules are needed to map high level semantic concepts to spatial and temporal segments of the video (eg., because ratings and classi¯cations could only apply to parts of the content, ie., a scene including crime is only suitable for adults) ² Fine grain semantic descriptions as of bandwidth, user e®ort, or cost reason to transfer the whole content is not possible. Thus parts of the content should be described detailed enough.
To reach out, we want to provide answers to the question: Why do we need rich semantic descriptions of media assets on the Web, and (why) is there a need to bundle these descriptions together with the multimedia assets? Simultaneous, we want to provide answers to the questions: How can descriptions be provided? Why are the metadata features of multimedia standards not enough? Consequently, we elaborate on the answer to the question stated in the title of this paper \Why Multimedia Assets Fail to Enter the Semantic Web? " by ¯rst collecting the requirements for the description of multimedia assets (section 2), secondly by analysing the environment (section 3), and thirdly by collecting requirements for multimedia assets on the Semantic Web (section 4). In section 5 we analyse existing ontology languages for their usefulness regarding the requirements and conclude in section 6 with a discussion of the question stated in the title of this paper and give a brief outlook on the open issues.
Requirements for multimedia content descriptions have been
researched in a number of papers [
² Neutral Representation: The ideal multimedia metadata format has a platform and application independent representation, and is both human and machine processable; ² Formal Grounding: Knowledge about media assets must be represented in formal languages, as it must be interpretable by machines to allow for automation.
mat at hand is extensible, e.g., via an extension mechanism
as found in MPEG-7. It should be possible to integrate or
reference existing vocabularies [
Multimedia Characteristics and Linking. The format should re°ect the characteristics of media assets, hence allow linking between data and annotations: ² Description Structures. The format should support description structures at various levels of detail, including a rich set of structural, cardinality, and multimedia data-typing constraints; ² Granularity. The language has to support the de¯nition of the various spatial, temporal, and conceptual relationships between media assets in a commonly agreed-upon format; ² Linking. It has to facilitate a diverse set of linking mechanisms between the annotations and the data being described, including a way to segment temporal media.
Authoring & Consumption. A major drawback of existing metadata approaches is its lacking support for authors in creating annotations along with the lacking bene¯ts of generated annotations.
² Engineering support. Appropriate tools are a prerequisite for uptake of new vocabularies. There is the need for at least authoring and consumption environments making use of the vocabularies to demonstrate their usefulness. ² Deployment. Multimedia Assets need to be exchangeable, and there must be ways to deploy descriptions along with the assets.
descriptions that can be stored, processed, exchanged and queried e®ectively and e±ciently.
MPEG-7. MPEG-7 [
A good starting point for the analysis of our targeted
hosting environment|the Semantic Web|is the Architecture of
the World Wide Web [
Any multimedia metadata format that is after the successful application on the Semantic Web has to be in-line with the above listed characteristics. While some requirements, as formats (e.g. XML) are rather easy to meet, other can pose serious problems regarding the integration into the Semantic Web.
Firstly, addressing the environmental requirements together with an e±cient layering of the semantic descriptions on top 7a portmanteau word; contributor and user of the existing metadata (sub-symbolic level - symbolic level - semantic level) is a necessary prerequisite for multimedia assets to enter the Semantic Web successfully. Secondly, from the requirements gathered in section 2 and the environmental analysis done in section 3 we deduce the following characteristics for multimedia assets on the Semantic Web:
basic building blocks of the Semantic Web. To enable
automatic handling like retrieval, and negotiation of multimedia
assets formality of descriptions is a pre-requisite.
Three di®erent (semantic) levels of multimedia metadata
can be identi¯ed [
Efficient layering and referencing of descriptions. It is necessary to support di®erent levels of meaning attached to multimedia assets, i.e., meaning at the bit-level, traditional metadata and semantic (high-level) information. As there are already widely adopted standards available for the description of multimedia assets, the semantic layer must be e±ciently put upon those traditional description layers and should not aim to replace it. Furthermore semantic descriptions from these traditional layers shall be re-used. As content, parts of content, and traditional and semantic descriptions may be distributed, e±cient referencing mechanisms for multimedia content must be present. Based on recent discussions8 we give a summarisation of possible approaches in the following. The multimedia asset is denoted with A, for the multimedia metadata (M3) format, such as MPEG-7, we write M, the ontology (language) is written as O, and ¯nally an external reference mechanism9 are labelled with R. The linking is depicted with ,!: ² M ,! A. the content is referenced from the M3 format; the ontology layer has to deal with it, separately; ² M ,! O. The M3 format references the ontology layer; ² O ,! M. The ontology layer references the M3 format; ² O ,! A. The ontology layer references the content directly; ² O, M ,!R A. The ontology layer and the M3 format use a common reference mechanism to link to the content. However, it has to be noted that there is no standardised way for the layering or the referencing, yet. 8http://lists.w3.org/Archives/Public/ public-xg-mmsem/2007Apr/0002.html 9http://www.annodex.net/TR/URI_fragments.html
in various communities cause interoperability problems when
dealing with multimedia content. To overcome this, an RDF
based semantic layer should be added on top of these
numerous formats to ease their semantic and syntactic integration.
However, there are some open problems regarding the
integration of existing annotation standards and semantic
approaches [
Subjectivity and granularity of descriptions. Opinions and views on the content di®er among users because of their personal background, culture or previous experiences. As many users are potential contributors to descriptions of assets, opinions may di®er. Many of these opinions sometimes do not serve to a unique whole opinion. This is why it should be possible to separately attach these opinions to multimedia assets and keep them separate.
Trust and IPR issues. The Web consists of decentralized
authorities and a huge number of contribusers. As
descriptions of content|especially in the new changing Web 2.0
environment|are subject to vandalism, there need to be
ways to guarantee the validity of the descriptions and to
secure descriptions that are just read-only for a user group.
Popular portals like Flickr or YouTube show that there is no
need to own content in order to annotate it. Furthermore
copyright is critical when dealing with multimedia content.
Functional Descriptions. Sometimes the fact that
metadata is created to support some speci¯c function is forgotten
when summarizing the requirements for a metadata schema.
For the metadata creator it should be clear beforehand for
what purpose the metadata will be used and what bene¯ts
he gains from it [
This in turn also applies to the consumer of the metadata, functional descriptions of what type of information can be inferred from the attached metadata or what type of actions can be performed on the content are essential: this is especially true for information that is obfuscated prior to a possible negotiation phase of the content.
Engineering Support. The presence of metadata is a
prerequisite to make multimedia assets accessible, and
deployable on the Semantic Web, hence to enable their automated
processing. From a developers perspective, there must be
tools and standards enabling an integrated authoring,
testing, and deployment of multimedia assets along with their
associated metadata. In the following the most important
areas of engineering support are listed:
² Edit & Visualise. To aid the engineer in handling
the annotations, editor tools, and IDEs10 are needed.
These may include validator services11, converter or
mapper, and visualisation modules.
² Libraries & Applications. When developing
applications, the availability of APIs is a core requirement.
In special for Semantic Web applications, interface and
mapping issues are of importance [
In this part ontology languages which are thought to be used for the advanced requirements which were identi¯ed in the sections before are introduced. In its core it comprises a a comparison of two families of ontology languages against the requirements postulated in section 4.
The reader is invited to note that not all of the existing
languages have the same expressiveness and not all have the
same inferential capabilities. Further, the underlying
knowledge representation paradigms can di®er (eg., Description
Logics, Logic Programming, etc.). Corcho and Gomez-Perez
[
A number of logical languages have been used for the description of di®erent kinds of knowledge (i.e. ontologies and rules) on the Semantic Web: First Order Logic, Description Logics, Logic Programming and Frame-based Logics. Each of which allow the description of di®erent statements and each imply di®erent complexity results for certain reasoning tasks with these languages.
In this section we want to introduce two of the most
promising ontology language families, ie., the OWL- and the
WSMLfamily of languages. The OWL family of languages is a
standardisation e®ort of the W3C and the WSML family of
languages is an e®ort of the WSMO working group, whereas
WSML is a formal language for the description of ontologies
and Semantic Web Services. Other ontology languages like
10as for example http://www.topbraidcomposer.com/
11http://phoebus.cs.man.ac.uk:9999/OWL/Validator
F-Logic [
The Web Ontology Language (OWL) family was designed in a W3C standardisation process because of the need for an ontology language that can be used to formally describe the meaning of terminology used in Web documents, thus, making it easier for machines to automatically process and integrate information available on the Web. This language should be layered on top of XML and RDF (W3C's Resource Description Framework13) in order to build on XML's ability to de¯ne customized tagging schemes and RDF's approach to representing data.
Currently OWL 1.114 is under development; it extends OWL DL in several ways: the underlying DL now is is SROIQ, which provides increased expressive power with respect to properties and cardinality restrictions. Further, OWL 1.1 has user-de¯ned datatypes and restrictions involving datatype predicates, and a weak form of meta-modelling known as punning.
The usage of rules in combination with DL has been
investigated for some time [
The activities of the WSMO Working group15 have yielded
proposals of new ontology languages, namely WSML
(WSMLCore, WSML-DL, WSML-Flight, WSML-Rule, WSML-Full),
OWL- ("OWL minus") [
Due to well-known limitation of the expressive power of the
Description Logics language family [
In the following a high-level comparison of formal description paradigms for multimedia assets is performed. We chose OWL+RIF on the one side, and WSML/OWL-Flight on the other to achieve a somehow realistic scenario; the result can be found in Table 117: The table indicates for which requirement an ontology language (resp. OWL / WSML) can be utilised to overcome the identi¯ed shortcomings of traditional approaches and thus ful¯ll the requirements stated in 4.
Requirement
Trust & IPR issues Functional Descriptions Engineering Support Datatype Support
OWL + RIF
1.1 ++ + ++ ++ +
WSML-/ OWLFlight ++ + + * + ++
In the following, we elaborate in detail on each of the items in Table 1, and argue therefore our ¯ndings regarding the comparison of OWL 1.1 + RIF vs. WSML/OWL-Flight.
Both OWL and WSML provide a framework for the formal (machine-processable) description of ontologies. An ontology in WSML consists of the elements concept, relation, instance, relationInstance and axiom. The primary elements of an OWL ontology concern classes and their instances, properties, and relationships between these instances. The formality of the descriptions is based on logics that allow machines to reason on the information. Whereas OWL is based on Description Logics, the WSML family members are based on di®erent logic languages (ie. Description Logics, Logic Programming or First Order Logic).
Despite the fact, that OWL is more widely adopted and used we believe that WSML with its layered framework is conceptually superior to OWL. A major di®erence between ontology modeling in WSML and ontology modeling in OWL 17++ . . . good support, + . . . available , - . . . not supported, * . . . supported because of WSML's constructs for the description of Semantic Web Services is that WSML separates conceptual modelling for the nonexpert users, and logical modeling for the expert user as it| unlike OWL|uses an epistemology, which abstracts from the underlying logical language making the surface syntax nicer. Even if an application later requires OWL, one is able to use WSML tools to convert ontologies that reside in popular logic/language fragments automatically into equivalent OWL ontologies. Furthermore the WSML family framework enables one to choose exactly the language with the needed expressiveness to be used, and later allows an easy switch to another family member as a consequence of its common grounding. WSML Rule and WSML Flight also include rule-support. Thus, unlike with OWL, no additional rule language is needed.
An array of existing multimedia metadata (M3) formats have been used for years in diverse application areas. However, when one aims at using these formats (as MPEG-7, ID3, etc.) in the context of the Semantic Web, the options are limited. Hence, to enable an e±cient layering of RDFbased vocabularies on top of existing multimedia metadata, one may use hybrid techniques.
As a result of our works in the media semantics area, we
recently proposed the RDFa-deployed Multimedia Metadata
(ramm.x) speci¯cation [
A di®erent but as well Web compatible approach is described
in [
To adhere to the architecture of the WWW, OWL uses (1) URIs for naming and (2) RDF to provide extensible descriptions. (3) OWL builds on RDF and RDF Schema and adds additional vocabulary for describing properties and classes. (4) The datatype support for OWL is grounded on XML Schema.
WSML has a number of features which allow to integrate it
seamlessly in the Web: (1) WSML uses IRIs18 [
To reach compatiability between WSML and OWL, WSML
has a set of de¯ned translators between OWL and WSML
[
As stated above, when referring to granularity, we understand the support of the de¯nition of various spatial, temporal, and conceptual relationships regarding annotations. In this sense, OWL and WSML meet the minimal requirements, but do not explicitly address this issue. Depending on the granularity, obviously scalability and performance issues come along. In this respect, again, OWL and WSML can be perceived comparable.
In an interdependent, interconnected environment as the
Semantic Web, two important aspects immediately arise: data
provenance and trust [
Both WSML and OWL do not have explicit provisions for handling trust and IPR issues, respectively.
OWL and the WSML's part for the description of ontologies
do not have support for such kind of descriptions.
However, WSML is a language for the speci¯cation of
ontologies and di®erent aspects of Web services. As such it
not only provides means for modeling and description of
ontologies but also functional (service) descriptions, i.e. the
description of a service capability by means of precondition,
assumptions, postconditions and e®ects [
Tool Support for WSML and especially OWL is constantly
growing. However, the amount of tools available for OWL
[
The reader is invited to note that both OWL and WSML
ground their datatype support on XML Schema. In WSML,
XML Schema primitive datatypes, simple types and XML
Schema derived datatypes are supported [
The ¯rst question we kept open is "What are real-world multimedia assets"? Real-world multimedia assets are multimedia objects which can be currently found embedded in HTML pages on the Web, as images, videos, etc. We see three main reasons why media assets fail to enter the Semantic Web: 1. There is a lack of the critical mass of annotated content which is mainly due to the large scale automation of (semantic) visual analysis has not gone that far. This is why the user is the central person in the process in order to provide manual annotations. Motivating user to attach complex annotations to content is not easy to achieve. 2. Current traditional and Web 2.0 based approaches to multimedia annotation are not useful to achieve the goals of the Semantic Web: The most important aspects that the Semantic Web intends to solve are (i) Annotation, (ie., how to associate metadata to a resource), (ii) Information Integration (ie., how to integrate information about resources), and (iii) Inference (ie., reasoning over known facts to unleash hidden facts).
Existing multimedia metadata standards as MPEG-7 can be used to annotate but keep a certain amount of ambiguity amongst these annotations. As it is a standard it allows easy integration based on it (a requirement for that is that everyone adheres to this standard!) but inference is not possible with the information attachable to a MPEG-7 ¯le. The problem with tagging is manifold; there are open issues, such as consistency among tags, reconciliation of tags, and how to associate tags with parts of the tagged content. This huge amount of uncertainty will not allow reliable information integration, nor allow to reason on it. 3. As we argued in this paper, more requirements have to be ful¯lled, which can not be solely solved by traditional or Web 2.0 based approaches and which make more formalized descriptions of content necessary. However, before not being able to attach these directly to the media being described, multimedia assets will not be able to enter the Semantic Web.
Regarding deployment of M3 format on the Semantic Web,
we recently proposed to use ramm.x in the Cultural Heritage
domain [
The research leading to this paper was partially supported by the European Commission under contract FP6-027026, \Knowledge Space of semantic inference for automatic annotation and retrieval of multimedia content - K-Space" and SALERO (contract number FP6-027122).