- The management and exchange of multimedia data is a challenging area of research due to the variety of formats, standards and the many interesting intended applications. Semantic web technologies are very promising to enable interoperability and integration of media. Many research groups are active in finding and proposing interesting solutions or standards. Within the MUSCLE NoE research is focusing on standards, technologies and techniques for integrating, exchanging and enhancing the use of multimedia within a variety of research areas. At CNR ISTI, we are developing an infrastructure for MultiMedia Metadata Management (4M) to support the integration of media from different sources. This infrastructure enables the collection, analysis and integration of media for semantic annotation, search and retrieval. In this paper we discuss the independent units that are used within the infrastructure and the semantic web technologies that are being used to support them.
Tdue to the availability of off-the-shelf, modern digital HE production of multimedia data is rapidly increasing devices that can be used by even inexperienced users. It is likely that this volume of information will only increase in the future.
Multimedia management on the Web is a hot topic and
many research teams, projects and working groups are active
in this area. To mention only few within the European
framework, see for example W3C [
Facilitating the exchange of documents within the signals and imaging domain is an interesting and challenging problem due to the variety of content, formats, modes and standards used. The challenge is to provide an infrastructure that enables disparate groups to integrate, combine and disseminate research data. The achievement of this goal requires the use of standards and the development of tools to assist in the extraction and conversion of multimedia metadata.
Thus, our activity is mostly concerned with the setting up of a methodology for the NoE to develop, maintain, and facilitate the exchange of multimedia metadata and data sets. To this purpose, we aim to provide an integrated metadata environment to support different metadata standards and tools for browsing, search, media transformation and dissemination.
In this paper we present an infrastructure for the integration of multimedia metadata and their management that we have proposed within the MUSCLE NoE.
While designing this infrastructure we have considered its use in two main contexts. These applications cover a range of requirements in both personal and professional management of multimedia information: a. The management of personal multimedia collections of data (e.g., photos, videos, music etc) that includes the archiving and the retrieval of specific items under particular semantic conditions (e.g., photos showing smiling persons, etc.); b. The management of professional multimedia data within a network to share multimedia resources and related semantic information, where ownership and authorization rights should be taken into account.
Therefore, the following capabilities should be provided: • to store, organize and retrieve distributed multimedia resources; • to manage algorithms for information processing; • to add semantic annotations; • to access, protect and/or share information.
Our proposed infrastructure has been designed taking into account the use of (i) Semantic Web technology; (ii) multimedia metadata standards; (iii) existing tools, (iv) opensource software. In particular, we propose an infrastructure composed of five main units: an MPEG-7 feature extraction and processing unit, an XML database management unit, an algorithm unit, a multimedia semantic annotation unit and an integration unit. In Section 3, the characteristics of the infrastructure are presented and each units is discussed with respect to its purpose and the issues it raises. In Section 4, we discuss the use of ontologies and their integration to facilitate interoperability. Section 5 presents possible future work and conclusions.
The multi-dimensional nature of multimedia metadata and
the challenges this presents when integrating media,
particularly in a web-based system, is a well-known problem
[
The use of technologies coming from Semantic Web,
promoted by the W3C office, could facilitate the overall
vision of distributed, machine readable metadata on Internet.
To enable this scenario, standardized frameworks have been
developed to express semantic relationships between
resources (RDF [
Multimedia on the Semantic Web is a topic of some interest
with the chartering of a W3C Incubator Group [
Related work is also being conducted by Dasiopoulou et al.
[
III. CHARACTERISTICS OF THE INFRASTRUCTURE
Fig. 1 illustrates the 4M infrastructure at a high level. The infrastructure consists of five units: an MPEG-7 feature extraction and processing unit (M), an XML database management unit (X), an algorithms ontology unit (O), a multimedia semantic annotation unit (A) and an integration unit (I).
In particular:
• Unit “M” is devoted to MPEG-7 features extraction
and processing from multimedia objects;
• Unit “X”, based on an XML [
… </mpeg-7>
M DB X
O A I
The implementation of the proposed 4M infrastructure is being conducted in parallel on the different units. Each unit is composed of multiple sub-components (tools, inferencing engines, ontologies, etc.) This section describes the intended purpose and challenges of each of these units.
A. MPEG-7 Features Extraction and Processing Unit MPEG-7 is the most mature and widely recognized standard for multimedia description. Furthermore its format, that is XML, facilitates interoperability with other metadata standards. Our need for using different multimedia objects drives us to adopt a system able to extract features from multimedia objects with a high level of interoperability.
We begun by collecting information on metadata schemas and frameworks standards and considered MPEG-7: among programs allowing MPEG-7 features extraction, few are completely open-source and fewer are able to extract features from different kinds of multimedia objects. Thus we decided to build an integrated system able to extract MPEG-7 features from audio, video, images and text by combining and extending existing open-source programs.
Currently, we are able to extract almost all MPEG-7 features from audio, color and texture from still-images, while for video we are still investigating a solution.
A tool has been implemented able to extract all features together from a multimedia object, building the XML files of MPEG-7 descriptors ready to be used to populate the XML Database.
An example of MPEG-7 XML feature extraction from a still image is shown in Fig. 2.
As it is nowadays widely recognized, MPEG-7, among its advantages related to its completeness to represent metadata of image, video and sound and its suitability to be used in connection with Semantic Web technology, still presents important limitations. Thus, we worked towards possible approaches for extending it by adding some semantics, in particular for annotation, (intelligent) retrieval and, possibly, reasoning.
To this aim we looked at existing tools and, in particular, tools to define and manage ontologies, and the integration of existing ones. <?xml version="1.0" encoding="UTF-8" ?> <Mpeg7 xmlns="urn:mpeg:mpeg7:schema:2001" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:mpeg7="urn:mpeg:mpeg7:schema:2001" xsi:schemaLocation="urn:mpeg:mpeg7:schema:2001 Mpeg72001.xsd"> <Description xsi:type="ContentEntityType"> <MultimediaContent xsi:type="ImageType"> <Image> <MediaInformation> <MediaProfile> <MediaFormat> <Content href="MPEG7ContentCS"> <Name>image</Name> </Content> <FileFormat href="urn:mpeg:mepg7:cs:VisualCodingFormatCS:2001:1"> <Name>jpg</Name> </FileFormat> <FileSize>16367</FileSize> <Frame height="400" width="500" /> </MediaFormat> <MediaInstance> <MediaLocator> <MediaUri>pict2.jpg</MediaUri> </MediaLocator> </MediaInstance> </MediaProfile> </MediaInformation> </Image> </MultimediaContent> </Description> . . . <VisualDescriptor xsi:type="DominantColorType"> <SpatialCoherenchy>1</SpatialCoherenchy> <Values> <Percentage>0</Percentage> <ColorValueIndex>6 4 16</ColorValueIndex> <Variance>0 0 0</Variance> </Values> . . . </VisualDescriptor> <VisualDescriptor xsi:type="ScalableColorType"
numOfCoeff="64" numOfBitplanesDiscarded="3"> <Coeff>-143 40 57 27 20 18 16 9 54 31 21 23 -5 -1 </Coeff> </VisualDescriptor> <VisualDescriptor xsi:type="ColorLayoutType"> <YDCCoeff>56</YDCCoeff> <CbDCCoeff>36</CbDCCoeff> <CrDCCoeff>51</CrDCCoeff> <YACCoeff5 xmlns="">28 20 20 12 11</YACCoeff5> <CbACCoeff2 xmlns="">12 15</CbACCoeff2> <CrACCoeff2 xmlns="">1 10</CrACCoeff2> </VisualDescriptor> . . . </Mpeg7>
In order to manage information extracted by the previous unit we have decided to use a XML Database. This way, the internal representation of MPEG-7 descriptors can be directly inserted onto the database and data structures should be extended only to include additional descriptors. Furthermore, our priorities were also to fulfil the following requirements: fine-grained representation, access and update; typed representation and access; structured indexing; Java interface; multi-user access and extensibility.
We examined four possible open-source projects –
Berkeley DB XML [
eXist is an open source, native XML database featuring
efficient, index-based XQuery processing [
We created collections of MPEG-7 XML documents on the base of the multimedia content type. Java classes have been implemented able to query the collections using XQuery language in order to extract low-level features and select multimedia objects by similarity. An interface has been also implemented to search for images in the database (see Integration). Given that an URI (Unique Resource Identifier) is a basic building block for Semantic Web applications, we denote every multimedia object by a unique identifier, named MediaURI, that includes the type of the object and a hash of the object content. Through a MediaURI, any multimedia object is univocally identified and can be accessed in our XML database.
Algorithms for image analysis (e.g., edge detection, noise reduction, segmentation etc.) are difficult to manage, understand and apply, particularly for non-expert users. For instance, a researcher needs to reduce the noise and improve the contrast in a radiology image prior to analysis and interpretation but is unfamiliar with the specific algorithms that could apply in this instance. This unit aims to provide user support for the discovery, orchestration and application of media analysis algorithms. This enables users to define, store and retrieve the procedures by which multimedia objects have been produced or processed.
Quantifying and integrating knowledge related to analysis
algorithms for media, particularly describing visual outcomes,
is a challenging problem. Currently there exists a
taxonomy/thesaurus for image analysis algorithms [
Our proposed solution is to use the algorithm ontology to record and describe available algorithms for application to image analysis. This ontology can then be used to interactively build sequences of algorithms to achieve particular user outcomes or goals in accordance with the user’s preferences. In addition, the record of processes applied to the source image can be used to define the history and provenance of data.
An example of problem that could be addressed by the algorithm ontology could be the suggestion of possible clinical descriptors (e.g.: pneumothorax) given a chest x-ray.
An hypothesis of solution could be 1) Get a digital chest x-ray of patient P (image A). 2) Apply on image A a digital filter to improve the signalto-noise ratio (image B). 3) Apply on image B a region detection algorithm. This algorithm segments image B according to a partition of homogeneous regions (image C). 4) Apply on image C an algorithm that 'sorts' according to a given criterion the regions by their geometrical and densitometric properties (from largest to smallest, from darkest to clearest, etc.) (array D). 5) Apply on array D an algorithm that searching on a database of clinical descriptors detects the one that best fits the similarity criterion (result E).
However, we should consider the following aspects: step 2) Which digital filter should be applied on image A? We can consider different kinds of filters (Fourier, Wiener, Smoothing, etc. ) each one having different input-output formats and giving slightly different results. step 3) Which segmentation algorithm should be used? We can consider different algorithms (clustering, histogram, homogeneity criterion, etc.). step 4) How can we define geometrical and densitometric properties of the regions? There are several possibilities depending on the considered mathematical models for describing closed curves (regions) and the grey level distribution inside each region (histogram, Gaussian-like, etc.). step 5) How can we define similarity between patterns? There are multiple approaches that can be applied (vector distance, probability, etc.).
Each step could be influenced by the previous ones. Finally, there are two types or levels of interoperability to be considered: 1) low-level interoperability, concerning data formats and algorithms, their transition or selection aspects among the different steps and consequently the possible related ontologies (algorithm ontology, media ontology, etc.); 2) high-level interoperability, concerning the semantics at the base of the domain problem, that is how similar problems (segment this image; improve image quality) can be faced or even solved using codified 'experience' extracted from well-known case studies ?
This unit addresses the issue of (semi-)automatic semantic annotation of multimedia. It aims to exploit the standardized media analysis data produced by the MPEG-7 Feature Extraction and Processing Unit (M) and integrate technologies such as semantic inferencing rules and machine-learning approaches to associate domain terms with media objects. Semantic annotations are highly valuable but generally expensive to create manually and can be overly subjective. Quality semantic annotation of media can facilitate sophisticated semantic search and retrieval, re-use of media objects and support advanced reasoning applications.
The breach between the automatically extracted, low-level
feature metadata and the difficult-to-generate, high-level,
semantic metadata is often termed the “semantic gap”.
Smeulders et al. define it as “the discrepancy between the
information that one can extract from the visual data, and the
interpretation that the same data has for a user” [
Therefore, previous work by Hunter and Little [
Finally, annotations can also include user-created, naturallanguage, subjective comments relating to media objects or possibly media objects themselves (e.g., an audio commentary) that can be associated with a media segment. Future work in this area will investigate how to record, store and manage other annotation types in conjunction with the 4M infrastructure.
All of the units previously discussed, interact and can be
accessed and managed by an Integration unit which supports
the retrieval and insertion of information through suitable
tools and interfaces. The principal purpose of this unit is to
provide interfaces and controllers between the individual units
and the user. To assist in this we are investigating inference
engines based on OWL and SPARQL [
At present, a web-based interface has been implemented so that a user can select a sound or image from the database, choose a set of features and then extract from the database all the sounds and images which are similar to a given one according to the features themselves. The interface has been realized using Java Server Page forms sending the selected parameters to a Java Servlet. Apache Tomcat has been used as application server.
Overall the independence of the units provides a scalable infrastructure that allows new technologies to be easily integrated.
IV. INTEGRATION AND EXTENSION OF ONTOLOGIES Within the infrastructure described in section III, a number of requirements for structured, formal definitions of concepts can be identified. The best way to approach this is by means of ontology that provides a method for structuring a universe of discourse and the possibility to increase such given knowledge through inference engines and inferred knowledge. The use of ontologies is necessary to support: interoperability of multimedia metadata; advanced reasoning using low-level data (e.g., for pattern detection, semantic annotation, etc.); semantic search and retrieval and integration and application of analysis algorithms. Therefore not only one, but five distinct ontologies are required: • Multimedia ontology – media types, descriptions of low-level features, creation metadata, etc.; • Algorithm ontology (described in section III.C); • Possibly a web-services ontology for the algorithm unit (e.g., OWL-S); • Domain ontologies – recording domain specific terms and concepts (e.g., MeSH, FOAF, etc.); • Upper or core ontologies for integration; We are now working on defining, extending and integrating these ontologies.
A number of projects have used the MPEG-7 standard to
derive a multimedia ontology [
The existing taxonomy lacks the specific, formal details required to integration the algorithms within the 4M infrastructure. For example, detailed definition of the required input formats such as ‘binary’, ‘JPG’, etc. and structured descriptions of the goals or outcomes of applying the algorithm which may include the association of example media. The challenge is to develop methods for quantifying ‘visual’ characteristics to assist users (or agents) in evaluating the usefulness of an algorithm for their particular purpose.
We focused on extending the available technology towards multimedia ontologies to add semantics in order to handle applications that require annotation, retrieval, and summarization of multimedia documents. Such an extension is being done in line with the Semantic Web technology, so that integration and interoperability with other existing applications and tools can be provided.
Research communities working on standards are developing
upper ontologies in order to achieve interoperability among
metadata, and integration of multimedia data. An upper level
ontology defines structures and concepts upon which single
domain ontologies could be implemented. An upper ontology
is defined through abstract concepts, which are generic
enough to be exploited by a wide range of domains. And in
fact they are especially suitable for multimedia data
interoperability and integration as demonstrated in [
The use of an upper ontology facilitates the integration of multi-source multimedia information. By combining metadata from various initiatives (Dublin Core, MPEG-7, MPEG-21, CIDOC/CRM, etc.), an upper ontology also provides a basis for semantic interoperability and the development of services based on deductive inferencing. Moreover, providing a common model with a single set of semantic definitions facilitates the efficiency and interoperability of multimedia systems based on the lower-level integrated standards.
Currently work is continuing in parallel on the algorithm and semantic annotation units. In addition there is ongoing development on the functionality and implementation of the integration unit. Possible future extensions to the infrastructure include: distributed data storage and access in the database unit; enhanced functionality for fine-grained, role-based access control and incorporating reasoning capabilities into the integration unit to further improve search and retrieval capabilities.
An initial prototype version of the infrastructure has been developed that integrates the prototype versions of the MPEG-7 feature extraction and database units. This prototype demonstrates some of the technical challenges faced in integrating multimedia metadata.
Overall, the architecture proposed here enables media to be combined and managed. In addition valuable semantic services can be supported, such as semantic search and retrieval, algorithm discovery and application and semantic annotation.
valuable contribution. This work has been partially supported by EU MUSCLE Network of Excellence.