Multimedia information system design has been recently influenced by state-of-the-art technologies such as those provided by semantic web. This paper introduces an information search and retrieval methodology that employs semantic web technologies for its data representation and reasoning tasks. Creating a meaningful multimedia presentation is an attempt to answer a user's query using a knowledge representation structure. In this case, the system instead of providing a list of results, as happens in many typical information search and retrieval systems, collects a list of selected items based on the relevancy to the queried topic and also meaningful relationships between data objects. The collected information is used for the construction of an automated presentation to demonstrate the results to the user. The paper concentrates in particular on the content selection, narration structuring, and presentation design and generation processes. The implementation of an automatic presentation generation facility in an integrated system, called MANA, is described through the paper. MANA is designed to generate adaptive and automatic presentations based on users' perspectives and preferences on the queried topic.
Current advances in Web technologies and information systems have led to the
provision of rich media descriptions and enhanced reasoning techniques to interpret
the contents and meaningful relationships of multimedia data. Although current
advances of the Web and information presentation systems allow having rich media
enabled information presentation in the form of hypermedia, most of the current
information search and retrieval systems only demonstrate a list of the results to the
user. Among the various research works in the information search and retrieval area, a
part is focused on developing efficient methods to provide more relevant information
and enhanced ranking mechanisms which are implemented in the algorithms such as
PageRank [
Initial work in this area was carried out by introducing a standard reference model
for intelligent multimedia presentation systems (SRM-IMMPS) [
This paper focuses on multimedia data annotation and multi levels of knowledge representation in its design to address different aspects of an adaptive and automatic multimedia presentation generation process. We describe how semantic support can be applied to multimedia data descriptions and discuss how implicit data hidden in meaningful relationships between the objects is used to extract explicit knowledge to acquire the relevant data. The main goal is in organising the objects in a presentation scenario in response to information queries. 2
The presentation generation process attempts to develop a narrative structure to illustrate the relevant information regarding a main topic. The main goal is answering the user’s query with a hypermedia presentation instead of providing a listing of results. The relationships between the participant objects of a presentation have to be identified and organised in order to represent a smooth and meaningful narrative to describe the retrieved results to the user. An automated presentation generation system facilitates discovery of knowledge from information resources, and presents this knowledge to the users, i.e. obtaining a knowledge-driven presentation tailored to the user’s query and preferences. The proposed system enables searching, accessing, and presenting the information based on two main approaches: 1) information discovery to facilitate processing of enriched multimedia data. The information discovery is the data collection process. It refers to the content descriptions to address the relevant multimedia data to a query topic 2) information presentation to allow the user to view the relevant information through a presentation construction. The presentation structure is intended to be constructed in a flexible and adaptable form with respect to the user’s browsing environment and device, and her/his contextual preferences on the queried topic.
We have developed a specific document annotation and description structure for multimedia data based on semantic web information representation structures. The semantic annotation and description model is incorporated with higher-level domain and design knowledge which allow reasoning methods to interpret the contextual information and meaningful relationships for the multimedia data. An automatic presentation generation system is then developed with respect to the data annotation and specification model. The development of the presentation generation in the MANA system consists of the following steps: - implementation of the data annotation model and organisation of information structures based on content and media specific attributes of multimedia objects and meaningful relationships between the objects. - adaptation of specific search and categorisation methods for content selection and collection purposes. - development of intelligent information filters responsible for the discovery of meaningful relationships and addressing the relevant data based on the collected contents. - construction of narration structures based on the collected data and user preferences to view the results. - presentation generation to construct a structure based on data objects, and their relationships, and to deliver the results in a suitable format to the user.
All of these cooperatively provide an integrated architecture to retrieve information and obtain knowledge from the annotated multimedia objects to generate a presentation to answer information queries. 3
The system utilises an annotation structure to describe the multimedia data with
specific reference to those aspects which are necessary to facilitate the presentation
generation process. The descriptions should provide two main views to the objects:
the content-specific data (e.g. what, where, who, is described in the object), and
technical and media-dependent descriptions (e.g. size, width, height, etc). The
implementation of this structure includes a query and retrieval mechanism which
facilitates the collection of suitable objects from the repository for the presentation
generation purpose. The system needs to access the annotations and to process the
specifications in order to select the appropriate data items based on the submitted
query. Although there are tools to retrieve and process the annotations, but when it
comes to interpreting the content and extracting useful information, the capabilities of
current software are still very limited. An alternative approach is to represent the data
in a form that is more machine-accessible and interpretable. The semantic web
technologies [
In order to create an annotation and specification structure capable of expressing different aspects of multimedia data, the following requirements have been considered.
- including content-specific features and also media dependent attributes to
satisfy both the content selection and technical presentation generation
requirements with regards to common annotation structures and vocabularies
such as MPEG-7 [
While the specified requirements describe the main aspects of a data model scheme, a proposed model that satisfies these requirements also has to remain adequately simple and flexible to be applicable. The model also needs to remain sufficiently expressive for the majority of multimedia data types. 4
We define an annotation and description model in which the structure of a multimedia
data representation consists of different distinct specifications described through the
schemas shown in a UML diagram in Fig. 1.
The annotation structure consists of content, semantic, and media schemas. The
content schema describes the multimedia contents and explains their conceptual
notions. The schema involves entities such as objects, abstract concepts and
relationships, topic, and application domain related attributes. The media schema
describes multimedia data from its media and type dependent viewpoint. This schema
involves entities such as size, type, length, and other temporal and spatial features of
multimedia data. The semantic schema defines a scheme that is used to describe the
events, theme, contextual and internal relations of the multimedia data. We have
chosen fine arts as the application domain and in particular painting as a sample to
choose data sets and to describe our framework. For our sample collection, we use
multimedia data taken from the Getty Museum [
We realised that some of the vocabularies such as Dublin Core are not specifically
designed to describe multimedia data. Consequently, using general vocabularies such
Dublin Core for metadata description would not be an efficient and sufficient
solution. For example there are special spatio-temporal considerations for the media
items which are not addressed in the Dublin Core specifications. There are also some
other standards like MPEG-7 which are more specifically designed for multimedia
data description. MPEG-7 uses XML based descriptors to provide content
descriptions for multimedia data. The basic structure of MPEG-7 to describe the
multimedia content follows a hierarchical structure. The content descriptors could be
represented as a graph in which the description schemas are linked together through
relationships between different content specifications [
The content extraction functions and methods are not considered in the model. The content extraction functions are beyond the scope of this research. We assume that the content-specific and technical information are available to the archivist and the system has access to multimedia data through a local repository (or Web links). The following describes each aspect of the proposed annotation model. A simplified structure of the represented features in the model is also shown in Fig. 2. In Fig. 2 “dc”,”mpeg-7”, and “mana” respectively refer to “Dublin Core”, “mpeg-7”, and “mana” namespaces1. The “mana” namespaces is developed for the purpose of this research, and “Dublin Core”, “mpeg-7” namespaces are adapted from the existing standards. 1 The description of namespaces and OWL documentation of the data model is available at: http://csit.nottingham.edu.my/~bpayam/mana/dm/ 4.1
Content-specific information refers to explicit and implicit data obtained from the extracted features and meta-data descriptions. The content of a multimedia object specifies its own intrinsic features. This can be described as explicit data like author, title, keywords, subject, etc. Additionally, the implicit data such as action, scene, etc. could be specified through this schema. The content specific aspects are associated with the ontology concepts. This provides a common vocabulary to annotate the multimedia data, and also to describe meaningful relationships between the data objects in the context of the discourse domain. 4.2
The semantics of multimedia data is provided independently of its contentspecification structure. Both, content and semantic information, however, are needed for the complete contextual description of a media item. The semantic description and the content description illustrate the perceptual structure of the media items. The semantics are associated to the concepts in the domain ontology, which represents the expert’s knowledge of the discourse domain. The semantics represents the contextual subject, and predetermined relationships between the objects (direct associations). The features of this schema are also used for determining the relevancy of objects to the user’s contextual perspectives.
The media schema describes technical aspects of multimedia data. Media information for an object specifies the media dependent characteristics of the object in which the specified information is taken from the binary representation of the multimedia object. Each multimedia object has an identifier, a resource link and a set of media features (such as, type, size, duration, etc.) that is dependent on the binary file, or stream representing the media item. An atomic media scheme instance of a specific multimedia object represents spatial and temporal characteristics of the media item. These features lead to define the presentation layout. The media schema in addition to describing the main temporal and spatial features of the object can also describe different versions of a multimedia element such as a part of a video stream, or a part of an image file, etc. 4.4
The model inherits properties from different description standards and integrates them
in a unified structure. The “mana:depicts” property is used for associating media
objects to the domain concepts. This facilitates the description of the multimedia
objects and their relationships according to the domain ontology concepts. Deploying
the proposed model and using ontology-based annotation will provide a unified and
coherent annotation mechanism and overcome the problem of using vast and
inconsistent terminology to describe the multimedia data. Fig. 3 illustrates annotation
attributes of an instance multimedia object according to the schemas specified in the
data model. The annotations in the system are specified based on OWL-DL [
In the previous section we discussed multimedia data representation requirements and
proposed a multimedia data annotation model. The proposed model indicates the
explanatory attributes for multimedia data in order to be used in an automated
presentation generation system. The model alone, however, is effective if it is not
incorporated with other aspects of the system in order to create an adaptive and
automated multimedia presentation. In other words, multimedia data is required to be
annotated and described through the data model guidelines, but in addition to such
basic requirements, the authoring units need to access and interpret this data to
supports other aspects of the presentation generation process, i.e., data collection,
organisation of the collected data, narration construction, template design/selection,
and transformation. A brief but interesting overview of the current efforts that has
been carried out in meaningful multimedia presentation generation is provided by
Hardman and Ossenbruggen in [
DISC [
In the same context, Rutledge et al [
Aroyo et al [
The HERA system [
Little et al [
Weitzman and Wittenburg [
The presentation structure could be specified by predefined templates which are
filled-in with media items during the presentation generation process. In this case all
the spatial, temporal and navigational structures are fixed and predetermined. The
user selects a presentation theme, and then the content specification unit proceeds to
collect data items (based on a query or a selected topic). The selected objects are
subsequently used to fill-in the predefined template. An enhanced presentation
structure includes flexible spatial, temporal specifications. In this context, the design
rules are taken into account during the presentation structuring process to define the
dynamic template for the presentation. The presentation design rules are translated to
the spatial, temporal and navigational constraints during the presentation realisation
process. The rhetorical structure specifies how the relationship between the objects
could be used in the presentation structure. It is similar to what is called the design
ontology in Rutledge et al’s work [
In the discussed works the key idea is in using domain and design knowledge and reasoning in different aspects of automated multimedia presentation generation process. In the next section, we show how the idea of knowledge-driven presentation generation has been adapted to design an integrated automated presentation generation system. 6
We define our architecture based on the proposed data model and reasoning
techniques to process machine-interpretable data representations. The design aims to
define an adaptive presentation generation framework to answer information request
queries. The system architecture is divided into four main layers namely, the resource
layer, discourse layer, aggregation layer and presentation layer, as shown in Fig 4.
The layers are analogous to the standard reference model’s layered architecture [
The resource layer contains the multimedia data and the annotations. The
multimedia data is stored in a repository as binary objects (local resource) or could be
addressed by means of a URI (external resource). The explanatory data represented in
terms of the data annotation model specifications and stored in an additional
repository. We use Sesame [
In order to collect the relevant information of a specific query topic there are two possibilities: either the media items are selected based on the specified metadata and keyword-based search (i.e. direct search), or the selection process addresses the items based on their meaningful relationships to the query topic using an inference process (i.e. indirect search). The inference process results in the extraction of explicit knowledge from implicit information represented through the meaningful relationships between the data objects in the knowledge-base3. For example, if the query term is “Rembrandt”, Rembrandt painted a scene that is called “The Mill”. The data items which describe “The Mill” (i.e. image and documents related to this painting in terms of locale, style, etc) are relevant to the query topic to some extent and could be considered as candidate objects to be included in the presentation structure. The data selection process is a recursive progression and the media items would be selected through multiple selection steps based on their semantic association to the query topic. The inferring and content collection processes are both implemented in the discourse layer.
We have implemented a semantic association ranking mechanism to evaluate the
complex relationships between the entities in the knowledge-base. In particular, to
develop an automatic presentation about a topic, the system attempts to select a main
topic from the knowledge-base and then semantic association search mechanism starts
spanning over the topic in the knowledge to find the related data. The ranking
mechanism measure the robustness of the semantic associations between the main
entity and other entities in the knowledge-base. This produces a weighted graph
which is used as the fundamental narrative structure to organise the final presentation.
We are aware that the discovery query and selection of the main topic is also an
essential part of the system. It is necessary that the system is able to find a topic from
the knowledge-base and then process the relations to identify the semantic
associations. The ranking mechanism is described in detail in [
The main components of the presentation narrative are defined in the narration ontology4. These components describe the primitive structure of the presentation. The narration construction process develops a progression structure which describes the query topic and also the related information. The progression structure is represented as a graph. This graph is extracted from the main RDF/XML graph (i.e. knowledgebase) based on the narrative components, and selecting the main entity and extending this main topic by analysing the semantic associations. The graph structure describes the related entities to the queried topic and the ranking weights express the degree of relevancy of each related entity to the user’s query. The relevant data is then organised based on the precedence of the events that are defined in a presentation theme. The definition of the presentation aspects and the processing of progression graph are implemented in the presentation layer. 3 The OWL description of the domain ontology is available at: http://csit.nottingham.edu.my /~bpayam/mana/mana-kb.owl 4 The OWL description of the narration ontology is available at: http://csit.nottingham.edu.my /~bpayam/mana/narrationOnto.owl
The main presentation aspects that need to be defined in the construction are
defined as: temporal layout, spatial layout, styles and anchor links. The layout
definition for the presentations in our architecture is similar to predefined template
specifications in HERA [
a) the table of contents slide b) a sample content slide
In this paper we discuss using the concepts of existing annotation and multimedia
specification standards into a richer document model to represent multimedia data.
We use semantic web knowledge representation and reasoning mechanisms to
provide enhanced specification and interpretation for the multimedia data. The paper
states the requirements and describes the architecture for an automated presentation
generation system. We follow a knowledge-driven methodology to generate
multimedia presentations. The SRM-IMMPS [
The paper describes a knowledge representation method to describe the different aspects of multimedia data. The proposed system employs an inference engine which is responsible to process the attributes and relationships of multimedia data that is represented in the knowledge-base. A ranking mechanism is employed to evaluate the semantic associations between the objects. The system collects the relevant data to a query topic by referring to the knowledge-base and meaningful relationships between the objects. The relevant data is then ranked according to the semantic associations and user selected preferences. The results of the ranking mechanism incorporating with the other components of the system provide an integrated architecture to generate automated multimedia presentations. While we feel that the components designed in the system are functioning adequately, they do not necessarily present the most comprehensible architecture for an automated presentation authoring system. Questions can be asked on how the system will perform for a huge set of data when the result of the reasoning process and semantic associations search may retrieve an enormous number of candidate objects. Another important issue is how the individual phases in the presentation authoring can be improved to deal with external data (with unknown attributes) and how narration structuring can be enhanced to generated more improved presentation constructions.
The work reported in this paper is, however, only a part of a complete environment for the creation, storage, annotation, construction, manipulation, transmission and play-back of a hypermedia presentation. Additional requirements are approaches to manipulating documents compatible with the proposed document model, and methodological approaches to process and interpret the external resources (i.e. the Web data) and analysing them to define the sets of annotated documents. The challenge is to guarantee that these methodologies will cooperate with the other developed parts of the system to create more meaningful and adaptive presentations to answer the user’s information request queries. 29. Karvounarakis G., Alexaki S., Christophides V., Plexousakis D., and Scholl M. (2002).
RQL: a declarative query language for RDF. In Proceedings of the 11th international conference on World Wide Web table of contents, pages 592 - 603. 30. The Extensible Stylesheet Language Family (XSL) (1999). Available at: <http://www.w3.
org/Style/XSL/> 31. XSL Transformations (XSLT) (1999). Version 1.0, W3C Recommendation 16 November 1999, < http://www.w3.org/TR/xslt>