=Paper=
{{Paper
|id=None
|storemode=property
|title=More than the Sum of its Parts: CONTENTUS – A Semantic Multimodal Search User Interface
|pdfUrl=https://ceur-ws.org/Vol-694/paper6.pdf
|volume=Vol-694
}}
==More than the Sum of its Parts: CONTENTUS – A Semantic Multimodal Search User Interface==
https://ceur-ws.org/Vol-694/paper6.pdf
More than the Sum of its Parts:
CONTENTUS – A Semantic Multimodal Search
User Interface
J. Waitelonis, J. Osterhoff, H. Sack
Hasso-Plattner-Institute
Prof.-Dr.-Helmert-Str. 2-3, 14482 Potsdam,
Germany
{joerg.waitelonis |
johannes.osterhoff | harald.sack}
@hpi.uni-potsdam.de
ABSTRACT the retrieval and search process in every part. New media
This paper presents the semantic search engine CONTEN- analysis and retrieval methods produce metadata, which is
TUS [7] and its user interface, a multimedia retrieval sys- not necessarily textual data anymore, as e.g. color features
tem developed in the context of the German research project of an image. Image and video search is evolving into a more
THESEUS1 . This interface uses content-based suggestions and more high level and fine granular multimedia retrieval.
and a multi-modal presentation of search results to support New metrics of similarity have to be applied to adapt the
semantic search. In addition, the system deploys a combi- search result ranking. New user interfaces are needed to en-
nation of a facetted browsing and breadcrumb-based navi- able query formulation and intelligent visualization of search
gation; a time-line enables time based filtering of the search results. Todays search engines are not only focussed on find-
results and the system suggests related search results accord- ing keywords in documents anymore. Moreover their main
ing to the user’s preferences. Finally, CONTENTUS has goal has shifted to achieve a satisfying search experience for
become more than the sum of its parts. Its unique feature the user. E. g. personalization enables to adapt the search en-
combination facilitates search to become a more efficient gine characteristics to the user’s behavior and personal pref-
and overall more pleasant user experience. erences. Semantic search technology promises more accu-
rate and complete search results. Ontologies and semantic
Author Keywords data repositories, as e.g., Linked Open Data [8] enable to in-
Semantic Search, Multimedia Retrieval, Search Interfaces corporate external resources into the search process to satisfy
the user’s information needs and to enable a new more com-
prehensive search experience. Bringing together multimedia
ACM Classification Keywords
and semantic search applications bears new challenges in vi-
H.5.2 Information Interfaces and Presentation: Miscellaneous
sualizing search results and facilitating navigation. In com-
bination with semantic technologies, searching multimedia
INTRODUCTION on-line in the WWW or in closed repositories, is subject of a
According to ComScore the total U.S. Internet audience en- paradigm shift – the route with the user’s experience is more
gaged in more than 5.2 billion video viewing sessions dur- important than the original goal of fact finding.
ing September 20102 alone and the demand is still growing.
The immense presence of multimedia in the WWW requires While taking all these aspects into consideration, this paper
technologies for managing, organizing, and searching mul- presents a web based graphical user interface (GUI) for the
timedia content in an efficient way. Within WWW search multimodal semantic search engine CONTENTUS. The ob-
engines, the tremendous diversity of multimedia data affects jective of the interface design was not to extend the research
1 on a single aspect of current approaches in semantic mul-
CONTENTUS is an application scenario of THESEUS, supported
by the Federal Ministry of Economics and Technology on the basis timedia retrieval, but to create an interface whose strength
of a decision by the German Bundestag. lays in exploiting synergies by combining state-of-the-art se-
2 mantic and multimedia technologies with interface design
http://www.comscore.com/Press Events/Press Releases/2010/9/
Video Makes it Big on the Small Screen in Europe approaches.
The paper is structured as follows: In section 2 we describe
related work and introduce the relevant technologies and pa-
radigms. Section 3 deals with the realization of the user in-
terface including design aspects. Then, section 4 presents
and discusses different show cases. Finally, section 5 con-
cludes the paper with a short discussion of achieved results
Workshop on Visual Interfaces to the Social and Semantic Web and an outlook on future work.
(VISSW2011), Co-located with ACM IUI 2011, Feb 13, 2011, Palo Alto,
US. Copyright is held by the author/owner(s).
Figure 1. The CONTENTUS processing chain.
DESIGN ASPECTS AND RELATED SYSTEMS proach is based on the semantic information, that was au-
Germany’s 30.000 libraries, museums and archives contain tomatically extracted from historical documents and it pro-
an incredible wealth of knowledge in the form of millions vides one unified interface. The ‘Parallax’ interface5 of the
of books, images, tapes and films. Researchers involved in Freebase project uses a combination of faceted search, fact
CONTENTUS are exploring how these cultural assets can views, time-lines, and geo-maps guiding the users to explore
be made available to as many people as possible and be pre- a wide range of heterogeneous semantic data. But, in con-
served for future generations. The CONTENTUS search en- trast to the CONTENTUS system, the Freebase interface is
gine demonstrates how semantic technologies and interface based on manually and collaboratively composed semantic
design can be deployed in concert to facilitate a better and datasets from the Freebase repository and does not include
more livelier search experience within large virtual collec- automatically extracted information from multimedia con-
tions of cultural heritage domain assets. Fig. 1 illustrates tent. The CONTENTUS user interface is not explicitly de-
the complex processing chain implemented by the CON- signed to enable exploratory search, such as established in
TENTUS framework [10]. After digitizing all types of me- the video search engine yovisto [9]. In yovisto, exploratory
dia assets such as books, newspaper, images, music/audio, search enables to broaden the search scope by expanding the
and videos, complex workflows including media optimiza- search with related information.
tion, content analysis, and semantic metadata generation pro-
vide a highly heterogenous collection of semantically an- Hence, the state-of-the-art systems provide either multimodal
notated data. This comprises also automatically generated or semantic search. The objective of the CONTENTUS user
transcripts from speech, optical character recognition, or the interface is to melt together these paradigms by improving
extraction of semantic entities such as persons, places, and search results and navigation while preserving usability.
events. The main challenge of the user interface is to harness Therefore, the interface provides not only information found
this heterogeneity of multimedia and semantic metadata in a in documents, but also information about documents and re-
consistent, clear, and meaningful way [6]. sources, its comprising entities, and their in-between rela-
tionships. For example, an instance of a person can arise as
Another already existing system trying to meet these require- an author of a document, but also as a subject a document
ments is the the MultimediaN E-Culture Demonstrator3 . It describes. This subtlety enables a more specific search and
offers access to virtual cultural heritage collections based on solidifies a new standard in semantic multimedia search.
semantic technologies[4]. MultimediaN maps painters and
their oeuvre to a time-line to visualize historical references. The next section describes the CONTENTUS user interface
Furthermore, search results are clustered based on the dis- in detail and points out its new features and their purpose.
tance in the RDF graph between corresponding instances
of the given search terms [3]. The ‘Europeana’ project4
THE CONTENTUS USER INTERFACE
supports searching for text, images, video, and sound in a
Considering Fig. 2, the layout of the CONTENTUS user in-
tremendous collection of paintings, music, films, and books
terface is arranged in a Search Area (1) and a Result Area6 .
from Europe’s galleries and archives. A defaulted multi-
The Search Area spreads over the whole width of the layout.
modal view shows text, images, videos and sounds within
The Result Area below is divided into three columns. The
a single result page. With simple facet filtering the results
left column (2) hosts the functions to organize and filter, the
can be refined, but a distinct selection on persons, locations,
right column (3) the functions to refine and filter the search
events, or other semantic entities is not possible. ‘Culture-
query. The main column (4) in the middle shows the actual
Sampo’ is a system for creating a collective semantic mem-
search results. The Search Area contains the Search Field
ory of the cultural heritage of a nation. It provides vari-
and the Search Path. The Search Field is the central element
ous interfaces for faceted semantic recommendations, orga-
and is located in the upper left area. When entering a new
nizing places, people, and relations from a collaboratively
search term, achieved results are listed in the main column.
generated ontology [2]. In contrast, the CONTENTUS ap-
5
http://www.freebase.com/labs/parallax/
3 6
http://e-culture.multimedian.nl/demo/search Search results have been modified to English language for better
4
http://www.europeana.eu readability. Actually, the search engine content is in German.
Figure 2. The CONTENTUS user interface.
Search Results examining results on different pages disappear.
The search interface combines different media and entity
types within one result page. The types comprise books, Timeline Filter
newspaper-articles, videos, images, audio, locations, and per- To filter the search results, a horizontal slider has been placed
sons. Thereby, locations and persons are entities and not in the upper area of the right column. The handles of the
documents. All types can be visually distinguished by an slider select the starting and ending date of search results.
individual layout and icon set. For example, search results An intuitive use of a time-line based navigation is difficult
for locations reveal a geo-map, and an image preview is pre- to realize because usually there are too many different dates
sented in video image results. The relevance ranking of the available for different contexts. In the case of a person this
search result is supported by different sizes of the single re- can be the date of birth, the date of death, or any other event
sult items. The more relevant an item is, the more space related to that person. In the case of a document it is not clear
is provided to show related information. This enables to ex- if the date of production or dates related to the documents’
pose the most relevant results in more detail and thus provide content should be depicted on the time-line. The time-line
more information to the user. Every search result is featured in the CONTENTUS user interface exclusively makes use
with a title and specific information depending on its media of the publication dates. In comparison to MultimediaN the
or entity type. For example, text results and results based on CONTENTUS user interface does not use plenty of room
transcripts (e.g. video, audio) are presented with highlighted for its time-line. MultimediaN has more space to show dates
text snippets and are underpinned with semantic entities ex- and therefore lets its user render more complex time-based
tracted form the documents content (e.g. as shown in the operations. The time-line of CONTENTUS plays a signifi-
first result item). cant role in the interplay of the interface elements, yet it does
not play the lead. It has to share the space with other equally
Navigation important elements.
The design of the search process outweighs the design of the
single page and the former page based navigation on the web Facet Filter
changes by the means of Ajax to a more coherent user expe- The list of facets is located below the time-line. The facets
rience. The interface makes use of this technologies with the used by CONTENTUS are extracted entities grounded to
result that elements add to and change on the screen with the Wikipedia articles. The location facet includes various places,
aim to make the separation between entering a search and sites, cities and countries. The person facet lists persons that
are mentioned in the various media items. As opposed to Demonstration Screencast
this, the Contributor facet lists people that have been en- Because of copyright protection on its content, a public live
gaged in the media document creation process, as e.g., au- demo of CONTENTUS7 can only be provided at the confer-
thors of articles and books, directors of videos, musicians or ence. Anyway, a screencast demonstrates the user interface
composers. Facets for music style is derived from the meta in all details (c.f. http://www.yovisto.com/labs/vissw2011).
data attached to audio files. Publication years are listed in The following section describes some of the scenarios shown
the publication year facet. Various accumulations, political in the screencast.
parties and companies are listed in the organization facet.
The concept facet collects subjects and notions not fitting SEARCH SCENARIOS AND DEMONSTRATION
into the other facets. This section demonstrates the capabilities of the CONTEN-
TUS user interface by means of various search scenarios.
Breadcrumbs
Selected facets are collected as boxes to the right end of the Disambiguation
search path (1). As a result the user is always aware of her The example starts with querying CONTENTUS for the cur-
current refinements by viewing an ordered list of activated rent German chancellor. While typing the first name ‘An-
filters. After several iterations the search path usually con- gela’ a drop down box with a disambiguation of the term
tains a number of filter boxes. Since the users should be comes up. In this case the user selects the suggested term
able to refine their search, the filter boxes in the search path ‘Angela Merkel’ and several results of various media types
contain buttons to deactivate and to remove items from the dealing with the German Bundeskanzler are displayed. To
search path and thus to change the results quickly. make sure that the interface only lists documents related to
‘Angela Merkel’ as a person, the user now can select the
History person facet. After the selection only Angela Merkel and
The search history enables the user to return to any possible ministers appointed to her cabinet are shown in the results.
step of the preceding search. Within a list the user can ac-
cess recent actions sorted by date in ascending order. Icons Disambiguating Roles of Persons
indicate, if a new search term was entered, a filter was set or A query for ‘Hanns Eisler”, a famous German composer and
unset, or if a detail has been accessed. companion of playwright Bertolt Brecht, lists documents that
contain the search terms ‘Hanns Eisler”, but lets the user dis-
Collections
tinguish between Hanns Eisler as detected entity, as author,
and as topic in the media.
The user can store and collect relevant search result items
in his own collection simply by clicking the item’s collect
button. A preview of this collection is always shown on the Persons and Exploratory Search
lower end of the left column. This presence collection of When the user toggles the facet box named ‘Hanns Eisler’
media items is used to generate a user profile. The user can among the bread-crumbs, again more search results are shown
adapt the profile according to her preferences. The user pro- and Bertolt Brecht is listed as a related person to Hanns
file can be utilized to personalize the search results. Eisler in the person facet on the right hand’s side. The user
now clicks on ‘Bertolt Brecht’ to get documents related to
the German author. The user now starts a new search and
Media and Entity Details enters ‘Bertolt Brecht’, confirms the suggestion and finds
By clicking on an item within the search results, the media Brecht’s entity page listed as first search result. She clicks
or entity detail page displays all relevant information about on it and finds relations derived from the underlaying on-
this item. An example for a newspaper article is shown in tology. She selects Ernst Busch from the relationship Has
Fig. 3. The article viewer (left) displays the original scanned Professional Contact and is directed to Busch’s entity page.
document page. All detected semantic entities within the There, she finds a graph depicting the relationships of Ernst
page are highlighted. The color key encodes the entities Bush. As e.g., she confirms the relation to Brecht and learns
class (e.g. red for locations, purple for persons, etc.). On the that Busch was not only a musician, and singer, but also a
right, all pages of the underlying newspaper can be selected. playwright and director.
Below, the article text extracted by OCR is provided and all
related entities extracted from the text are registered. By
Video Search Example
clicking on an entity, its detail page appears (not depicted)
and provides the further information. On the entity detail This example assumes that a researcher wants to find addi-
page, relationships to other resources and entities are ex- tional information for a report on the rescue of the Chilean
posed in a relationship graph and allow further navigation. miners that took place in Oct. this year. Therefor she re-
The relationships are derived from the PND [5] provided by quires footage of good quality showing similar events. She
DNB. Furthermore, entities are mapped to DBpedia [1] to has a vague idea that a similar rescuing took place in Ger-
expose related information. The depicted article detail page many in the 1960s. She queries for ‘Bergleute’ (German for
(cf. Fig. 3) is just an example of visualizing media types in ‘miners”), limits the results by the means of the Timeline on
CONTENTUS. For video and audio items a media player, a the right-hand side to the 1960s and now sees two videos,
transcript, and a temporal segmentation are displayed on the one from November 7, 1963 and one from November 10,
7
detail page to enable quick navigation within the media. http://mediaglobe.yovisto.com:8080/contentus/
Figure 3. Details page of newspaper article with highlighted semantic entities.
1963. The first video deals with the accident of the miners, 3. G. Schreiber et al. . Semantic annotation and search of
the second with their rescue. cultural-heritage collections: The MultimediaN
E-Culture demonstrator. Web Semant., 6:243–249,
November 2008.
Pre-selections and Final Decisions
In an exploratory search many users first pick up some can- 4. G. Schreiber et al. MultimediaN E-Culture
didates, continue their search and then make a final decision Demonstrator. In I. Cruz, S. Decker et al., editor, The
among these pre-selections. The interface makes it easy to Semantic Web, volume 4273 of Lecture Notes in
select the findings in the Media Shelf collection. A click Computer Science, pages 951–958. Springer Berlin /
on the icon on top of a result tile or or the detail page adds Heidelberg, 2006.
the item to the this collection. A quick preview supports the 5. German National Library. Liste der fachlichen
user’s decision-making procedure. Nachschlagewerke zu den Normdateien (GKD, PND,
SWD). Leipzig ; Frankfurt, M. ; Berlin – ISSN
CONCLUSION AND FUTURE WORK 1438–1133., April 2009.
When showing various media types on one result page, spe- 6. A. Heß. Technologies for next-generation multi-media
cial care has to be devoted to the feedback that lets the user libraries: the contentus project. In Proceedings of the
know why exactly the listed items have been presented. To 3rd international workshop on Automated information
find an appropriate balance between the demand for details extraction in media production, AIEMPro ’10, pages
on the one hand and ease of use on the other hand is sub- 1–2, New York, NY, USA, 2010. ACM.
ject to further research on CONTENTEUS. Currently CON-
TENTUS is a system for individual users. In future versions 7. J. Nandzik, A. Heß, J. Hannemann, N. Flores-Herr, and
we plan to expand the possibility to connect with friends and K. Bossert. Contentus - towards semantic multimedia
to share results with them. Then, it will be possible to make libraries. In World Library and Information Congress:
suggestions to the user not only by her personal profile but 76th IFLA General Conference and Assembly,
also by the profiles of her friends. Gothenburg, 2010.
8. Semantic Web Education and Outreach Interest Group.
REFERENCES http://esw.w3.org/topic/sweoig/taskforces/communitypro
1. S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, jects/linkingopendata/, 2009.
R. Cyganiak, and Z. Ives. DBpedia: A Nucleus for a 9. J. Waitelonis and H. Sack. Towards Exploratory Video
Web of Open Data. In Proc. of 6th Int. Semantic Web Search Using Linked Data. In ISM ’09: Proc. of the
Conf., 2nd Asian Semantic Web Conf., pages 722–735, 2009 11th IEEE Int. Symp. on Multimedia, pages
November 2008. 540–545, Washington, DC, USA, 2009. IEEE
Computer Society.
2. E. Hyvönen et al. CultureSampo – A National
Publication System of Cultural Heritage on the 10. S. Weitbruch, I. Doser, and R. Zwing. Automatic
Semantic Web 2.0. In Proc. of the 6th European structured content archiving towards improved
Semantic Web Conference, Heraklion, Greece, May multimedia search. In NAB (National Association of
2009. Springer-Verlag. Broadcasters) Conference, Las Vegas, 2009. NAB.