=Paper=
{{Paper
|id=None
|storemode=property
|title=ConnectME: Semantic Tools for Enriching Online Video with Web Content
|pdfUrl=https://ceur-ws.org/Vol-932/paper11.pdf
|volume=Vol-932
|dblpUrl=https://dblp.org/rec/conf/i-semantics/NixonBBKP12
}}
==ConnectME: Semantic Tools for Enriching Online Video with Web Content==
https://ceur-ws.org/Vol-932/paper11.pdf
Proceedings of the I-SEMANTICS 2012 Posters & Demonstrations Track, pp. 55-62, 2012.
Copyright © 2012 for the individual papers by the papers' authors. Copying permitted only
for private and academic purposes. This volume is published and copyrighted by its editors.
ConnectME: Semantic Tools for Enriching Online Video
with Web Content
Lyndon Nixon1, Matthias Bauer1, Cristian Bara2, Thomas Kurz3, and John Pereira3
1
STI International, Neubaugasse 10/15, 1070 Vienna, Austria
{lyndon.nixon, matthias.bauer}@sti2.org
2
Seekda GmbH, Neubaugasse 10/15, 1070 Vienna, Austria
cristian.bara@seekda.com
3
Salzburg Research, Jakob Haringer Str. 5/3, 5020 Salzburg, Austria
{thomas.kurz, john.pereira}@salzburgresearch.at
Abstract. This poster and demo submission presents a set of tools and an ex-
tended framework with API for enabling the semantically empowered enrich-
ment of online video with Web content. As audiovisual media is increasingly
transmitted online, new services deriving added value from such material can
be imagined. For example, combining it with other material elsewhere on the
Web which is related to it or enhances it in a meaningful way, to the benefit of
the owner of the original content, the providers of the content enhancing it and
the end consumer who can access and interact with these new services. Since
the services are built around providing new experiences through connecting dif-
ferent related media together, we consider such services to be Connected Media
Experiences (ConnectME). This paper presents a toolset for ConnectME – an
online annotation tool for video and a HTML5-based enriched video player – as
well as the ConnectME framework which enables these media experiences to be
generated on the server side with semantic technology.
Keywords: Hypervideo, clickable video, Web media, Linked Data, media link-
ing, annotation, enrichment
1 Introduction
ConnectME is a project which began in June 2011 as a nationally funded project in
Austria. The participating partners are STI International (research.sti2.org), Salzburg
Research (www.salzburgresearch.at), Planet Digital (www.planet-digital.com) and
Yoovis GmbH (www.yoovis.tv). The goal of ConnectME is to develop a hypervideo
platform based on open Web standards for the delivery of interactive video experi-
ences and Web services which support the conceptual annotation of video, Web-based
linkage between concepts and content, and on-the-fly augmentation of video with
content including aspects of personalisation and contextualisation. In this submission,
we present the Web based annotation tool for video, which generates storable and
sharable RDF based media annotations, the Web based hypervideo player, and the
ConnectME framework, which extends an existing system known as the Linked Me-
55
ConnectME: Semantic Tools for Enriching Online Video with Web Content
dia Framework, which handles the server side processing from the media annotations
to the final content for the enriched video.
2 Related Work
While hypervideo – the idea of hyperlinking to content from within video – has been
around since the 1980s1, the combination of online video, semantic annotation and
Web linking in ConnectME is to the best of the authors’ knowledge unique in the
field. Online video is a clear trend in media consumption, yet the automated associa-
tion of videos to related Web material is still a subject of technology demos like
Mozilla’s Popcorn2 which uses textual tags associated to video to link into Wikipeda
articles, maps and so on. Semantics could solve the inherent ambiguities of textual
tagging. Work on semantic annotation of video has focused on using the rich metada-
ta captured in improving multimedia indexing, search and retrieval, but the role it
could play in enabling an enriched playout of the video is taken up anew in Connect-
ME. Traditionally, multimedia presentation systems [1] have indeed relied on formal
knowledge about the multimedia but not agreed on a shared model for that
knowledge. Earlier work on the Cuypers presentation engine [2] did explore use of
RDF based knowledge models [3]. The emergence of Linked Data has meant seman-
tic annotations can refer to freely accessible Web based metadata which can be re-
used in UIs for content selection and browsing, but work has gone not much further
than the limited media linked to directly from Linked Data descriptions [4]. Automat-
ed linking from semantic annotations to online content related to the annotation needs
to incorporate Multimedia Information Retrieval techniques [5] and benefit from in-
creased publication of media metadata in a structured/Linked Data form [6]. The
state of the art in Web hypervideo today does not have answers to these issues being
addressed by research in ConnectME, and hence focuses on manual annotation and
linkage to other content in the video (see Web based offers by companies such as
WireWax, Videoclix, Overlay.TV or Klickable).
3 ConnectME workflow
The ConnectME workflow uses a set of executable Web services called from a server
side platform which also provides for the workflow’s data storage and retrieval in
order to generate, from the starting point of a semantic annotation of an online video,
a final set of content linked to spatial and temporal moments in the video that can be
played out as a form of dynamic content enrichment in the ConnectME hypervideo
player. Figure 1 provides a high level view of this workflow. The main steps in the
workflow, printed on the left, are to identify objects in video, annotate them with
1
Systems such as Hypersoap (www.media.mit.edu/hypersoap/) demonstrated the possibility of
interactive product placement in a broadcast setting
2
http://webmademovies.etherworks.ca/popcorndemo/
56
ConnectME: Semantic Tools for Enriching Online Video with Web Content
(Linked Data3) concepts and make use of this annotation to link the video objects to
other Web content. From an implementation perspective, this means a hypervideo
annotation tool (section 3) to help humans generate the semantic annotation of video,
a ConnectME framework (section 4) which orchestrates the use of various compo-
nents to support the steps of annotation, concept selection, content selection and
packaging for playout on the client, and a hypervideo player (section 5) to give the
client access to the enriched video.
Fig. 1. ConnectME workflow
4 ConnectME Annotation Tool
ConnectME has developed a Web-based hypervideo annotation tool in PHP. The user
interface uses HTML5, the Video.js player and jQuery with extensions to provide for
video loading and manipulation, such as selecting spatial and temporal parts of the
video, and hence works across all latest versions of Web browsers.
Using the HTML 5 video tag for embedding video files allows playing videos
without need for any Flash-based plug-in and the Video.js library provides several
useful video control methods. Ajax powers as-you-type concept suggestion from the
DBPedia4 concept base to support annotators in quickly finding the right concept: also
when a concept is selected, the bottom right hand area shows some explanatory text
(the DBPedia concept’s abstract) to help annotators be sure they choose the correctly
3
http://linkeddata.org provides a Web based concept space with URI based metadata look-up
for more information about concepts
4
http://dbpedia.org publishes Wikipedia data in RDF, hence every Wikipedia page has a
DBPedia Linked Data URI for its subject
57
ConnectME: Semantic Tools for Enriching Online Video with Web Content
intended concept. Furthermore, the annotation tool supports searching for geographic
locations in Geonames5. The results are displayed by using the Google Maps JavaS-
cript API V3. Locally the annotation being made is stored as JSON in a file on the
server and the file itself gets referenced by using a Cookie, so that even if the browser
page is reloaded or closed/opened again, the annotation task can be continued without
loss of information. When the annotation is saved, the tool is directly connected to an
instance of the ConnectME framework which stores the annotations and makes them
available to the ConnectME workflow when the video is requested from the hyper-
video player. For the video annotation schema a RDF based format has been selected
which is based around the W3C Media Ontology with some extensions for enabling
an annotation of the annotations (who made them, when, what are the rights for re-
use) re-using the Open Annotation Model, and a backwards compatible ConnectME
specific extension for describing how concepts are represented by the video object,
which is leveraged in the framework.
Fig. 2. ConnectME annotation tool
5 ConnectME Framework
ConnectME draws on the concept of Linked Media to enable a Web based connection
between concepts from the Linked Data concept space and Web content which, for
the purposes of this linking, have been annotated in terms of Linked Data concepts.
The key principles of Linked Media are:
Web media needs to be annotated in terms of its online parts
5
http://www.geonames.org offers a geographical database which covers all countries and con-
tains over eight million placenames all over the world.
58
ConnectME: Semantic Tools for Enriching Online Video with Web Content
Web media needs to be annotated with terms which represent a shared understand-
ing of a domain or identification of a thing
Web media needs to be annotated using a media ontology which supports the
above two issues
The expressed representation of different concepts by different media fragments in
different ways shall be the basis to interlink media across the Web
The first three points are covered in the annotation tool (W3C Media Fragments
specification, Linked Data as concept namespace, W3C Media Ontology and exten-
sions as annotation scheme). The fourth point is part of a Linked Media implementa-
tion in our framework.
5.1 Linked Media Framework
The Linked Media Framework6 is an easy-to-setup server application that bundles
central Semantic Web technologies to offer advanced services. The Linked Media
Framework consists of LMF Core and LMF Modules. The core component of the
Linked Media Framework is a Linked Data Server that allows to expose data follow-
ing the Linked Data Principles. The Linked Data Server implemented as part of the
LMF goes beyond the Linked Data principles by extending them with Linked Data
Updates and by integrating management of metadata and content and making both
accessible in a uniform way. As extension for the LMF Core, we are working on a
number of optional modules that can be used to extend the functionality of the Linked
Media Server:
• LMF Permissions implements and extends the WebID and WebACL specifica-
tions for standards-conforming authentication and access control in the Linked
Media Framework
• LMF Media Interlinking will implement support for multimedia interlinking
based on the work in the W3C Multimedia Fragments WG and the W3C Multi-
media Annotations WG
• LMF Reasoner implements a rule-based reasoner that allows to process Datalog-
style rules over RDF triples; the LMF Reasoner will be based on the reasoning
component developed in the KiWi project, the predecessor of the LMF
• LMF Versioning implements versioning of metadata updates; versioning itself is
already carried out by LMF Core, but the management of versions will be carried
out by this module
• LMF Enhancer offers semantic enhancement of content by analysing textual and
media content; the LMF Enhancer will build upon UIMA, Apache Tika, and the
semantic lifting engine of the Apache Stanbol framework7
6
http://www.newmedialab.at/LMF
7
http://incubator.apache.org/stanbol/
59
ConnectME: Semantic Tools for Enriching Online Video with Web Content
5.2 Connected Media Framework
In order to implement the Connected Media Framework, we chose to build upon the
Linked Media Framework, which already offered out of the box much of the neces-
sary basis functionality such as storage and retrieval of the semantic media annota-
tions, as well as a means to access media or its metadata in a straightforward manner,
following Linked Data principles. Since additional functionalities are plugged in via
modules, ConnectME develops its own specific modules to turn the Linked Media
Framework into a Connected Media Framework: The concept extraction module sup-
ports the video annotation tool by suggesting concepts to link to the video via textual
analysis of available subtitling or transcript files for that video. For this, an instance of
Apache Stanbol has been specifically trained to handle the particular corpus of con-
cepts in ConnectME materials. The Linked Media engine is a specific component
implementation which exposes multimedia object descriptions to the ConnectME
workflow in a common structured metadata format. The provision of usable multime-
dia object descriptions on the Web as Linked Data is referred to in the project as the
Linked Media layer. To find objects relevant for any concept in the video annotation,
media repositories need to be queried and their responses provisioned as Linked Me-
dia. Hence the engine incorporates a semantic service middleware which brokers
between ConnectME and heterogeneous media sources (Web APIs, SPARQL end-
points, etc.).
Fig. 3. ConnectME extending the Linked Media Framework.
6 ConnectME Hypervideo Player
The current implementation of the hypervideo player is HTML5 based (making us of
the video tag, video.js, JQuery, CSS3, JSON2, backbone.js, underscore.js, RDFQuery
and VIE library) and runs in any latest version of any of the main browsers. The play-
er incorporates support for the W3C Media Fragment syntax that should allow video
to be accessed not as an entire media resource but in terms of a temporal and/or spa-
60
ConnectME: Semantic Tools for Enriching Online Video with Web Content
tial part thereof. As the video plays, a Javascript code checks for annotations on the
next active video segment, and enables access to additional content when it is relevant
to the concept annotating that segment via a plugin and widget architecture. Annota-
tions refer to Linked Data resources and the ConnectME framework has collected
links to content relevant to those resources. The Hypervideo player has a core that
sustains the video playback mechanism and connects to the ConnectME Framework
to retrieve the annotations in an initialization phase. A set of plugins is then attached
to the core, each of which is specialized in recognizing a certain type of annotation
resource. Plugins will retrieve and render relevant content for given resources and
display them in the form of widgets. In Figure 4, a depiction plugin rendered a pic-
ture of Zorbing and created a widget, displayed on the left hand side; an abstract
plugin fetched the abstract and concept label fields from DBPedia that describes
Zorbing and composed a widget displayed on the right hand side. Since plugins can
be configured for any Linked Data source, the player architecture is very flexible
regarding the content selected and displayed in a widget.
Fig. 4. ConnectME hypervideo player.
7 Conclusions
The ConnectME annotation tool, framework and player act as a proof of concept for
semantics-based dynamic enrichment of videos based on Linked Data annotations. In
the project we continue to explore how we can better automate and simplify the anno-
tation step for the user, maximize the flexibility and relevance of the linkage to online
media resources, and improve the intuitiveness of the user interaction with the hyper-
video player in order to easily and effectively access and browse video enrichments,
on desktop, tablet and SmartTV platforms.
61
ConnectME: Semantic Tools for Enriching Online Video with Web Content
8 Acknowledgements
This work has been performed in the Austrian project ConnectME
(http://www.connectme.at) which is funded by the FFG.
9 References
1. Bordegoni, M., Faconti, G., Maybury, M.T., Rist, T., Ruggieri, S., Trahanias, P., Wilson,
M.: A Standard Reference Model for Intelligent Multimedia Presentation Systems. Com-
puter Standards & Interfaces, 18:477-496, 1997.
2. Ossenbruggen, J. R., Cornelissen, F. J., Geurts, J. P., Rutledge, L. W., Hardman, L.
Cuypers: a Semi-Automatic Hypermedia Generation System. Technical Report. UMI Or-
der Number: INS-R0025., CWI (Centre for Mathematics and Computer Science), 2000.
3. Lindley, C. A., Davis, J. R., Nack, F., Rutledge, L. W.: The Application of Rhetorical
Structure Theory to Interactive News Program Generation from Digital Archives. Tech-
nical Report. UMI Order Number: INS-R0101, CWI (Centre for Mathematics and Com-
puter Science), 2001.
4. Schreiber, G., et al.: Semantic annotation and search of cultural-heritage collections: The
MultimediaN E-Culture, Demonstrator. Journal of Web Semantics (JWS), 6(4):243–249,
2008.
5. Hanjalic, A., Lienhart, R., Ma, W.-Y., Smith, J. R.: The Holy Grail of Multimedia Infor-
mation Retrieval: So Close or Yet So Far Away? In: Proceedings of the IEEE, 96(4):541–
547, 2008.
6. Bürger, T., Simperl, E.: A Conceptual Model for Publishing Multimedia Content on the
Semantic Web. In: Proceedings of the 4th International Conference on Semantic and Digi-
tal Media Technology (SAMT ‘09), 02.12-04.12., Graz, Austria, 2009
62