This poster and demo submission presents a set of tools and an extended framework with API for enabling the semantically empowered enrichment of online video with Web content. As audiovisual media is increasingly transmitted online, new services deriving added value from such material can be imagined. For example, combining it with other material elsewhere on the Web which is related to it or enhances it in a meaningful way, to the benefit of the owner of the original content, the providers of the content enhancing it and the end consumer who can access and interact with these new services. Since the services are built around providing new experiences through connecting different related media together, we consider such services to be Connected Media Experiences (ConnectME). This paper presents a toolset for ConnectME - an online annotation tool for video and a HTML5-based enriched video player - as well as the ConnectME framework which enables these media experiences to be generated on the server side with semantic technology.
ConnectME is a project which began in June 2011 as a nationally funded project in Austria. The participating partners are STI International (research.sti2.org), Salzburg Research (www.salzburgresearch.at), Planet Digital (www.planet-digital.com) and Yoovis GmbH (www.yoovis.tv). The goal of ConnectME is to develop a hypervideo platform based on open Web standards for the delivery of interactive video experiences and Web services which support the conceptual annotation of video, Web-based linkage between concepts and content, and on-the-fly augmentation of video with content including aspects of personalisation and contextualisation. In this submission, we present the Web based annotation tool for video, which generates storable and sharable RDF based media annotations, the Web based hypervideo player, and the ConnectME framework, which extends an existing system known as the Linked Media Framework, which handles the server side processing from the media annotations to the final content for the enriched video. 2
While hypervideo – the idea of hyperlinking to content from within video – has been
around since the 1980s1, the combination of online video, semantic annotation and
Web linking in ConnectME is to the best of the authors’ knowledge unique in the
field. Online video is a clear trend in media consumption, yet the automated
association of videos to related Web material is still a subject of technology demos like
Mozilla’s Popcorn2 which uses textual tags associated to video to link into Wikipeda
articles, maps and so on. Semantics could solve the inherent ambiguities of textual
tagging. Work on semantic annotation of video has focused on using the rich
metadata captured in improving multimedia indexing, search and retrieval, but the role it
could play in enabling an enriched playout of the video is taken up anew in
ConnectME. Traditionally, multimedia presentation systems [
The ConnectME workflow uses a set of executable Web services called from a server side platform which also provides for the workflow’s data storage and retrieval in order to generate, from the starting point of a semantic annotation of an online video, a final set of content linked to spatial and temporal moments in the video that can be played out as a form of dynamic content enrichment in the ConnectME hypervideo player. Figure 1 provides a high level view of this workflow. The main steps in the workflow, printed on the left, are to identify objects in video, annotate them with 1 Systems such as Hypersoap (www.media.mit.edu/hypersoap/) demonstrated the possibility of interactive product placement in a broadcast setting 2 http://webmademovies.etherworks.ca/popcorndemo/ (Linked Data3) concepts and make use of this annotation to link the video objects to other Web content. From an implementation perspective, this means a hypervideo annotation tool (section 3) to help humans generate the semantic annotation of video, a ConnectME framework (section 4) which orchestrates the use of various components to support the steps of annotation, concept selection, content selection and packaging for playout on the client, and a hypervideo player (section 5) to give the client access to the enriched video. ConnectME has developed a Web-based hypervideo annotation tool in PHP. The user interface uses HTML5, the Video.js player and jQuery with extensions to provide for video loading and manipulation, such as selecting spatial and temporal parts of the video, and hence works across all latest versions of Web browsers.
Using the HTML 5 video tag for embedding video files allows playing videos without need for any Flash-based plug-in and the Video.js library provides several useful video control methods. Ajax powers as-you-type concept suggestion from the DBPedia4 concept base to support annotators in quickly finding the right concept: also when a concept is selected, the bottom right hand area shows some explanatory text (the DBPedia concept’s abstract) to help annotators be sure they choose the correctly 3 http://linkeddata.org provides a Web based concept space with URI based metadata look-up for more information about concepts 4 http://dbpedia.org publishes Wikipedia data in RDF, hence every Wikipedia page has a DBPedia Linked Data URI for its subject intended concept. Furthermore, the annotation tool supports searching for geographic locations in Geonames5. The results are displayed by using the Google Maps JavaScript API V3. Locally the annotation being made is stored as JSON in a file on the server and the file itself gets referenced by using a Cookie, so that even if the browser page is reloaded or closed/opened again, the annotation task can be continued without loss of information. When the annotation is saved, the tool is directly connected to an instance of the ConnectME framework which stores the annotations and makes them available to the ConnectME workflow when the video is requested from the hypervideo player. For the video annotation schema a RDF based format has been selected which is based around the W3C Media Ontology with some extensions for enabling an annotation of the annotations (who made them, when, what are the rights for reuse) re-using the Open Annotation Model, and a backwards compatible ConnectME specific extension for describing how concepts are represented by the video object, which is leveraged in the framework. ConnectME draws on the concept of Linked Media to enable a Web based connection between concepts from the Linked Data concept space and Web content which, for the purposes of this linking, have been annotated in terms of Linked Data concepts. The key principles of Linked Media are:
Web media needs to be annotated in terms of its online parts 5 http://www.geonames.org offers a geographical database which covers all countries and contains over eight million placenames all over the world.
Web media needs to be annotated with terms which represent a shared understanding of a domain or identification of a thing
Web media needs to be annotated using a media ontology which supports the above two issues
The expressed representation of different concepts by different media fragments in different ways shall be the basis to interlink media across the Web
The first three points are covered in the annotation tool (W3C Media Fragments specification, Linked Data as concept namespace, W3C Media Ontology and extensions as annotation scheme). The fourth point is part of a Linked Media implementation in our framework. 5.1
The Linked Media Framework6 is an easy-to-setup server application that bundles central Semantic Web technologies to offer advanced services. The Linked Media Framework consists of LMF Core and LMF Modules. The core component of the Linked Media Framework is a Linked Data Server that allows to expose data following the Linked Data Principles. The Linked Data Server implemented as part of the LMF goes beyond the Linked Data principles by extending them with Linked Data Updates and by integrating management of metadata and content and making both accessible in a uniform way. As extension for the LMF Core, we are working on a number of optional modules that can be used to extend the functionality of the Linked Media Server: • LMF Permissions implements and extends the WebID and WebACL specifications for standards-conforming authentication and access control in the Linked Media Framework • LMF Media Interlinking will implement support for multimedia interlinking based on the work in the W3C Multimedia Fragments WG and the W3C Multimedia Annotations WG • LMF Reasoner implements a rule-based reasoner that allows to process Datalogstyle rules over RDF triples; the LMF Reasoner will be based on the reasoning component developed in the KiWi project, the predecessor of the LMF • LMF Versioning implements versioning of metadata updates; versioning itself is already carried out by LMF Core, but the management of versions will be carried out by this module • LMF Enhancer offers semantic enhancement of content by analysing textual and media content; the LMF Enhancer will build upon UIMA, Apache Tika, and the semantic lifting engine of the Apache Stanbol framework7
5.2
In order to implement the Connected Media Framework, we chose to build upon the Linked Media Framework, which already offered out of the box much of the necessary basis functionality such as storage and retrieval of the semantic media annotations, as well as a means to access media or its metadata in a straightforward manner, following Linked Data principles. Since additional functionalities are plugged in via modules, ConnectME develops its own specific modules to turn the Linked Media Framework into a Connected Media Framework: The concept extraction module supports the video annotation tool by suggesting concepts to link to the video via textual analysis of available subtitling or transcript files for that video. For this, an instance of Apache Stanbol has been specifically trained to handle the particular corpus of concepts in ConnectME materials. The Linked Media engine is a specific component implementation which exposes multimedia object descriptions to the ConnectME workflow in a common structured metadata format. The provision of usable multimedia object descriptions on the Web as Linked Data is referred to in the project as the Linked Media layer. To find objects relevant for any concept in the video annotation, media repositories need to be queried and their responses provisioned as Linked Media. Hence the engine incorporates a semantic service middleware which brokers between ConnectME and heterogeneous media sources (Web APIs, SPARQL endpoints, etc.).
The current implementation of the hypervideo player is HTML5 based (making us of the video tag, video.js, JQuery, CSS3, JSON2, backbone.js, underscore.js, RDFQuery and VIE library) and runs in any latest version of any of the main browsers. The player incorporates support for the W3C Media Fragment syntax that should allow video to be accessed not as an entire media resource but in terms of a temporal and/or spatial part thereof. As the video plays, a Javascript code checks for annotations on the next active video segment, and enables access to additional content when it is relevant to the concept annotating that segment via a plugin and widget architecture. Annotations refer to Linked Data resources and the ConnectME framework has collected links to content relevant to those resources. The Hypervideo player has a core that sustains the video playback mechanism and connects to the ConnectME Framework to retrieve the annotations in an initialization phase. A set of plugins is then attached to the core, each of which is specialized in recognizing a certain type of annotation resource. Plugins will retrieve and render relevant content for given resources and display them in the form of widgets. In Figure 4, a depiction plugin rendered a picture of Zorbing and created a widget, displayed on the left hand side; an abstract plugin fetched the abstract and concept label fields from DBPedia that describes Zorbing and composed a widget displayed on the right hand side. Since plugins can be configured for any Linked Data source, the player architecture is very flexible regarding the content selected and displayed in a widget. The ConnectME annotation tool, framework and player act as a proof of concept for semantics-based dynamic enrichment of videos based on Linked Data annotations. In the project we continue to explore how we can better automate and simplify the annotation step for the user, maximize the flexibility and relevance of the linkage to online media resources, and improve the intuitiveness of the user interaction with the hypervideo player in order to easily and effectively access and browse video enrichments, on desktop, tablet and SmartTV platforms.
8 9 This work has been performed in the Austrian (http://www.connectme.at) which is funded by the FFG. project