=Paper=
{{Paper
|id=None
|storemode=property
|title=Synote: Weaving Media Fragments and Linked Data
|pdfUrl=https://ceur-ws.org/Vol-937/ldow2012-paper-01.pdf
|volume=Vol-937
|dblpUrl=https://dblp.org/rec/conf/www/LiWOSW12
}}
==Synote: Weaving Media Fragments and Linked Data==
https://ceur-ws.org/Vol-937/ldow2012-paper-01.pdf
Synote: Weaving Media Fragments and Linked Data
Yunjia Li,Mike Wald,Tope Omitola,Nigel Shadbolt and Gary Wills
School of Electronics and Computer Science
University of Southampton
UK
{yl2,mw,tobo,nrs,gbw}@ecs.soton.ac.uk
ABSTRACT how these relationships can be processed automatically. How-
While end users could easily share and tag the multime- ever, most applications are yet to expose sufficient semantic
dia resources online, the searching and reusing of the in- annotations for media fragments, which leads to the diffi-
side content of multimedia, such as a certain area within an culty of processing complex searching and providing accu-
image or a ten minutes segment within a one-hour video, rate searching results. To solve this problem, some research,
is still difficult. Linked data is a promising way to inter- such as Core Ontology of Multimedia (COMM) [3], has en-
link media fragments with other resources. Many applica- abled semantic annotations for media fragments. But this
tions in Web 2.0 have generated large amount of external approach has not yet proved to be scalable on the Web
annotations linked to media fragments. In this paper, we [13]. Linked data [6] is more light-weight compared with
use Synote as the target application to discuss how media traditional semantic Web technologies and is highly scalable
fragments could be published together with external anno- on the Web. There are some research interests in apply-
tations following linked data principles. Our design solves ing linked data principles to media fragments [13, 21]. The
the dereferencing, describing and interlinking methods prob- recent announced schema.org1 also defined vocabularies to
lems in interlinking multimedia. We also implement a model improve the online presence of image, audio and video ob-
to let Google index media fragments which improves media jects, but the media fragment is undefined in schema.org.
fragments’ online presence. The evaluation shows that our Most research about media fragments focuses on exposing
design can successfully publish media fragments and anno- the closed data, such as tracks and video segments, within
tations for both semantic Web agents and traditional search the multimedia file or the host server using semantic Web
engines. Publishing media fragments using the design we and linked data technologies. But these efforts are inade-
describe in this paper will lead to better indexing of multi- quate for the fully interlinking and indexing of media frag-
media resources and their consequent findability. ments, because the large amount user-generated annotations
in existing Web 2.0 applications are rarely published, such
as YouTube interactive transcript2 and photo tagging func-
Categories and Subject Descriptors tions in Facebook3 . When a person is tagged in a photo, an
I.7.2 [Document Preparation]: Hypertext/hypermedia, annotation is created and connected to the media fragment,
Multi/mixed media, Standards; H.5.m [Information Sys- but the annotation is saved in an external source. More at-
tems]: Miscellaneous tention should be given to how these external annotations
from outside of multimedia host servers could benefit the
media fragment interlinking and search.
General Terms In this paper, we will present the design and evaluation of
Theory, Design an example of publishing media fragments and annotations
following linked data principles. We also discuss how the
media fragments could be made available for both semantic
Keywords indexing services, such as Sindice [40], and Google Search.
annotation,linked data,media fragment,schema.org,search The external annotation data is from Synote [22], which is
a Web 2.0 application allowing users to embed audio-visual
resources from other domains and make synchronised anno-
1. INTRODUCTION tations. The discussion in this paper focuses on the tempo-
The term “media fragment” refers to the inside content of ral dimension of audio-visual resources, which could also be
multimedia objects, such as a certain area within an image, extended to images and spatial dimension.
or a ten minutes segment within a one-hour video. With In the reminder of this paper, Section 2 reviews related
the rapid development of multimedia applications, such as vocabularies that could be used for media fragment publish-
YouTube and Flickr, end users could easily upload, share ing, and the state of art of the existing applications about
and tag the multimedia resources online. Compared with multimedia and linked data. Section 3 briefly introduces
traditional annotations, semantic annotations can tell a com- the Synote system and discusses the requirements of me-
puter how media fragments are related to other data and
1
http://schema.org
2
Copyright is held by the author/owner(s). http://goo.gl/t1nMj
3
LDOW2012, April 16, 2012, Lyon, France. http://www.facebook.com/help/photos/tag-photos
dia fragment publishing. Section 4 discusses the design to framework to describe and annotate complex multimedia
solve the problems in interlinking multimedia (iM) princi- resources. It fills the gap between the structured metadata
ples [13]. Then we present the implementation of the design models, such as SMIL and EXIF [35], and semantic anno-
and a model to allow Google indexing media fragments. Sec- tations. M3O can be integrated with various standards to
tion 5 presents the evaluation results and Section 6 provides provide semantic annotations. The key concept of M3O is
conclusions and recommendations for future work. separation of information objects and information realiza-
tions. Based on this idea, M3O comes up with a core set of
2. RELATED WORK ontology design patterns, including annotation, decomposi-
tion, collection and provenance patterns.
Linked data describes a series of methods of publishing
International Press Telecommunication Council (IPTC)
structured data using semantic Web technologies, such as
designed an ontology for IPTC News Architecture [36] in
Resource Description Framework (RDF) [24] and SPARQL
order to integrate news and multimedia metadata from the
query language [27]. Linked data enables machines to au-
industrial area into the existing knowledge on the semantic
tomatically discover more data from the data they already
Web. In [36], a local view of a video resource is introduced
know. Generally, there are four rules which must be followed
so that arbitrary sequences of a video, i.e. media fragments,
when publishing linked data on the Web [6]:1) Use URIs as
can be played in the semantic browser. As the further de-
names for things 2) Use HTTP URIs so that people can look
velopment of IPTC News Architecture, rNews vocabulary
up those names 3) When someone looks up a URI, provide
has be adopted by schema.org, which provides shared vo-
useful information 4) Include links to other URIs, so that
cabularies in the form of HTML Microdata [15]. A great
they can discover more things.
advantage of using schema.org is that the structured data
Heath and Bizer [14] summarised that there are some com-
including the audio and video objects could be recognised
mon steps which developers need to take from choosing URIs
and highlighted by major search engines, such as Google,
to the final testing and the publishing of linked data fall
Bing and Yahoo!. The vocabularies in schema.org can also
into several patterns. They also pointed out that developers
be used in the form of RDFa4 [1].
should not totally abandon “existing data management sys-
tem and business applications”, but add an “extra technical
layer of glue to connect these into the Web of Data”.
2.2 Multimedia Applications for Linked Data
Many applications have already published multimedia and
2.1 Multimedia Annotation Vocabularies annotation as linked data, which offers experience for us
If we want to publish media fragments in Synote, we need on multimedia resource publishing. Yovisto open academic
to choose how different dimensions are encoded in URIs. We video search platform publishes its database containing video
also need to consider which vocabularies should be used to and annotations as linked data using MPEG-7, COMM to
describe multimedia resources and annotations. Thus this describe multimedia data [42]. Annomation [20] is a tool to
section reviews related standards and vocabularies. handle the input annotations from users developed by the
The standardisation of media fragments is identified as an Open University in the UK. The annotations are saved in
urgent need [37]. Many standards use non-URI based mech- RDF quad store with users’ own privacy and provenance
anisms to identify media fragments, such as MPEG-7 [25] data. SemWebVid [33] can automatically generate RDF de-
and Synchronized Multimedia Integration Language (SMIL) scription by analysing plain-text resources using multiple
[17]. In these standards, the descriptions of temporal and Natural Language Processing (NLP) Web services. The en-
spatial dimensions are divided into several attributes, thus tities extracted from the NLP services are used to generate
the media fragment is not represented by a single URI. In the the RDF description of the video resources.
URI-based approaches, temporal, spatial and other dimen- LEMO multimedia annotation framework [12] publishes
sions of media fragments are encoded in query (“?”), hash media fragments using MPEG-21. LEMO converts existing
(“#”) or after slash (“/”). The temporalURI [26] defines me- video files to MPEG compatible versions and streams them
dia fragments after URI query. But Query parameter has a from LEMO server. LEMO also derived a core annotation
weakness that the resulting URI loses the direct link to the schema from Annotea Annotation Schema [19] in order to
parent resource and it does not follow the “Cool URIs” prin- link annotations to media fragments identifications. Nin-
ciple [30], so it is difficult to be applied in linked data. The Suna (Metadata-driven media adaptation and delivery) [9]
proposed URIs for Linked Stream Data [31] applied slash has a server-side implementation of MFURI 1.0. The system
URIs to include real time and space information in URIs can “ingest” different formats of multimedia resource and
together or separately. MPEG-21 [8] specifies a normative save the fragments information as semantically represented
URI fragment syntax to address fragment in MPEG com- data blocks in RDF. When delivering the media resource,
patible files. But the syntax of MPEG-21 can be ambiguous one or more data blocks can be returned to the user agent
and it is difficult to fully follow the standard [23]. (UA) according to the Range header in HTTP request as
The W3C Media Fragment Working Group has proposed well as the adaptive context of the UA. Even though some
Media Fragment URI 1.0 (MFURI) [38], which defines hash of the applications introduced above publish user-generated
URI syntax to address media fragments from temporal, spa- content with media fragments, most of them are built from
tial, track and named dimensions. MFURI is included in scratch in a linked-data-driven method. Our work on Syn-
Ontology for Media Resource 1.0 (OMR) [29], which is an ote in this paper focuses more on publishing media fragments
ontology to describe multimedia resource. Hausenblas et al. and annotations based on typical Web 2.0 applications.
have discussed the issues about iM using MFURI specifica-
tion, such as media fragment dereferencing, legacy metadata 3. PUBLISHING REQUIREMENTS
alignment and interlinking methods [13].
4
Multimedia Metadata Ontology (M3O) [28] provides a http://goo.gl/Zd8Eo
media, Synchronised Bookmark (Synmark, compromised by
one text Note and many Tag resources), Transcript (in We-
bVTT [16] format) and Presentation resource7 . Each com-
pound resource is aggregated by one or more single resource.
Every resource, no matter single or compound, has a unique
id in Synote. Multimedia and Slide resources have URI refer-
ences to online resources, such as YouTube video, audio files
and images, etc. Annotation class models the annotation
relationship between two compound resources. In Synote,
only Multimedia is the target resource and Synmark, Pre-
sentation, Transcript could be the source in an annotation.
In an annotation, Synpoints associate each single resource
in Synmark, Transcript and Presentation with a time-span
(“target start” and “target end”) of the Multimedia.
Synote Player is the main page to display all these re-
sources and annotations in an interactive and synchronised
manner. Users could click on a Synmark, transcript block or
slide (part 2, 3 and 4 respectively in Figure 2) and the mul-
timedia player (identified as 1 in Figure 2) will jump to the
Figure 1: Synote Object Model corresponding start time. In Synote Player, users can also
create Synmarks, Transcript and Presentation slides, and
synchronise them with the multimedia (Figure 3). Techni-
cally, the Synote system follows the Model-View-Controller
framework. Synote Player is developed using jQuery8 , so it
contains dynamically generated content by javascript (Ajax).
The general principles for iM have been introduced in [13]
and most of them are still the requirements for Synote. How-
ever, as Synote is not the host of multimedia resources, it
is not Synote’s responsibility, for example, to deploy the
legacy multimedia metadata and we can only publish the
metadata saved in Synote’s database. For the audio-visual
presentation of media fragments, we can still use the inter-
active annotation highlighting function in Synote Player’s
page. So the audio-visual presentation of media fragments
is actually implemented in the HTML page.
Except for the general requirements of iM, Synote also
has its own publishing requirements. Similar to major mul-
timedia applications on the Web, Synote has ready built
its own resource management system. So it is unwise to to-
tally abandon the old infrastructure and it is better that the
linked data could act like an extra layer in order to minimise
Figure 2: Screenshot of Synote Player the changes to the existing system. This requirement also
means we are not abandoning Synote Player, which could be
used as the HTML representation of different resources. Un-
This section will briefly introduce the Synote system and like a company’s registration data, media fragments and an-
discuss the requirements of media fragment publishing for notations in Synote could be generated frequently by users,
Synote. This section sets up the background for Section 4. so it is better to publish data dynamically instead of dump-
Synote5 is a Web-based multimedia annotation applica- ing data to RDF periodically like DBpedia [4]. Since the
tion [22], which synchronises user-generated tags, notes, tran- original goal of Synote is to improve the indexing of PART
scripts and images with a time-span of multimedia resources of multimedia resource [22], we need to consider how to make
(i.e. media fragments). Synote does not have its own mul- major search engines index the media fragments defined in
timedia repository and Synote’s database only saves the Synote. In addition, Synote has a permission system where
URLs of audio-visual resources and images freely available users can create private resources and they are only acces-
online. User-generated annotations and the synchronisation sible after the owner signs in. So the media fragments and
points (Synpoint) are saved in the Synote database. Fig- annotations about private resources should not be included
ure 1 demonstrates the backend object model of Synote6 . in the published datasets. Except for Multimedia resource
and media fragments, we also need to make other resources
Synote categorises the resources into single and compound and annotations dereferencable, because they are all linked
resources. Single resources are either URI references or to media fragments and should be published together.
text. Compound resources are further divided into Multi- To sum up, there are several requirements we need to fulfil
5 7
http://www.synote.org We capitalise the first character of the resource name to
6
Classes and relationships not related to this paper are not indicate that it is a class defined in Synote Object Model
8
presented in this figure http://jQuery.com
when publishing media fragments. Even though some of embed multimedia resources, reuse the media fragments and
them are special requirements for Synote, they could also be interlink them with their own annotations. Annotations in
concerns of major multimedia applications when publishing Type Two are mainly generated by users and they reflect
media fragments and annotations. users’ interests in different areas. Applications providing
Type Two annotations are actually interlinking their an-
notations to the media fragments in another domain, but
4. DESIGN AND IMPLEMENTATION they have no control of what could not be dereferenced or
This section discusses the design solutions to satisfy the retrieved via these media fragment URIs.
publishing requirements of iM and other requirements men- We must emphasise that the two sources are not neces-
tioned in Section 3. sarily in different domains. For example, YouTube has its
own video repository but it also has a replay page which em-
4.1 Interlinking Multimedia beds the video and allows users making annotations. The
The ultimate goal of iM is to “derive both semantic and data published in Type One and Two can be about the same
audio-visual representations from multimedia resources on property of a multimedia resource. An mp4 file can embed
the Web” [13]. In this section, we will discuss how Synote is the title information using MPEG-7, but when the mp4 file
designed to solve the key issues in iM. is played in another application, it may be given another ti-
MFURI is designed to deal with the closed character of tle. This is acceptable because vocabularies describing mul-
multimedia resources on the Web. It expects the cooper- timedia resources, such as OMR, do not require that all the
ation of “smart servers” and proxy caches to deliver media metadata come from the file itself. For example, the subti-
fragments via HTTP protocol [23]. Hausenblas et al. [13] tles in OMR could be an external link of the subtitle file or
proposed a solution, which uses “HTTP 206 Partial Con- a track media fragment embedded in the file10 .
tent”, content negotiation and 303 redirect to retrieve the The annotations in Synote belong to Type Two as Synote
original and RDF representation of a multimedia resource. does not host any video or image resources. In the following
There are arguments about the appropriateness of this solu- subsections, we discuss how Synote provides audio-visual
tion under the Web Architecture [18] because “a description and semantic representations of multimedia resources, which
of a multimedia resource is not the same as the resource it- is also useful for other multimedia applications.
self” [13]. As far as we are concerned, the third principle of
linked data only requires a description of the URI and it is 4.1.1 Dereferencing Media Fragment URIs
not necessary, as well as not possible sometimes, to provide It is not a good practice to augment the URIs in an-
the semantic description which could totally reflect the other other domain and include it in your own dataset (see Sec-
representation of this URI. Even though some UAs, such as tion 4.1.2.1 of [14]). Synote has no control of the hosts of
Firefox 109 , and servers, such as Ninsuna, support part of multimedia resources, so we need to mint our own URIs
the functions defined in MFURI, the general support for this for each multimedia resource and fragment. In addition,
solution is very limited currently on either client or server. we need to think about how to include both HTML and
Another solution mentioned in [13] is to embed semantic RDF representations of media fragments as well as other re-
metadata in the multimedia file like XMP [2], which means sources in Synote. We already have the replay page, which
the two representations share the same URI. Whichever so- is http://synote/recording/replay/id 11 . The replay page dis-
lution is chosen, the server, which hosts the multimedia re- plays all resources and annotations related to a multimedia
sources, is responsible to provide both audio-visual and RDF resource (Figure 2), so we use it as the HTML representation
representations. for all resources and annotations, including media fragments
Multimedia resources could be hosted in one domain (like as explained in Section 3.
YouTube), but embedded in some HTML document (also The URI of the replay page replay/id is not intuitive to
called “replay page” or “landing page”) in another domain, indicate the RDF representation of a multimedia resource.
such as wiki or blogs. So many user-generated annotations As mentioned in Section 3, we want each resource (Multime-
are saved separately from the servers which host the multi- dia, Synmark, Tag resource, etc in Figure 1) and annotation
media files. In YouTube, for example, title, keywords, com- to have a unique URI. In the semantic description of a mul-
ments and interactive transcript are displayed on the same timedia resource, URIs of other resources in Synote could be
page, i.e. the replay page, as the video content. The replay included and they have their own semantic descriptions as
page also controls the audio-visual representation of the me- well. We use resources/id to indicate the ids of resources in
dia fragments via the embedded player to provide the in- Synote as non-information resources, and resources/data/id
teractive transcript function. MFURI specification also pro- to indicate the RDF representation of each resource. Then
poses that the embedded player in the replay page should we use 303 See Other to redirect the request to different
implement the retrieval functions of “smart user agents”. documents based on content negotiation [30].
From the discussion above, we can conclude that the se- Even though we mint our own URIs for multimedia re-
mantic annotations of media fragments are mainly coming sources hosted in other domains, the media fragments are
from two types of sources. Type One: The applications, still parts of the parent resource. So it is appropriate to
which offers multimedia delivery service, publishes media use hash fragment to specify the media fragments, instead
fragments with metadata they have. The server has to make of using query or slash URIs mentioned in Section 2.1. For
sure the media fragments are dereferencable using methods Synote, we attach the fragment string in MFURI syntax at
such as content negotiation, embedded semantic annotations the back of resource id URIs, such as resources/36513#t=3,7
with HTTP streaming, etc, and they could be reused by (the resource with id 36513 must be a multimedia resource
other applications. Type Two: The applications, which 10
http://www.w3.org/TR/mediaont-10/#example2
9 11
https://bugzilla.mozilla.org/show_bug.cgi?id=648595 we use replay/id for short
in Synote). The content negotiation still works for media so ma:location is also an attribute of media fragment. We
fragment URIs. If semantic representation is requested, the simply attach the MFURI fragment string at the back of the
request will be redirected to resources/data/36513 and an location URI, which indicates the real media fragment that
RDF file containing all the media fragments and annota- should be delivered from the host server.
tions about resource 36513 is returned. If the HTML rep- As we discussed earlier, the media fragment retrieval func-
resentation is requested, the request will be redirected to tion proposed in MFURI has not been widely implemented
replay/36513#t=3,7. Then the fragment #t=3,7 will be on either client or server side, so we cannot expect YouTube
attached at the back of the real URI of the multimedia re- provides semantic description for youtu.be/abc#t=1,14 14 .
source that is to be played, for example example2/1.mp4. However, including this URI in the semantic description of
Even though the multimedia host server may not support media fragment is still safe. According to the discussion of
retrieving the media fragment example2/1.mp4#t=3,7 and httpRange-14 issue15 , as long as the server returns 200 or
the native player in the browser cannot highlight the media the 303 redirect can lead us to a 200 response, the URI is
fragments, Synote Player can control the embedded player considered dereferencable. So if a semantic agent tries to
to start playing from 3s to 7s using javascript and the corre- dereference youtu.be/abc#t=1,14 or any other multimedia
sponding annotations are highlighted straight away. In this files with hash fragment attached, the server can at least
way, the audio-visual representation of the media fragment return 200 because youtu.be/abc does exist. Maybe the se-
is derived based on the HTML representation. One problem mantic description is not provided by the server, but it will
about this solution is that different browsers have their own not return 4XX or 5XX errors, which means nothing can be
fragment precedence during redirects12 and some of them determined about the URI. This is also another reason we
do not attach the fragment in request URI to the redirected choose hash URIs to denote media fragment. We can attach
URI. Then the replay page will not be able to highlight the query ?t=1,3 or slash /t=1,3 at the back of the multime-
media fragment on opening the page. dia file example.com/1.ogv to indicate the media fragment.
But we cannot control whether the query and slash is valid
on the server and it is quite likely that 4XX or 5XX error
Listing 1: Description for media fragment will be returned. The hash fragment will not be passed to
@prefix ma:.
the server in HTTP header of the request, unless the smart
@prefix <#> .
@prefix dc:.
agent embeds it in HTTP Range header by implementing
@prefix dbpedia:. the proposed functions in MFURI specification [23].
@prefix oac:. This solution gets around the problem that most multime-
@prefix lode:. dia host and user agents have not implemented the proposed
@prefix rdfs: . functions in MFURI specification, but we still want to pub-
<> a ma:MediaResource; lish media fragments with annotations generated elsewhere.
a oac:Target;
This solution is still compatible with the multimedia host ap-
ma:hasFragment :t=1,14;
rdfs:seeAlso ;
plications if the functions in MFURI are implemented later.
rdfs:isDefinedBy ;
dc:title "Tim Berners-Lee:The next Web of open,linked data"; 4.1.2 Describing Resources and Annotations
rdfs:label "Tim Berners-Lee:The next Web of open,linked data"; As mentioned in Section 4.1.1, each resource in Synote is
ma:hasSubtitling ;
given a URI and the semantic description about it could be
ma:locator .
:t=1,14 a ma:MediaFragment;
dereferenced. In Listing 1, resources/36636 and resources/36635
a oac:Target;
are Tag resource and resources/36535 is the Transcript re-
ma:hasKeyword ; source in Figure 1. Each annotation object defined in Fig-
ma:hasKeyword ; ure 1 is also give an unique URI. We use Open Annotation
lode:illustrate _:event1; Collaborative (OAC) vocabulary to model the general anno-
ma:description "This talk took place at Terrace Theater"; tation relationship. As can be seen in Listing 1, the multime-
ma:locator ;
dia resource is the the oac:Target in an annotation instance.
ma:isFragmentOf ;
ma:title "TED talk";
The oac:Body could be Synmark, Transcript or Presenta-
rdfs:seeAlso ;
tion resource which annotates this multimedia. OAC only
rdfs:isDefinedBy . defines the annotation relationship in a higher level, so more
_:event1 a lode:Event; specific properties are also applied in the semantic descrip-
rdfs:seeAlso ; tions. For example, we use ma:hasSubtitling to represent
lode:involvedAgent ; that a Transcript resource is the subtitle of a multimedia re-
lode:atPlace .
source. Another example is, if a Synmark resource annotates
a media fragment, we can use ma:hasKeyword to model the
Listing 1 presents a N3 format of a sample semantic descrip- relationship between the Tag resource in the Synmark and
tion13 for resources/36513 and its media fragments. We use the media fragment.
rdfs:isDefinedBy and rdfs:seeAlso to connect the HTML and In Synote, each compound resource has one or more sin-
RDF representations as suggested in [30]. OMR is used to
describe the multimedia resource and media fragments. In 14
youtu.be/abc is an HTML document and the actual video
this example, ma:locator is used to indicate the real location content embedded in this document is protected by tokens.
of the multimedia file or the service which delivers the file. YouTube does not give a public URI to its video content,
In OMR, MediaFragment is a subclass of MediaResource, and we suppose that the token can be mapped to the video
content. So we use youtu.be/abc as the indirect reference to
12
http://trac.tools.ietf.org/wg/httpbis/trac/ticket/43 the video file.
13 15
Some parts of URIs are ignored for short http://goo.gl/XC821
gle resources. This relationship is modelled using Open
Archives Initiative Object Reuse and Exchange (OAI-ORE)
vocabulary16 . The following example means resources/36633,
which is a Synmark resource, aggregates resources/36634,
which is a Tag resource.
ore:isAggregatedBy .
We also consider the permission control over private re-
sources. If the requested resource or annotation is private
or does not exist, our server will return 404 Not Found. We
could use 403 Forbidden for private resources, but we do not
want to expose to the client that these resources are private
(see Section 10.4.4 of [11]).
There is a problem to align the legacy multimedia meta-
data in Type One with the vocabulary we use to describe
multimedia resources [13]. This problem is not critical to
Synote because we reuse the vocabularies, which have been
accepted by major communities, and we are not creating a Figure 3: Edit Synmark and Transcript in Synote
vocabulary ourselves which no one else uses. Besides, we Player
have seen some major progress in OMR, which provides the
mapping between OMR and other vocabularies.
so this media fragment from YouTube is also linked to Tim
4.1.3 Interlinking Methods and Terrace Theater. In addition, once the Synmark Note
Synote references the URIs of images and audio-visual re- resource with RDFa has been saved into Synote’s database,
sources from another domain, and they are published with this change can be immediately spotted in the dereferenced
other resources and annotations in Synote. This could be semantic description for media fragments.
perceived as “interlinking”. However, the real value of pub- Ninsuna has proposed a way to embed temporal metadata
lishing media fragments is that they could be reused in dif- represented by Media Fragment URIs and its description in
ferent context, especially some domain specific areas. The RDF format in a WebVTT file20 . Synote also uses WebVTT
media fragments published in Type One are usually linked to as the format for the transcript, so we can use this method in
low level metadata about the multimedia resource, such as a similar way as RDFa editor in Synmark. When users add
framerate, subtitling, etc. To enrich media fragments with RDFa or pure RDF triples in the cue body of a WebVTT
domain specific knowledge, we need to think about how the file, we can extract the triples and add them in the semantic
user-generated annotations could be further linked to other description of media fragments. Furthermore, this solution
datasets in the linked data cloud. in the future could be used as embedding structured data in
We have been inspired by the work of RDFaCE17 , which is the tracks of multimedia files and the objects will become
a RDFa Content Editor based on rich text editor TinyMCE18 . searchable by specialised search engines [41].
In RDFaCE, users could manually enrich the text by adding
RDFa to the content in a user-friendly manner. RDFaCE 4.1.4 Summary
can also connect to external semantic indexing and anno- The sources of annotations for media fragments can be di-
tation services, such as Sindice [40] and Open Calais19 , to vided into two categories. The multimedia server can embed
automatically generate RDFa for the content. In Synote, annotations in the multimedia file, or save them separately
when a Synmark is created in Synote Player, rich text or on the server but delivered using Web standards, such as
HTML could be added into the Note resource (Figure 3). So HTTP streaming. The multimedia sharing applications fo-
we can allow users to manually add or automatically gener- cus more on user-generated annotations, which are saved ex-
ate RDFa for the Note resource in Synmark. The content of ternally to the multimedia file and multimedia host server.
Note resource will be connected to the media fragment using It has been stated in [38] that “enabling the addressing of
ma:description property as shown in Listing 1. But before media fragments ultimately creates a means to attach anno-
publishing the data, we can extract the triples from RDFa tations to media fragments”. The “annotation” here should
and add them in to the RDF file as the semantic description refer to the annotations from both sources and both of them
of the media fragment. are important. But Synote only has external annotations at-
In Listing 1, some triples have been added to the Note tached to media fragments, so our design mainly focuses on
resource as RDFa to describe the event of the talking us- the annotations in Type Two.
ing Linking Open Descriptions of Events (LODE) [32]. The Our design solves the key issues of iM when deploying
lode:involvedAgent of this media fragment is Tim Berners Synote into the linked data cloud. We totally reuse existing
Lee and it took place at Terrace Theater. Then the fragment vocabularies to describe resources and annotations, and we
“#t=1,14” is further linked to Tim and Terrace Theater. As did not develop any vocabulary of our own. The current
the location of the media fragment is youtu.be/abc#t=1,14, vocabularies are enough for us to publish our resources and
16 annotations, and make sure that all necessary information
http://www.openarchives.org/ore/
17 are included. To describe the annotations about multime-
http://aksw.org/Projects/RDFaCE
18 dia resources and media fragments, it is better to use more
http://www.tinymce.com/
19 20
http://www.opencalais.com/ http://ninsuna.elis.ugent.be/node/39
crodata. On the other hand, we use the existing vocabular-
ies in schema.org and the semantic search engine on index-
ing the Synote Player page can also find more information
through “itemid”. This design can also make full use of the
existing permission control system in Synote as the private
resource will not be displayed in Synote Player nor included
in the RDF. As the RESTful API generates RDF at runtime
from database, user-generated annotations will be immedi-
ately published. The information of start and end time in
the media fragment string is saved in Synpoint (Figure 1),
so they could be easily retrieved from the business logic of
Synote to build the fragment string in MFURI syntax.
The problem of this design is that no SPARQL endpoint
is provided by Linked Data Wrapper and Rich Snippet, so
it is difficult to execute complex queries over the datasets.
Figure 4: Publishing Patterns Used in Synote That is why we are still expecting to include RDB-to-RDF
service as a complement pattern for Synote. Another possi-
ble solution is writing a RESTful API. Users can send the
specific properties instead of general annotation vocabulary, query data through the API and the server returns the se-
because the semantic agents can clearly know different as- mantic description as the response. Another problem is we
pects of media fragments and thus more efficiently tailor the only created several demo recordings with media fragments
search and reasoning results. as a proof of concept, but it is still unclear if this solution is
We try to make the media fragment publishing compatible efficient for large datasets with millions of media fragments.
with current user experience. Users in Synote can still create
Synmarks and transcripts without knowing that the data 4.3 Online Presence for Media Fragments
has been published with media fragments and interlinked The ultimate goal of this work is enhancing the online
with other datasets. However, the interlinking methods are presence of media fragments not only for semantic Web
still not very user-friendly as the knowledge about RDFa or agents, but also traditional search engines. Using Microdata
semantic Web are required to create Synmark Note. Some to describe media fragments and publish semantic descrip-
(semi-)automated methods of interlinking generation need tions as RDF only partially solve this problem. One impor-
to be considered in the future work. tant reason is that even though Synote, as well as other simi-
lar applications, provides synchronised annotations, they are
4.2 Choosing Publishing Patterns loaded together with the whole multimedia resource on the
same physical page. This is reasonable to provide an interac-
This section will introduce how the design of iM in the
tive experience for users. Other than Synote, TED Talks21
previous section is implemented by realising several linked
and YouTube interactive transcript allow users to click on
data publishing patterns.
the transcript block and the embedded media player on the
We want to build an extra layer on top of the existing
page will start playing from that time point.
Synote, so that non-RDF data could be published as RDF
This user-friendly function is not search-engine-friendly
data. But migrating the data store from the current re-
for two reasons. Firstly, most search engines only fetch pages
lational database to RDF store would result in the major
as the direct response from the server. So any dynamically
redeveloping of the whole system. Periodically dumping
generated content on the client-side is ignored. Both Syn-
RDF triples from the database and serving them via a Web
ote Player and YouTube load the transcript by Ajax, but
server could work, but we want to publish the data in real-
TED generates transcript by server-side script, thus does
time. So the patterns left for Synote are RDB-to-RDF ser-
not have this problem. Secondly, annotations related to me-
vice, Linked Data Wrapper and Rich Snippets [34]. Many
dia fragments usually share the same replay page with the
tools are designed for RDB-to-RDF service, such as D2R
whole multimedia resource and there is only one URI for
server [7], OpenLink Virtuoso [10] and Triplify [5]. These
the whole page in the search engine index. So it is impos-
tools often require developers to provide a mapping between
sible to link the keywords to a specific media fragment in
database schema and the vocabularies. RDB-to-RDF ser-
the search results and users still have to manually find the
vice only shares the database from the existing application,
keyword on the replay page. As the use of schema.org is
thus we do not need to change Synote at all. But implement-
based on the HTML and search engine indexing infrastruc-
ing access control over RDB-to-RDF service is not straight
ture, we need to make the Web pages, which contain media
forward. Besides, some functions are difficult to be imple-
fragments, search-friendly. One solution is slicing one page
mented such as the extraction of RDFa for interlinking (Sec-
into different pages according to media fragments, but the
tion 4.1.3).
interactive experience will be lost in that users have to open
Our decision is implementing a RESTful API, i.e. Linked
another page to play another media fragment.
Data Wrapper, with only HTTP GET method for the deref-
Google has developed a framework to crawl Ajax applica-
erencing of RDF representation and we also embed Micro-
tions22 . If the token “#!” is included in the original URL,
data in the original Synote Player page using vocabularies
the crawler will know that this page contains Ajax con-
defined in schema.org to improve the online presence of the
tent and in the request URL, “#!” will be replaced by “ es-
published resources (Figure 4). On the one hand, we em-
21
bed the URIs, which could be dereferenced through the API, http://www.ted.com/talks
22
into the Synote Player page using “itemid” attribute in Mi- http://goo.gl/dPc81
Figure 5: The model to improve media fragment presence based on Google crawler
caped fragment =” (Google calls it “ugly URL”). On re-
ceiving this “Ugly URL” request, the server can return the
snapshot page after the dynamic information is fully gen-
erated by javascript. The content in snapshot page will be
indexed in for the original URL. We design a model on the
Synote server to use this framework to index individual snap-
shot pages for each media fragment. Figure 5 explains how
this model could be used to index media fragments. The
returned page in step 4 only contains keywords related to
fragment “#t=3,7”. In the Google index, the original me-
dia fragment URLs are associated with the snapshot page.
So what Google actually indexed is the URL of the replay
page with MFURI syntax attached. Step 8 still returns Figure 6: 303 redirect and content negotiation for
the whole page, but in step 9, we control the embedded media fragment dereferencing
player to play the fragment from 3s to 7s using javascript
(see Section 4.1.1) and the corresponding annotations are
highlighted. But in IE and Safari, the fragment is missing.
highlighted straight away. This design not only makes sure
We have generated a sitemap containing all URIs of re-
media fragments are indexed precisely with the keywords re-
sources, media fragments and annotations and submit it to
lated to it, but also preserves the existing user interface and
Sindice. Figure 7 shows the semantic description for media
the interactive experience. This is another reason we choose
fragment resources/36513#t=00:00:01.000,00:00:14.000 in
MFURI syntax as it could be applied to Google’s Ajax ap-
Sigma. We can see that the triples (lode:illustrate property)
plication crawling framework with slight modification.
embedded in Synmark has been included in the semantic de-
scription. In addition, we can check the object URI (the URI
of the TED talk page) of lode:illustrate property in Sigma.
5. EVALUATION Figure 8 shows that the lode:involvedAgent and lode:atPlace
This section will present the evaluation results of media has been included in the description of the URI from the
fragment publishing in Synote. The evaluation mainly shows RDF published in Synote (the “source 1” in Figure 8). So
the following aspects according to the discussion in Sec- the media fragments published in Synote have been success-
tion 4: (1) Media fragments are presented using MFURI fully indexed by semantic search engine and interlinked with
syntax and Synote provides both HTML and RDF repre- other datasets through user-generated annotations.
sentation of the media fragments. (2) The published media We also submitted the sitemap containing the URIs like
fragments can be indexed by semantic search engines (we use replay/1#!t=3,7 to Google for indexing. To have a quick
Sindice and Sig.ma [39] in this evaluation) and interlinked to evaluation of what Googlebot fetches from these URIs, we
other datasets. (3) Google can index media fragments and submit several URIs via Google Web Master Tools23 . The
the Microdata can be recognised. Figure 6 shows how we use result shows that on fetching this URL,
303 redirect and content negotiation to dereference HTML
and RDF representations of the media fragment as proposed recording/replay/36513#!t=00:00:01.000,00:00:14.000
in Section 4.1.1. When asking for application/rdf+xml, the
the snapshot page is returned with annotations only related
returned RDF file will include the semantic descriptions for
to fragment “#t=00:00:01.000,00:00:14.000”. The Micro-
all its media fragments. The triples extracted from RDFa
data in the snapshot page can be extracted by Live Mi-
in Synmark note resource will also be included in this file as
crodata24 and Linter Structured Data25 . If the keyword
shown in Listing 1. For HTML representation, if the URI
23
is entered directly in the address bar, some browsers, such http://www.google.com/webmasters/tools/
24
as Firefox and Google Chrome, will attach the hash frag- http://foolip.org/microdatajs/live/
25
ment after the 303 redirect and the media fragment can be http://linter.structured-data.org/
both semantic and traditional search engines, we implement
multiple linked data publishing patterns. We also design
a model to let Google index media fragments enriched by
vocabularies in schema.org. The evaluation shows that re-
quirements listed in Section 3 have been satisfied and the ex-
isting Synote still keeps its architecture and user experience.
The approach of this work can be further applied to other
similar applications, and the solutions could be integrated
with multimedia delivery services when the functions pro-
posed in MFURI are implemented on the client and server.
In the future, we need to consider other aspects of search
engine optimisation to improve the online presence of media
Figure 7: The view of semantic description of media fragments. We also need to think about applying this solu-
fragments from Sigma tion to other major search engines such as Yahoo! and Bing.
We recommend more media fragments to be published using
the solutions described in this paper in order to benefit the
indexing of multimedia resources.
7. REFERENCES
[1] B. Adida and M. Birbeck. RDFa Primer, Oct. 2008.
http://www.w3.org/TR/xhtml-rdfa-primer/.
[2] Adobe Corp. XMP Specification, 2005.
[3] R. Arndt, R. Troncy, S. Staab, L. Hardman, and
M. Vacura. COMM: Designing a Well-Founded
Multimedia Ontology for the Web. In International
Semantic Web Conference, volume 4825 of Lecture
Notes in Computer Science, pages 30–43. Springer
Figure 8: Interlinking with other datasets Berlin Heidelberg, 2007.
[4] S. Auer, C. Bizer, G. Kobilarov, J. Lehmann,
R. Cyganiak, and Z. Ives. DBpedia: A Nucleus for a
“Terrace Theater”, for example, is searched in Google, the Web of Open Data. In K. Aberer, K.-S. Choi, N. Noy,
snapshot page can be successfully found in the search re- D. Allemang, K.-I. Lee, L. Nixon, J. Golbeck, P. Mika,
sults instead of the whole replay page. When we click the D. Maynard, R. Mizoguchi, G. Schreiber, and
link in search results, the Synote Player page in Figure 2 will P. Cudré-Mauroux, editors, The Semantic Web,
be opened and the button in top-left corner indicates that a volume 4825 of LNCS, pages 722–735. Springer, 2007.
media fragment is requested. The embedded player will start
[5] S. Auer, S. Dietzold, J. Lehmann, S. Hellmann, and
playing the media fragment “#t=00:00:01.000,00:00:14.000”
D. Aumueller. Triplify: light-weight linked data
when the button is clicked and the related annotation on the
publication from relational databases. International
right column will be highlighted. The screencast26 and the
World Wide Web Conference, pages 621–630, 2009.
live demo of this evaluation27 are available online.
[6] T. Berners-Lee. Linked Data - Design Issues, 2006.
[7] C. Bizer and R. Cyganiak. D2R Server - Publishing
6. CONCLUSION Relational Databases on the Semantic Web. In
Media fragments are important for the efficient indexing Proceedings of the 5th International Semantic Web
and searching of multimedia resources. In this paper, we use Conference. Springer, 2006.
Synote as the target application to show how media frag- [8] J. Bormans and K. Hill. MPEG-21 Overview v.5, 2002.
ments could be published with user-generated annotations [9] D. V. Deursen, W. V. Lancker, W. D. Neve,
following linked data principles. We argue that the semantic T. Paridaens, E. Mannens, and R. V. D. Walle.
descriptions of media fragments can come from two types of NinSuna: a fully integrated platform for
sources. Currently, there are plenty of data in Type Two, format-independent multimedia content adaptation
which is generated by multimedia sharing and annotating and delivery using Semantic Web technologies.
applications in Web 2.0 and they should be published as Multimedia Tools and Applications, 46(2):371–398,
linked data. To publish media fragments and annotations 2009.
in Synote, we use content negotiation to serve both audio- [10] O. Erling and I. Mikhailov. RDF Support in the
visual and semantic representation for media fragments. To Virtuoso DBMS. Proceedings of the 1st Conference on
describe multimedia resources and annotations, we reuse the Social Semantic Web CSSW, 221(ISBN
existing vocabularies online instead of creating new vocab- 978-3-88579-207-9):59–68, 2007.
ularies. We also suggest to embed RDFa in annotations to
[11] R. Fielding, J. Gettys, J. Mogul, H. Frystyk,
promote the interlinking with other datasets.
L. Masinter, P. Leach, and T. Berners-Lee. Hypertext
The design builds an extra layer on top of the existing Syn-
Transfer Protocol – HTTP/1.1. RFC 2616, June 1999.
ote application. In order to efficiently publish the data for
[12] B. Haslhofer, W. Jochum, R. King, C. Sadilek, and
26
http://goo.gl/4zl1V K. Schellner. The LEMO annotation framework:
27
http://linkeddata.synote.org/synote/ weaving multimedia annotations with the web.
International Journal on Digital Libraries, Semantic Web. W3C Note, 49(681):1–15, 2008.
10(1):15–32, 2009. [31] J. F. Sequeda and O. Corcho. Linked Stream Data: A
[13] M. Hausenblas, R. Troncy, T. Bürger, and Position Paper. In K. Taylor, A. Ayyagari, and D. D.
Y. Raimond. Interlinking Multimedia: How to Apply Roure, editors, International Workshop on Semantic
Linked Data Principles to Multimedia Fragments. Sensor Networks 2009, volume 522, pages 148–157.
WWW 2009 Workshop Linked Data on the Web CEUR, Kerry Taylor and Arun Ayyagari and David
LDOW2009, 2009. De Roure, 2009.
[14] T. Heath and C. Bizer. Linked Data: Evolving the [32] R. Shaw, R. Troncy, and L. Hardman. Lode: Linking
Web into a Global Data Space, volume 1. Morgan & open descriptions of events. The Semantic Web,
Claypool, 2011. 5926(Lecture Notes in Computer Science):153–167,
[15] I. Hickson. HTML Microdata, Feb. 2012. 2009.
http://dev.w3.org/html5/md/. [33] T. Steiner. SemWebVid - Making Video a First Class
[16] I. Hickson. WebVTT, Feb. 2012. Semantic Web Citizen and a First Class Web
http://dev.w3.org/html5/webvtt/. Bourgeois. In Proceedings of the 9th International
[17] P. Hoschka. An introduction to the synchronized Semantic Web Conferene ISWC 2010, pages 3–6.
multimedia integration language. Multimedia, IEEE, Springer, 2010.
5(4):84 –88, oct-dec 1998. [34] T. Steiner, R. Troncy, and M. Hausenblas. How
[18] I. Jacobs and N. Walsh. Architecture of the World Google is using Linked Data Today and Vision For
Wide Web. arq Architectural Research Quarterly, Tomorrow. In S. Auer, S. Decker, and M. Hauswirth,
2(01):1–56, 2004. editors, Linked Data in the Future Internet 2010,
[19] J. Kahan, M. Koivunen, E. Prud’Hommeaux, and Ghent,Belgium, 2010.
R. Swick. Annotea: An open rdf infraestructure for [35] Technical Standardization Committee on AV & IT
shared web annotations. In Proceedings of the 10th Storage Systems and Equipment. Exchangeable Image
international conference on World Wide Web, pages File Format for Digital Still Cameras: EXIF Version
623–632. ACM Press, 2001. 2.2, 2002.
[20] D. Lambert and H. Q. Yu. Linked Data Based Video [36] R. Troncy. Bringing the IPTC News Architecture into
Annotation and Browsing for Distance Learning. In the Semantic Web. Lecture Notes In Computer
The 2nd International Workshop on Semantic Web Science Vol 5318, 5318:483, 2008.
Applications in Higher Education (SemHE’10), Nov. [37] R. Troncy, L. Hardman, and J. V. Ossenbruggen.
2010. Identifying spatial and temporal media fragments on
[21] Y. Li, M. Wald, and G. Wills. Interlinking multimedia the web. W3C Video on the Web Workshop, pages
annotations. In ACM WebSci’11, pages 1–4, June 4–9, 2007.
2011. WebSci Conference 2011. [38] R. Troncy, E. Mannens, S. Pfeiffer, and D. V.
[22] Y. Li, M. Wald, G. Wills, S. Khoja, D. Millard, Deursen. Media fragments URI 1.0 (basic), Mar. 2012.
J. Kajaba, P. Singh, and L. Gilbert. Synote: http://www.w3.org/TR/media-frags/.
development of a web-based tool for synchronized [39] G. Tummarello, R. Cyganiak, M. Catasta,
annotations. New Review of Hypermedia and S. Danielczyk, R. Delbru, and S. Decker. Sig.ma: Live
Multimedia, 17(3):295–312, 2011. views on the web of data. Web Semantics: Science,
[23] E. Mannens, D. Van Deursen, R. Troncy, S. Pfeiffer, Services and Agents on the World Wide Web, 8(4):355
C. Parker, Y. Lafon, J. Jansen, M. Hausenblas, and – 364, 2010.
R. Van de Walle. A uri-based approach for addressing [40] G. Tummarello, R. Delbru, and E. Oren. Sindice.com:
fragments of media resources on the web. Multimedia Weaving the open linked data. In K. Aberer, K.-S.
Tools and Applications, pages 1–25. Choi, N. Noy, D. Allemang, K.-I. Lee, L. Nixon,
10.1007/s11042-010-0683-z. J. Golbeck, P. Mika, D. Maynard, R. Mizoguchi,
[24] F. Manola and E. Miller. RDF Primer, 2004. G. Schreiber, and P. Cudré-Mauroux, editors, The
[25] J. M. Martinez. MPEG-7 Overview, 2004. Semantic Web, volume 4825 of Lecture Notes in
[26] S. Pfeiffer, C. D. Parker, and A. T. Pang. Specifying Computer Science, pages 552–565. Springer Berlin /
Time Intervals in URI Queries and Fragments of Heidelberg, 2007. 10.1007/978-3-540-76298-0 40.
Time-Based Web Resources, 2005. [41] M. Verwaest, D. Van Deursen, R. De Sutter,
[27] E. Prud’hommeaux and A. Seaborne. SPARQL Query N. Laukens, D. Van Rijsselbergen, E. Mannens, and
Language for RDF, 2008. R. Van de Walle. Television on a tablet : how tv will
look like in the future and how to deal with
[28] C. Saathoff and A. Scherp. Unlocking the semantics of
professionally produced content. In Second W3C web
multimedia presentations in the web with the
and TV workshop : papers, page 8. W3C, 2011.
multimedia metadata ontology. Proceedings of the
19th international conference on World wide web [42] J. Waitelonis. Augmenting video search with linked
WWW 10, page 831, 2010. open data. In Proceedings of International Conference
on Semantic Systems (i-semantics), September 2-4,
[29] F. Sasaki, T. Bürger, J. Söderberg, V. Malaisé,
2009, Graz, Austria. Verlag der TU Graz, Austria,
F. Stegmaier, and W. Lee. Ontology for media
2009.
resource 1.0, Feb. 2012.
http://www.w3.org/TR/mediaont-10/.
[30] L. Sauermann and R. Cyganiak. Cool URIs for the