Video has become a rst class citizen on the Web with broad support in all common Web browsers. Where with structured mark-up on webpages we have made the vision of the Web of Data a reality, in this paper, we propose a new vision that we name the Web(VTT) of Data, alongside with concrete steps to realize this vision. It is based on the evolving standards WebVTT for adding timed text tracks to videos and JSON-LD, a JSON-based format to serialize Linked Data. Just like the Web of Data that is based on the relationships among structured data, the Web(VTT) of Data is based on relationships among videos based on WebVTT les, which we use as Web-native spatiotemporal Linked Data containers with JSON-LD payloads. In a rst step, we provide necessary background information on the technologies we use. In a second step, we perform a largescale analysis of the 148 terabyte size Common Crawl corpus in order to get a better understanding of the status quo of Web video deployment and address the challenge of integrating the detected videos in the Common Crawl corpus into the Web(VTT) of Data. In a third step, we open-source an online video annotation creation and consumption tool, targeted at videos not contained in the Common Crawl corpus and for integrating future video creations, allowing for weaving the Web(VTT) of Data tighter, video by video.
H.5.1 [Multimedia Information Systems]: Video
From <OBJECT> to <video>
In the \ancient" times of HTML 4.01 [
While the above paragraph is provocatively written in
past tense and while the <object> tag is still part of both
the current World Wide Web Consortium (W3C) HTML5
speci cation [
We are motivated by the vision of a Web(VTT) of Data, a global network of videos and connected content that is based on relationships among videos based on WebVTT les, which we use as Web-native spatiotemporal containers of Linked Data with JSON-LD payloads. The paper makes four contributions, including transparent code and data. i) Large-Scale Common Crawl study of the state of Web video: we have examined the 148 terabyte size Common Crawl corpus and determined statistics on the usage of the <video>, <track>, and <source> tags and their implications for Linked Data. ii) WebVTT conversion to RDF-based Linked Data: we propose a general conversion process for \triplifying" existing WebVTT, i.e., for turning WebVTT into a specialized concrete syntax of RDF. This process is implemented in form of an online conversion tool. iii) Online video annotation format and editor: we have created an online video annotation format and an editor prototype implementing it that serves for the creation and consumption of semantic spatiotemporal video annotations turning videos into Linked Data. iv) Data and code: source code and data are available.
The remainder of the paper is structured as follows. Section 2 provides an overview of the enabling technologies that we require for our approach. Section 3 describes a largescale study of the state of Web video deployment based on the Common Crawl corpus. Section 4 deals with the integration of existing videos into the Web(VTT) of Data through a tool called LinkedVTT. Section 5 presents an online video annotation format and an editor that implements this format. We look at related work in Section 6 and close with conclusions and an outlook on future work in Section 7. 2.
In this section, we lay the foundations of the set of technologies that enable our vision of the Web(VTT) of Data. The <track> tag allows authors to specify explicit external timed text tracks for videos. With the <source> tag, authors can specify multiple alternative media resources for a video. Both do not represent anything on their own and are only meaningful as direct child nodes of a <video> tag.
The Web Video Text Tracks format (WebVTT, [
The JavaScript Object Notation5 (JSON) is a (despite the
name) language-independent textual syntax for serializing
objects, arrays, numbers, strings, booleans, and null. Linked
Data [
Listing 1: Example WebVTT le with two cues
subtitles
captions descriptions chapters metadata
Transcription or translation of speech, suitable for when sound is available but not understood. Overlaid on the video. Transcription or translation of the dialogue, sound e ects, and other relevant audio information, suitable for when sound is unavailable or not clearly audible. Overlaid on the video; labeled as appropriate for the hard-of-hearing. Textual descriptions of the video component of the media resource, intended for audio synthesis when the visual component is obscured, unavailable, or unusable. Synthesized as audio. Chapter titles, intended to be used for navigating the media resource. Displayed as an interactive (potentially nested) list in the user agent's interface. Metadata intended for use from script context. Not displayed by user agent.
Media Fragments URI [
The Ontology for Media Resources [
Part of the objectives behind the Web(VTT) of Data is to create a truly interconnected global network of and between videos containing Linked Data pointers to related content of all sorts, where diverse views are not ltered by the network bubble, but where serendipitously new views can be discovered by taking untrodden Linked Data paths. In order to get there, we have conducted a large-scale study based on the Common Crawl corpus to get a better understanding of the status quo of Web video and timed text track deployment. 3.1
The Common Crawl Foundation7 is a non-pro t
organization founded in 2008 by Gil Elbaz. Its objective is to
democratize access to Web information by producing and
maintaining an open repository of Web crawl data that is
universally accessible and analyzable. All Common Crawl data
is stored on Amazon Simple Storage Service (Amazon S3)8
and accessible to anyone via Amazon Elastic Compute Cloud
(Amazon EC2),9 allowing the data to be downloaded in
bulk, as well as directly be accessed for map-reduce
processing in EC2. The, at time of writing, latest dataset was
collected at the end of 2013, contains approximately 2.3 billion
webpages and is 148 terabyte in size [
We have analyzed the entire 148 terabytes of crawl data
using an Elastic Compute Cloud job whose code was made
available as open-source.12 Rather than parse each
document as HTML, we have tested them for the regular
expression <video[ˆ>]*>(.*?)</video>, an approach that also
in previous experiments proved very e cient [
From all 2,963,766 <video> tags, only 1,456 ( 0:049%) had a <track> child node. Upon closer examination of the kinds of these 1,456 <track> nodes (see Table 1 for an explanation of the various kinds), we saw that the overwhelming majority are unsurprisingly used for subtitles or captions. Almost no chapter usage was detected and neither metadata nor description usage at all. The full details can be seen in Table 2. Looking at the languages used in the captions and subtitles, these were almost exclusively English and French, as can be seen in Table 3. The track labels listed in Table 4 indeed con rm this observation. In case of multiple tracks for one video, one track can be marked as the default track. This happens through a boolean attribute,15 whose value either needs to be the empty string or the attribute's name, which is \default" in the concrete case. Table 5 shows that this was used correctly in almost all cases. When we tried to determine the MIME type of the actual text tracks, we relied on the le extension of the values given in the <track src> attributes. As a signi cant amount of text tracks seems to be dynam11Common Crawl PageRank code: https://github.com/ commoncrawl/commoncrawl-crawler/tree/master/src/ org/commoncrawl/service/pagerank 12EC2 job: https://github.com/tomayac/postdoc/blob/ master/demos/warczenschwein/ 132,963,766 <video> tags: https://drive.google.com/ file/d/0B9LlSNwL2H8YdWVIQmJDaE81UEk 14Amazon EMR: http://aws.amazon.com/ elasticmapreduce/ 15HTML boolean attributes: http://www.whatwg.org/ specs/web-apps/current-work/#boolean-attributes ically generated on-the- y|and thus had no le extension but a video identi er in the URL instead|we used an approximation to check if some part of the URL matched the regular expression /\bvtt\b/gi. Based on this approximation, a little over half of all text tracks are in WebVTT format with the extension .vtt or rarely .webvtt. The predecessor SubRip le format16 can still be encountered in about a quarter of all text tracks. In between SubRip and WebVTT, a format originally called WebSRT (Web Subtitle Resource Tracks) existed that shared the .srt le extension. The full distribution details are available in Table 6. Looking at the number of text tracks per video, almost all videos had only exactly one text track rather than multiple, as detailed in Table 7, meaning that the broad majority of all videos are subtitled or captioned in only one language. 16SubRip le format: http://www.matroska.org/ technical/specs/subtitles/srt.html
As in Section 5 we will report on ways to make
semantic statements about videos on the Web, we have
additionally compiled some video statistics. Unlike with images on
the Web, where semantic statements in Resource
Description Framework (RDF) can be made based on the image's
URL [
The biggest issue with this practice of putting multiple
sources is that rather than having one unique identi er (URL)
per video, there can be multiple identi ers. Listing 2 shows
a minimal example. Unless one repeats all statements for
each source, there will always remain unclear sources
without structured data. We note that a video in encoding A
and the \same" video in encoding B may not be marked
as <owl:sameAs>, because statements about the encoding
format of one video do not apply to the other, the
identity symmetry condition would thus be violated. In
practice, a solution similar to specifying canonical URLs in Web
search [
org/licenses/by-sa/3.0/">
Creative Commons Attribution Share-Alike 3.0 </a> </div> <div about="kitten.mp4"> <video> <source src="kitten.mp4"/> <source src="kitten.webm"/> </video> <a rel="license" href="http://creativecommons.
org/licenses/by-sa/3.0/">
Creative Commons Attribution Share-Alike 3.0 </a> </div> Listing 2: Specifying a license for an image and attempt to do the same for a video with two sources (the license of kitten.webm stays unclear)
The WebVTT speci cation [
A WebVTT le de nes a set of cues, which are described
by a pair of timestamps and a payload. In other words, each
cue is an annotation of the video, associating a temporal
video fragment to the payload, delimited by the two
timestamps. As there is a standard way of identifying temporal
and spatial video fragments with a URI [
WebVTT is not limited to textual timed text tracks. As
Table 1 details, the HTML5 <track> tag supports di
erent kinds of tracks, one of them being metadata, a track
designed for machine rather than human consumption.
Although it was shown in Subsection 3.3 that there is no
measurable evidence of use for this kind of track yet|which
is understandable given that the technology is still under
development|we propose that JSON data is a good
candidate for cues of such tracks. JSON has a textual syntax that
is easy to author and easy to process in a Web browser and
elsewhere. Furthermore, JSON-LD [
In addition to the cues, WebVTT les can contain metadata headers described as key-value pairs. While the WebVTT speci cation de nes a number of metadata headers, it leaves it open for extensions. We propose three extended metadata headers listed below. Most WebVTT currently does not contain these metadata headers, but we argue that they allow for an easy transition from plain WebVTT to Linked Data WebVTT, just like JSON-LD makes it easy to turn plain JSON into Linked Data by adding a @context property. Further more, other metadata headers will be evaluated against the JSON-LD context, and can produce additional triples with the WebVTT le as its subject. @base Sets the base URI used for resolving relative URIs. This applies to any relative URIs that would be found in the JSON-LD descriptions, but also to generate URIs for cues based on their identi ers. It defaults to the URI of the WebVTT le. @context This key can be used multiple times; each value is the URI of a JSON-LD context that should be used to interpret the JSON payloads in the WebVTT le. @video Sets the URI for the video for generating media fragment URIs. If not present, the video URI must be provided externally, e.g., the <video src> attribute of the video containing the WebVTT track. This metadata header is a direct response to an issue that we have outlined in Subsection 3.5. 4.4
Given the currently rather manageable amount of videos
with captions or subtitles as outlined in Subsection 3.3,
approaches for the automated semantic lifting based on timed
text track data are feasible. These approaches extract the
transcribed text snippets from cues and either convert them
into one consistent block of text or treat each text snippet in
isolation before applying named entity extraction on them.
Representative examples based on this idea are [
Complementary to the conversion process presented in Section 4, in this section we focus on facilitating the online creation and consumption of metadata tracks for future video creations and videos not contained in the Common Crawl corpus. We begin with the annotation model. 5.1
Our annotation model is the same as the one produced by the interpretation process presented above. Annotations take the form of RDF statements (subject-predicate-object), where the subject is any temporal or spatiotemporal fragment of the video, identi ed by the corresponding Media Fragments URI. They are encoded as TextTrackCues with JSON-LD payloads such as the one shown in Listing 3. A dedicated data context de nes their semantics. 5.2
We have implemented this annotation model in form of an online demonstrator prototype. The demonstrator interprets the existing metadata track for a video and reacts on annotations when the currentTime of the media resource matches the startTime or endTime of a cue. We call existing annotations Read annotations. Users can add Write annotations by creating new TextTrackCues at the desired start and end times and by providing their JSON-LD payloads. The editor facilitates this task through a graphical user interface, abstracting the underlying details. Figure 1 shows a screenshot of the WebVTT editor. Newly generated annotations get directly interpreted and can be persistently stored locally or in the future remotely for collaborative editing. We have developed a WebVTT to JSON-LD converter, capable of transforming WebVTT metadata tracks following our annotation model into JSON-LD for the Web of Data. This allows for straight-forward local annotation creation with Semantic Web compliance upon global publication. 5.2.1
Our JSON-LD context eases common annotation tasks
by de ning the semantics of a few useful JSON properties
described below. According to this context, Listing 3 is
interpreted as in Listing 4 (RDF in JSON-LD syntax) and
Listing 5 (RDF in N-Triples syntax). More advanced
annotation tasks can be supported by extending the data context.
Plain Text Tags Annotations of type tags allow for
adding plain text tags to a media fragment. They are
interpreted as Common Tag [
linkedvtt/demonstrator-context.json", "@id": "http://ex.org/video#t
=0:0.0,0:12.0",
"tags": ["wind scene", "opening credits"],
"contributors": ["http://ex.org/sintel"]
Contributors The contributors annotation type
allows for denoting the contributors in a media fragment,
like its actors. They are interpreted as Ontology for Media
Resources [
Summary The summary annotation type allows for
summarizing a media fragment (note, not the whole video like
kind description tracks) with plain text. They are
interpreted as ma:description [
Presentation-oriented annotations|similar to temporal style sheets|do not generate RDF data, but only impact the way videos get presented.
Visual E ect Annotations of type visualEffect allow for applying visual e ects in the syntax of Cascading Style Sheets19 (CSS) to a media fragment, e.g., lters, zoom, transparency, and 2D/3D transformations and animations. Audial E ect The audialEffect annotation type allows for applying audial e ects to a media fragment. Currently, we support modifying the volume from 0 to 1.
Playback Rate The playbackRate annotation type allows for specifying the e ective playback rate of a media fragment. The playback rate is expressed as a oating point multiple or fraction of the intrinsic video speed.
HTML Overlay Via the htmlOverlay annotation type, overlays in freeform HTML code can be added to a media fragment. Examples are graphical, textual, or combined overlays that can contain links to (temporal fragments of) other videos or within the current video. 19Cascading Style Sheets: http://www.w3.org/Style/CSS/ <http://ex.org/metadata.vtt> <http://www.w3.org /1999/02/22-rdf-syntax-ns#type> <http://ex. org/VideoMetadataDataset> . <http://ex.org/metadata.vtt> <http://champin.net /2014/linkedvtt/onto#hasCue> <http://ex.org/ metadata.vtt#id=cue1> . <http://ex.org/metadata.vtt#id=cue1> <http:// champin.net/2014/linkedvtt/onto# describesFragment> <http://ex.org/video#t =0:0.0,0:12.0> . <http://ex.org/video#t=0:0.0,0:12.0> <http:// commontag.org/ns#label> "wind scene" . <http://ex.org/video#t=0:0.0,0:12.0> <http:// commontag.org/ns#label> "opening credits" . <http://ex.org/video#t=0:0.0,0:12.0> <http://www. w3.org/ns/ma-ont#hasContributor> <http://ex. org/sintel> .
Listing 5: RDF triples based on the JSON-LD code from Listing 4
In our WebVTT editor, we propose an interpretation layer capable of dealing with the herein de ned annotation types. We thus make an open world assumption by supporting a set of pre-de ned values for predicate and object listed below, and ignoring unknown ones. This permits others to extend|or even completely replace|our interpretation layer. If a TextTrackCue has a WebVTT identi er, we use it to address its annotations via the metadata track's URI and corresponding cue fragment identi er, allowing for meta annotations of annotations, e.g., to attach provenance or license information to them.
We evaluate or annotation model and related technology
stack based on a state-of-the-art hypervideo model by
Sadallah et al. [
Sadallah et al. de ne hypervideo as \interactive video-centric hypermedia document built upon audiovisual content". The authors identify three common hypervideo characteristics, namely (i) interactivity, which, e.g., can enable richer navigational possibilities, (ii) non-linearity, which allows for features like video montages, and nally (iii) enrichments that include all sorts of supplementary material besides and on top of hypervideos. The authors have examined hypervideo systems of recent years and found recurring patterns, summarized and compared to our approach in the following. Video player and controls Hypervideo systems by de nition provide one or multiple video players, however, the corresponding video controls are not necessarily exposed. X Our approach uses the (optionally customizable) default HTML5 player that includes hidable controls (Figure 1). Timeline A timeline is the spatial representation of temporally situated metadata in a video. The most common timeline pattern shows the time along the x-axis and corresponding metadata along the y-axis.
X Our approach supports temporal metadata. Customizable timeline visualizations exist20 and can be added. Textual or graphical overlay Additional textual or graphical information can be displayed in form of overlays on the video. Overlays can also serve as external or videointernal hyperlinks, referred to as hotspots.
X We realize overlays and links with htmlOverlay types. Figure 1 shows both a graphical (yellow box) and two textual overlays (red and green texts).
Textual or graphical table of contents If a video is logically separated into di erent parts, a table of contents lists these in textual or graphical form, makes them navigable, or visually summarizes them, referred to as video map. X Textual tables of contents are directly supported via WebVTT text tracks of type chapters. Graphical tables of contents can be created based thereon.
Transcript The textual document of the transcribed audiovisual content of a video allows for following along the video by reading and also serves for in-video navigation. X Subtitles and captions are natively supported by WebVTT tracks of the types subtitles and captions. Figure 1 shows active subtitles (white text).
With our annotation approach, we leverage WebVTT
metadata tracks as a means for tying semantic JSON-LD
annotations to temporal or spatiotemporal video fragments. As
each <track> tag by pure de nition is bound to exactly one
<video> tag, and as modern search engines parse and
interpret JSON-LD annotations, a unique relation of annotations
to video content is made. In consequence, related work can
be regarded under the angles of online annotation creation
and large-scale Linked Data e orts for video. Many have
combined Linked Data and video, typical examples are [
In this paper, we have introduced our vision of the Web(VTT) of Data, a global network of videos and connected content that is based on relationships among videos based on WebVTT les, which we use as Web-native spatiotemporal containers of Linked Data with JSON-LD payloads. With the recent graduation of the JSON-LD syntax as an o cial W3C Recommendation and a major search engine company23 supporting embedded JSON-LD documents in HTML documents,24 JSON-LD de nitely is here to stay. Likewise for WebVTT, which in the more recent past has been natively implemented by all major Web browser vendors, the future is bright. We combine both technologies in a fruitful way that is focused both at common Web search engines as well as at the entire Linked Data stack of technologies. Using WebVTT as a container for JSON-LD is both innovative and natural. Making commonly understood semantic statements about video fragments on the Web has become feasible thanks to Media Fragments URI, a standard that allows for applying Linked Data approaches to moving images on a temporal and spatiotemporal axis. We have organized this paper in three major steps. (i) in order to get a better understanding of the status quo of Web video deployment, we have performed a large-scale analysis of the 148 terabyte size Common Crawl corpus, (ii) we have addressed the challenge of integrating existing videos in the Common Crawl corpus into the Web(VTT) of Data by proposing a WebVTT conversion to RDF-based Linked Data, and (iii) we have open-sourced an online video annotation creation and consumption tool, targeted at videos not contained in the Common Crawl corpus and for integrating future video creations. In this paper, we have combined Big Data and Small Data. On the Big Data side, we have learned from the Common Crawl corpus which kind of timed text tracks are out there, which allowed us to propose a realistic approach to integrating it into the Web(VTT) of Data. On the Small Data side, we have implemented an online editor for the creation of semantic video annotations that can be applied video by video, so that the Web(VTT) of Data gets woven tighter and tighter with each new addition.
Future work has several dimensions. Beginning from video annotation, a rst concrete research task is to work on our editor prototype. While a lot of e orts can be put in the editor itself, far more added value is created by proposing an extension to the most well-known online video annotation stack, the Popcorn.js and PopcornMaker projects. A minimal Popcorn.js example annotation can be seen in Listing 6. Rather than storing the annotations as steps of a JavaScript program that \arti cially" need to be aligned to the corresponding parts of the video, an extension to 22PopcornMaker: https://popcorn.webmaker.org/ 23JSON-LD in Gmail: https://developers.google.com/ gmail/actions/reference/formats/json-ld 24Embedding JSON-LD in HTML Documents: http://www.w3.org/TR/json-ld/#embedding-jsonld-in-html-documents Popcorn.js could use our approach of leveraging naturally temporally aligned WebVTT cues with JSON-LD payloads for the annotations. We have been able to play video in Web browsers plugin-free for a couple of years now, the next step is adding resources to videos to make them more accessible and provide more options to the viewer. Straightforward things to do are to pro t from recent advances in machine-translation and speech recognition to evaluate the usefulness of automatically transcribed and translated captions combined with language-independent metadata annotations based on named entity extraction for providing Linked Data paths between videos no matter their original language. We have learned that with regard to HTML5 video with timed text track support, it is still early days, so at least in the short-term it will be inevitable to deal with legacy plugin-dependent content and ways to integrate it in the Web(VTT) of Data through adaptors or converters etc.
Concluding, the vision of the Web(VTT) of Data is a realistic one and all building blocks are in place. We are optimistic that by leveraging the popularity of existing tools like Popcorn.js, we can push the state of Web video forward toward an interconnected, semantic, and wall-free experience.
The research presented in this paper was partially supported by the French National Agency for Research project Spectacle En Ligne(s), project reference ANR-12-CORP-0015. <video id="video" src="http://ex.org/video.mp4"> </video> <div id="footnote-container"></div> <div id="wikipedia-container"></div> <script> // get a reference to the video var pop = Popcorn("#video"); // add a footnote from second 2 to second 6 pop.footnote({ start: 2, end: 6, text: "In this scene: George Clooney", target: "footnote-container" }); // add a reference to a Wikipedia article // from second 2 to 6 pop.wikipedia({ start: 2, end: 6, src: "http://en.wikipedia.org/wiki/George _Clooney", title: "George Clooney", target: "wikipedia-container" }); </script>
Listing 6: Popcorn.js example