=Paper=
{{Paper
|id=Vol-1352/paper7
|storemode=property
|title=LinkedCulture: Browsing Related Europeana Objects while Watching a Cultural Heritage TV Programme
|pdfUrl=https://ceur-ws.org/Vol-1352/paper7.pdf
|volume=Vol-1352
|dblpUrl=https://dblp.org/rec/conf/iui/NixonBO15
}}
==LinkedCulture: Browsing Related Europeana Objects while Watching a Cultural Heritage TV Programme==
https://ceur-ws.org/Vol-1352/paper7.pdf
LinkedCulture: browsing related Europeana objects while
watching a cultural heritage TV programme
Lyndon Nixon Lotte Belice Baltussen, Johan Oomen
MODUL University Sound and Vision
Am Kahlenberg 1 Sumatralaan 45
1190 Vienna, Austria Hilversum, Netherlands
lyndon.nixon@ modul.ac.at lbbaltussen,joomen@ beeldengeluid.nl
ABSTRACT being first choice to look up related information. As a
This short/demo paper describes LinkedCulture, a Web global trend, it has led to Forbes’ magazine to announce
based application which complements the viewing of a well “using a second screen while watching TV is the new
known Dutch cultural heritage TV program with the ability normal”2. LinkedTV’s own user trial [2] with viewers of
of viewers to explore art objects from Europeana related to TKK confirmed that, even when the viewers were older as
those in the program. is a typical demographic for such a program, there is a
significant interest in being able to explore further
INTRODUCTION information beyond what is provided by the program, as
The LinkedTV project (http://www.linkedtv.eu) believes long as it is easy enough to access and can be available also
future television must embrace new consumption patterns after the program has been viewed.
as it moves online if it is to retain relevance in a space
increasingly dominated by Web-centric offers. This In the commercial domain, there is not yet a widely
includes deeper integration with other Web content and successful application of second screen TV enrichment due
multi-screen viewing. LinkedCulture, using cultural to the excessive cost of annotating TV programming and
heritage content in Dutch made up of episodes of the TV manually preparing the enrichment. Shazam for TV
show Tussen Kunst & Kitsch (TKK) from AVROTROS (in (http://www.shazam.com/music/web/productfeatures.html?i
collaboration with Sound and Vision), is one of the d=1266), for example, focuses on a TV program as a whole,
LinkedTV scenarios implemented in the project. This e.g. actor information or episode trivia. Some demos have
short/demo paper refers briefly to TV consumption trends been made with concept-level approaches, e.g. using
and prior related work (Sec. 2) before turning to describe Mozilla Popcorn (https://popcorn.webmaker.org/), linking
the implemented demo of LinkedCulture (Sec. 3), some of terms in video subtitles to content shown in other frames
the technical aspects with a focus on how we dealt with alongside the video. These demos suffered from the lack of
describing art objects and using the Europeana API (Sec. 4) disambiguation of terms (such as “Paris”) and limited link
before concluding on future work and how demos like relevance (typically Wikipedia, a map etc.). LinkedTV
LinkedCulture can point to a new way for the public to provides an automated workflow (cf. Sec 4) with better
experience CHO collections like Europeana (Sec. 5). disambiguation of concepts referred to within audiovisual
material as well as richer linking to sets of relevant content
TRENDS AND RELATED WORK from varied online sources. We believe this provides
Households have more and more connected devices and significantly better automated results than any prior work
consumers are increasingly using devices in parallel: this is and thus also forms the basis for a usable cost-effective
clearest when it comes to viewing audiovisual solution by content owners. The rest of this paper will focus
programming on one device and browsing online content on on how this approach was used in enriching a cultural
another. A 2013 published survey [1] backed up earlier heritage TV program.
UK/US-focused surveys on “second screen” usage1 that
40% of continental Europeans were using a second device DEMONSTRATOR FOR CULTURAL HERITAGE
to follow what they were watching on TV, with Google LinkedCulture3 [3] shows the provision of complementary
information related to art objects being discussed in the TV
program. Trials with viewers [2] validated their interest in
being suggested links to other program segments where
similar objects are discussed and links to information on
1 2
http://www.forbes.com/sites/jeffbercovici/2014/07/10/using-a-
http://www.theguardian.com/technology/appsblog/2012/oct/29/soc second-screen-while-watching-tv-is-now-the-norm/
ial-tv-second-screen-research refers to 2012 surveys noting 37-
3
52% of viewers looking up information connected to the program Demo video: http://vimeo.com/108891238
Copyright held by the author(s).
similar art objects in digital collections (Europeana), which
they can explore while pausing or completing their current
viewing, on the same screen or - casting the TV program to
another screen (e.g. from tablet to a TV) – alongside
viewing. The application allows also for bookmarking so
that viewing can continue but the viewer can easily refer
back to the content they were interested in afterwards.
Figure 3. Viewing an information card for a concept
In the next screenshot, other examples of silver tea jars in
Dutch collections can be examined (Fig. 4). Providing the
most relevant related Europeana Cultural Heritage Objects
(CHOs) is the subject of the implementation of a dedicated
Europeana API wrapper, discussed in the next section.
Figure 1. Opening screen of LinkedCulture with the option to start
viewing a TKK episode segment4
After accessing the LinkedCulture application (Fig. 1) the
viewer can explore past episodes (vertically) and for each,
select which segment to start viewing (horizontally). Each
segment discusses an art object. An example from our demo
is a silver Frisian tea jar from the mid-18th century (Fig. 2).
Our viewer is very interested in Frisian heritage and the
expert in the TV program states that the tea jar is a typical
example of Frisian Silver. The viewer didn’t even know
there was such a thing, and would love to learn more about
what sort of other objects exist in this category. Figure 4. Viewing a related art object from Europeana.
TECHNICAL IMPLEMENTATION OF LINKEDCULTURE
LinkedCulture is built on top of the LinkedTV platform,
implementing a dedicated workflow which ingests video,
analyses and annotates it and generates links to related Web
content (enrichment) (Fig. 5). A Web-based Editor Tool
allows editors to view and curate the annotations and
enrichments of each episode prior to playout.
Figure 2. Screenshot of the Frisian silver tea jar (courtesy
AVROTROS, Tussen Kunst & Kitsch episode of 14 Nov 2007)
During any segment, the viewer can switch to further
information about the art object being discussed such as for
the silver Frisian tea jar, the screenshot shows an
information card about the location Friesland (Fig. 3).
Figure 5. Illustration of the workflow for the enrichment of TKK
episodes in the LinkedCulture application.
4
UI design in Figs. 3, 4 & 5 courtesy Lilia Perez Romero (CWI),
cf. LinkedTV D3.5 “Requirements Document LinkedTV User
Immediately at the video analysis step at the beginning of
Interfaces (v2)” http://slideshare.net/linkedtv/requirements- the LinkedTV workflow, the program is split into distinct
document-for-linkedtv-user-interfaces chapters each of which having a different art object as its
focus of discussion between the experts and the audience. The aggregated annotation is used for the subsequent
The segmentation approach uses the work of [4]. In the enrichment. Distinct Web services provide suggested links
TKK case, a set of visual cues have been identified for the to Web content based on those annotations. These
beginning of a new section of the TV episode where a new enrichment services are accessed by a single call to a Web
art object is introduced and discussed, such as the repetition service called TVEnricher which integrates the individual
of a textual overlay of the name of the expert discussing the services listed above to a single request/response. A shared
object. From a shot with this textual overlay our approach service called EntityProxy has also been implemented to
searches for prior and subsequent gradual transitions provide information cards on all entities selected in the
between shots to fix the chapter boundaries. video annotations similar to the Google Knowledge Graph
(seen in Fig. 3). We introduce here briefly only the
Once the “art object” chapters are known, Named Entity
Europeana enrichment service. The goal is to provide, for
Recognition [5] is performed across the transcript of each
an art object annotated in the TV program chapter, a set of
chapter so that entities (concepts) can be associated to them.
related art objects from the Europeana collection.
In the Editor Tool a specific interface to curators is
Considering the annotation of our silver Frisian tea jar:
available to complete the description of the art object in
each chapter (Fig. 6). The task of the editor is aided by this Type db:Container
extraction of candidate entities from the transcript (i.e.
instances of art object characteristics) which then are Material db:Silver
available as suggestions in the Editor Tool. Location db:Friesland
Period 1690 TO 1742
Since these properties can occur in the Europeana metadata
under different fields, we tested different approaches to
determine the most effective query. Since we query largely
textual metadata, it was immediately clear that we needed
! to consider two key aspects in a query:
Figure 6. Screenshot of Editor Tool. The editor annotates the Synonyms and similar terms. Metadata property values are
chapter in terms of the art object, with entities of different types not typed formally in Europeana to a taxonomy so that
(colours) suggested below to guide him/her in the task. generalisations or specialisations can be included in results.
Art object descriptions are stored in the LinkedTV platform Thus our query must expand its search terms to relevant
as RDF metadata. The usage of semantics (uniquely synonyms and similar terms.
identifying concepts as URIs, following Linked Data Language. Metadata is generally textual and in the
principles) allows us to disambiguate the intended meaning language of the providing organization. So queries need to
and retrieve additional data for each concept. The express terms in the local language.
annotation model focuses on a small set of characteristics of
CHOs typically present in Europeana metadata: object type, To address the above and temporal queries, SPARQL
creator, creation location, time period, material. We would be complex to model and execute. Thus we focused
followed Dublin Core just as the Europeana Data Model our experiment on using the Europeana API. Firstly we
(EDM) but defined more specific properties for location found the best fields to query for each art object property
and time: and then the best approach to expand each field-based
query to aim for the best precision in search results. With
Object type dc:type all queries restricted to “COUNTRY:netherlands”7:
Creator dc:creator Type. “what:container” finds 240 CHOs. In contrast,
Material dc:medium "skos_concept= http://vocab.getty.edu/aat/300045611” is
empty, and finds only 2 CHOs in the whole Europeana.
Creation location vra:locationCreationSite5
Material. “proxy_dc_format:zilver” finds 448 CHOs. Note
Time period dct:temporal6 the use of the Dutch zilver, the string silver returns no
matches in Dutch collections. In contrast, “what:zilver”
finds 10256 CHOs as often the material is referenced in the
object type or in subject categories.
5
This property was the most specific we found in use in CH
vocabularies to capture the sense of location where a CHO was
created. Europeana does not seem to have a clear approach to this, string, this property takes as value a dct:PeriodOfTime which is
often only the current location of the CHO is indicated. modelled with a distinct start and end date-time.
6
Compared to edm:year which takes a single integer for a 7
API query results from 28 January 2015
calendar year and dct:created which can take an unstructured data
Location. No metadata property clearly refers to creation overcoming problems of multilingualism and inconsistency
location, not even dc:source. “proxy_dc_source:Friesland” in the terms used. The final LinkedCulture application will
is empty. “location:Friesland” returns 1271 CHOs and use the annotations of ca. 35 art objects in 6 TKK episodes
“where:Friesland” returns 49 556 CHOs, yet both seem to and be trialed with TKK viewers to measure satisfaction
draw from the value of ‘geographical coverage’ with the suggested related Europeana CHOs.
Period. Temporal period queries are well supported, e.g. Europeana as an online portal to digitized collections of
“YEAR:[1690+TO+1742]” (60 440 CHOs) Europe’s cultural heritage is confronted with the problem
that the general public do not search in europeana.eu when
Creator. While not annotated for this object, the API offers
interested in exploring CHOs. Rather, applications are
the “who:” field over the metadata property of creator.
needed which can push relevant content from the portal to
Our approach was to expand the query to capture all people when appropriate. Additionally, exploration is made
possible variations of characteristic values in the metadata difficult by the complexity of the domain – even when
and focus on finding the Europeana CHOs closest in turning to Google while watching TV, how does the viewer
relevance by matching on at least 3 characteristics. seek other examples of silver Frisian tea jars? What
Expansion is importance since, e.g. type lacks a formal type happens when they are interested in a painting they see but
system and thus requires us to consider synonyms and missed the artist’s name? While not so apparent to the
related types in the query, or the spelling and formatting of viewer, we must add to this list the inconsistency in the
creator names can vary in the metadata while the API tries Europeana metadata. Until there is a more significant
to make an exact string match. Combining the queries, no amount of semantic annotation of CHOs, our approach
CHO matches on all four characteristics. We do find 4 successfully uses domain knowledge to expand queries and
CHOs for the combination of Frisian+silver+”from 1690 to address issues of term ambiguity, alternative spelling,
1742” and 4 CHOs for the combination of synonyms and multilingualism. LinkedCulture is a step
silver+container+”from 1690 to 1742” (if container is towards eased entry for the public into Europeana’s rich
expanded to also query on synonyms). Combining just two and deep collection of digital objects tied to the trending
characteristics in the query, result sets varied from 6 to activity of “second screen” usage with television.
1 287 CHOs, and thus often provided too many options on
CHOs which were only weakly related to the original. ACKNOWLEDGMENTS
The author(s) wish to acknowledge the hard work of the
To handle this for any arbitrary art object annotated in TKK entire project consortium (http://www.linkedtv.eu/about-
episodes we implemented a Europeana API wrapper which the-project/consortium/). We are indebted to AVROTROS
follows the above approach. Since English DBPedia is used for their permission to use TKK video. LinkedTV has only
in annotation, we follow owl:sameAs property links to the been feasible thanks to an EU research grant (FP7-287911).
Dutch DBPedia to get a Dutch label, while the
dbo:wikiPageRedirects property can indicate alternative REFERENCES
spellings and synonyms. For creators, the foaf:name 1. J. Abreu, P. Almeida, B. Teles, and M. Reis. „Viewer
property usually provides alternative forms. For locations, behaviors and practices in the (new) television
this is the values of the dbo:alternativeName property, environment”. In Proceedings of the 11th European
while dbo:isPartOf and dbo:part link to locations which Conference on Interactive TV and Video, EuroITV '13,
supersume or subsume this location. The value of this pages 5-12, New York, NY, USA, 2013. ACM.
approach is clear by simple introspection, e.g. if the editor
annotates the object’s creation location as the city of 2. Stanoevska, K. et al., “User Trial results”, LinkedTV
Franeker, which is quite correct, then the query on deliverable 6.3, March 2014.
Franeker+silver+”from 1690 to 1742” is empty, but since in 3. Nixon, L. and Baltussen, L.,“Engaging TV viewers with
DBPedia the concept Franeker dbo:isPartOf Friesland, the AudioVisual heritage on second screens”, EUScreenXL
query can be expanded and the 4 relevant CHOs found. conference, Rome, Italy, October 2014.
4. E. Apostolidis, V. Mezaris, "Fast Shot Segmentation
FUTURE WORK AND CONCLUSION Combining Global and Local Visual Descriptors", Proc.
The implementation of the described Europeana API IEEE Int. Conf. on Acoustics, Speech and Signal
wrapper and the semantic annotation of art objects it draws Processing (ICASSP), Florence, Italy, May 2014.
on has enabled the LinkedCulture application to semi-
automatically enrich TKK episodes with related art objects 5. Y. Li, G. Rizzo, J. L. R. García, R. Troncy, M. Wald,
from Europeana collections. Simple introspection can show and G. Wills. “Enriching media fragments with named
the benefits of the approach. Even for a single art object entities for video classification”. In 1st Worldwide Web
example, we show how a numerically restricted set of most Workshop on Linked Media (LiME'13), pages 469-476,
related art objects can be found via Europeana metadata Rio de Janeiro, Brazil, 2013.