This short/demo paper describes LinkedCulture, a Web based application which complements the viewing of a well known Dutch cultural heritage TV program with the ability of viewers to explore art objects from Europeana related to those in the program.
TRENDS AND RELATED WORK
Households have more and more connected devices and
consumers are increasingly using devices in parallel: this is
clearest when it comes to viewing audiovisual
programming on one device and browsing online content on
another. A 2013 published survey [
In the commercial domain, there is not yet a widely successful application of second screen TV enrichment due to the excessive cost of annotating TV programming and manually preparing the enrichment. Shazam for TV (http://www.shazam.com/music/web/productfeatures.html?i d=1266), for example, focuses on a TV program as a whole, e.g. actor information or episode trivia. Some demos have been made with concept-level approaches, e.g. using Mozilla Popcorn (https://popcorn.webmaker.org/), linking terms in video subtitles to content shown in other frames alongside the video. These demos suffered from the lack of disambiguation of terms (such as “Paris”) and limited link relevance (typically Wikipedia, a map etc.). LinkedTV provides an automated workflow (cf. Sec 4) with better disambiguation of concepts referred to within audiovisual material as well as richer linking to sets of relevant content from varied online sources. We believe this provides significantly better automated results than any prior work and thus also forms the basis for a usable cost-effective solution by content owners. The rest of this paper will focus on how this approach was used in enriching a cultural heritage TV program.
DEMONSTRATOR FOR CULTURAL HERITAGE
LinkedCulture3 [
Immediately at the video analysis step at the beginning of
the LinkedTV workflow, the program is split into distinct
chapters each of which having a different art object as its
focus of discussion between the experts and the audience.
The segmentation approach uses the work of [
Once the “art object” chapters are known, Named Entity
Recognition [
Time period dc:type dc:creator dc:medium vra:locationCreationSite5 dct:temporal6 5 This property was the most specific we found in use in CH vocabularies to capture the sense of location where a CHO was created. Europeana does not seem to have a clear approach to this, often only the current location of the CHO is indicated. 6 Compared to edm:year which takes a single integer for a calendar year and dct:created which can take an unstructured data The aggregated annotation is used for the subsequent enrichment. Distinct Web services provide suggested links to Web content based on those annotations. These enrichment services are accessed by a single call to a Web service called TVEnricher which integrates the individual services listed above to a single request/response. A shared service called EntityProxy has also been implemented to provide information cards on all entities selected in the video annotations similar to the Google Knowledge Graph (seen in Fig. 3). We introduce here briefly only the Europeana enrichment service. The goal is to provide, for an art object annotated in the TV program chapter, a set of related art objects from the Europeana collection. Considering the annotation of our silver Frisian tea jar:
db:Container db:Silver db:Friesland Since these properties can occur in the Europeana metadata under different fields, we tested different approaches to determine the most effective query. Since we query largely textual metadata, it was immediately clear that we needed to consider two key aspects in a query: Synonyms and similar terms. Metadata property values are not typed formally in Europeana to a taxonomy so that generalisations or specialisations can be included in results. Thus our query must expand its search terms to relevant synonyms and similar terms.
language of the providing organization. So queries need to express terms in the local language.
To address the above and temporal queries, SPARQL would be complex to model and execute. Thus we focused our experiment on using the Europeana API. Firstly we found the best fields to query for each art object property and then the best approach to expand each field-based query to aim for the best precision in search results. With all queries restricted to “COUNTRY:netherlands”7: Type. “what:container” finds 240 CHOs. In contrast, "skos_concept= http://vocab.getty.edu/aat/300045611” is empty, and finds only 2 CHOs in the whole Europeana. Material. “proxy_dc_format:zilver” finds 448 CHOs. Note the use of the Dutch zilver, the string silver returns no matches in Dutch collections. In contrast, “what:zilver” finds 10256 CHOs as often the material is referenced in the object type or in subject categories. string, this property takes as value a dct:PeriodOfTime which is modelled with a distinct start and end date-time. 7 API query results from 28 January 2015 Location. No metadata property clearly refers to creation location, not even dc:source. “proxy_dc_source:Friesland” is empty. “location:Friesland” returns 1271 CHOs and “where:Friesland” returns 49 556 CHOs, yet both seem to draw from the value of ‘geographical coverage’ Period. Temporal period queries are well supported, e.g. “YEAR:[1690+TO+1742]” (60 440 CHOs) Creator. While not annotated for this object, the API offers the “who:” field over the metadata property of creator. Our approach was to expand the query to capture all possible variations of characteristic values in the metadata and focus on finding the Europeana CHOs closest in relevance by matching on at least 3 characteristics. Expansion is importance since, e.g. type lacks a formal type system and thus requires us to consider synonyms and related types in the query, or the spelling and formatting of creator names can vary in the metadata while the API tries to make an exact string match. Combining the queries, no CHO matches on all four characteristics. We do find 4 CHOs for the combination of Frisian+silver+”from 1690 to 1742” and 4 CHOs for the combination of silver+container+”from 1690 to 1742” (if container is expanded to also query on synonyms). Combining just two characteristics in the query, result sets varied from 6 to 1 287 CHOs, and thus often provided too many options on CHOs which were only weakly related to the original. To handle this for any arbitrary art object annotated in TKK episodes we implemented a Europeana API wrapper which follows the above approach. Since English DBPedia is used in annotation, we follow owl:sameAs property links to the Dutch DBPedia to get a Dutch label, while the dbo:wikiPageRedirects property can indicate alternative spellings and synonyms. For creators, the foaf:name property usually provides alternative forms. For locations, this is the values of the dbo:alternativeName property, while dbo:isPartOf and dbo:part link to locations which supersume or subsume this location. The value of this approach is clear by simple introspection, e.g. if the editor annotates the object’s creation location as the city of Franeker, which is quite correct, then the query on Franeker+silver+”from 1690 to 1742” is empty, but since in DBPedia the concept Franeker dbo:isPartOf Friesland, the query can be expanded and the 4 relevant CHOs found. FUTURE WORK AND CONCLUSION The implementation of the described Europeana API wrapper and the semantic annotation of art objects it draws on has enabled the LinkedCulture application to semiautomatically enrich TKK episodes with related art objects from Europeana collections. Simple introspection can show the benefits of the approach. Even for a single art object example, we show how a numerically restricted set of most related art objects can be found via Europeana metadata overcoming problems of multilingualism and inconsistency in the terms used. The final LinkedCulture application will use the annotations of ca. 35 art objects in 6 TKK episodes and be trialed with TKK viewers to measure satisfaction with the suggested related Europeana CHOs.
Europeana as an online portal to digitized collections of Europe’s cultural heritage is confronted with the problem that the general public do not search in europeana.eu when interested in exploring CHOs. Rather, applications are needed which can push relevant content from the portal to people when appropriate. Additionally, exploration is made difficult by the complexity of the domain – even when turning to Google while watching TV, how does the viewer seek other examples of silver Frisian tea jars? What happens when they are interested in a painting they see but missed the artist’s name? While not so apparent to the viewer, we must add to this list the inconsistency in the Europeana metadata. Until there is a more significant amount of semantic annotation of CHOs, our approach successfully uses domain knowledge to expand queries and address issues of term ambiguity, alternative spelling, synonyms and multilingualism. LinkedCulture is a step towards eased entry for the public into Europeana’s rich and deep collection of digital objects tied to the trending activity of “second screen” usage with television. ACKNOWLEDGMENTS The author(s) wish to acknowledge the hard work of the entire project consortium (http://www.linkedtv.eu/aboutthe-project/consortium/). We are indebted to AVROTROS for their permission to use TKK video. LinkedTV has only been feasible thanks to an EU research grant (FP7-287911).