Authoring and Publishing Linked Open Film-Analytical Data

Abstract

Exploiting Linked Open Data for the annotation of audiovisual patterns in lm-analytical studies provides signi cant advantages, such as the non-ambiguous use of language as well as the possibility to publish and to reuse valuable data. On the other hand, lm scholars typically lack the know-how to cope with semantic annotations and, moreover, most software for annotating audio-visual material does not provide means to enter semantic annotations directly. The project presented in this paper aims to provide an ontology for lm-analytical studies complemented by a video annotation software adapted for authoring and publishing Linked Open Data by non-experts.

-

The study of audio-visual rhetorics of a ect scienti cally analyses the impact of auditory and visual staging patterns on the perception of media productions as well as the conveyed emotions [ 4 ]. The AdA-project4 aims to explore the hypothesis of TV reports drawing on audio-visual patterns in cinematographic productions to emotionally a ect viewers, by analyzing TV reports, documentaries and genre- lms of the topos \ nancial crisis". In a large-scale corpus analysis lm scientists identify and annotate low- to high-level audio-visual patterns, such as shot duration, dominant colors, major-minor tonality and depicted visual concepts. Comparison of di erent annotations from di erent scenes and genres allows lm scientists to analyze this opinion-forming level of reporting. In order to avoid ambiguities within and enable reuse of and reasoning based on the generated pattern annotations, we pursue two main objectives: 1) creating a standardized annotation vocabulary to be applied for semantic annotations and 2) enabling non-expert users to adopt and bene t from these annotations. 4 AdA-project | http://www.ada.cinepoetics.fu-berlin.de/ In the next section of this paper, we describe the vocabulary used for annotation of audio-visual material, which is based on Linked Open Data principles. The annotation process is performed by lm scientists using Advene [ 3 ] | a software toolkit to annotate audio-visual documents. To accommodate speci c needs of the project, Advene was extended to improve the manual annotation process, with integration of automatic and semi-automatic helpers, and with support of RDF interoperability through the import of OWL ontologies and export of RDF data. The third section of this paper will therefore present these extensions in more detail and show how Advene helps lm scientists to apply semantic annotations without having to deal with the technical challenges. The last part of this paper will give a small outlook on the presented demo. We provide some screenshots and screencasts of the Advene video annotation software at https://ProjectAdA.github.io/ekaw2018/. 2

Open Film-Analytical Data

Film scientists carry out an in-depth corpus analysis by precisely describing feature lms, documentaries and TV news according to a lm-analytical annotation method called eMAEX5. The description involves a lot of manual e ort, creating hundreds of annotations per scene as ground truth data. One goal of the project [ 1 ] is to publish this valuable data as Linked Open Data to make these annotations available to other lm scientists as well as researchers from other domains. Therefore, we developed an ontology for ne-grained semantic video annotation, constructed semantic metadata for our video corpus, and implemented linked data extensions for the Advene annotation software.

The AdA ontology o ers a vocabulary with a number of categories under which a movie is analyzed (e.g., camera, image composition, acoustics). Each category includes the respective concepts with which the segments of a movie are annotated (e.g., camera movement speed, eld size). About 75% of the concepts have associated prede ned values (e.g., long shot, medium shot, closeup and others for eld size). Others are free-text annotations, such as dialog transcriptions. Currently, the AdA ontology includes 9 categories (annotation level), 78 concepts (annotation types), and 435 prede ned annotation values. We provide an online version6 and a download at the GitHub page of the project7.

The ontology contains a data model that uses the latest Web Annotation Vocabulary8 to express annotations and Media Fragments URIs 9 for timecodebased referencing of video material. This allows the publication of semantic audio and video annotations as Linked Open Data. Semantic metadata of the video corpus is described using the classes and properties of DBpedia, Schema.org, 5 eMAEX - Electronically-based Media Analysis of EXpressive movements | https: //empirische-medienaesthetik.fu-berlin.de/en/emaex-system/ 6 http://ada.filmontology.org/ 7 https://github.com/ProjectAdA/public 8 https://www.w3.org/TR/annotation-vocab/ 9 https://www.w3.org/TR/media-frags/

Authoring and Publishing Linked Open Film-Analytical Data and Linked Movie Database, and movies are linked to DBpedia and Wikidata if present in the respective knowledge base. 3

Bridging the Gap with Advene

Advene is a free (GPL) video annotation software aiming at a great exibility to accommodate the various needs of di erent users. Some adaptations have been implemented for the project in order to facilitate the annotation task, allowing faster annotation, better collaboration and exporting data into RDF, continuing previous e orts [ 2 ] in this domain. One of the goals of the project is to allow non-technical users to manage semantic information, more speci cally semantic metadata linked to audiovisual fragments. In addition to the generic constraints of video annotation, a common hurdle is the tediousness of specifying precise URIs for semantic content. The Advene platform de nes a notion of package, which contains the annotations themselves, but also the de nition of the annotation structure and their visualizations. In the context of a speci c Advene package, URIs for all ontology elements are speci ed as metadata, which gives the semantic context for the non-semantic information contained in the annotations. Users are facing mostly basic types and keywords, linked to their expertise domain. When exporting annotation data to RDF, keywords are translated to proper URIs using information present as annotation type metadata.

OWL Import The Advene data model consists of user-de ned annotation types and relation types. These types can be grouped into schemas that materialize a speci c analysis frame. One of the goals of the project was to provide a way to map the AdA ontology into an Advene set of types and schemas, accompanied with appropriate metadata. The mapping of the AdA ontology structure translated the notion of Annotation Level into Advene schema, and Annotation Types as Advene annotation types. The annotation content can be part of a xed vocabulary, de ned by the ontology. Each prede ned value from the ontology has been mapped to a simple keyword (shortname), with metadata allowing to remap it to its original URI during the RDF export phase. An OWL import plugin was developed for Advene. As AdA ontology development underwent many iterations, Advene's merge functionality has been improved to better support data migration.

RDF Export One of the characteristics of Advene is the possibility for users to de ne their own visualisations, through a template language initially aimed at producing XML data, but which can also produce any kind of data. This approach, which is used for the majority of Advene export lters, has initially been used to implement an RDF export of the annotation data, using the AdA ontology model. However, some project-speci c complex structures like evolving or contrasting values could not easily be implemented through this simple syntax-based approach. A new RDFLib-based export lter has thus been developed, o ering better robustness, expressiveness and performance. As with the ontology import lter, it is rather speci c to the AdA ontology, but its code can be used as a reference implementation for other RDF export lters. Interface Adaptation The annotation interface has been streamlined and extended, to allow faster input of prede ned vocabularies. In order to increase the speed of manual annotation by means of automatic content classi cation, the interface allows for calling external webservices. As a rst proof-of-concept, a convolutional neural network-based approach for visual concept detection in keyframes has been implemented. Advene tries to be agnostic about the data that can be stored in annotations. In order to ensure that newly added annotations are part of the AdA ontology, the Advene constraint checker, that gives a constant feedback, has been extended with ontology-speci c checkers. 4

Demo and Conclusion

In the demo it will be demonstrated how to create semantic annotations with Advene. This includes importing the project's ontology, using Advene templates, creating annotations in the timeline view, using the constraint checker and exporting RDF. Also insights will be provided into the linked data that is already published under http://ada.filmontology.org/: the AdA Ontology, video corpus metadata as well as RDF annotations of the use-case movie. Finally, it will be demonstrated how this data can be queried using the SPARQL endpoint.

In this paper, we presented a small overview about how semantic annotations are applied in lm-analytical studies. An ontology has been developped specifically for the purpose of annotating audio-visual patterns using Linked Open Data and it has been presented how Advene can be applied to hide the complexity of semantic annotations for non-expert users. Future work will focus on the integration of semi-automatic analysis in order to speedup the expensive annotation process.

Acknowledgments. This work is partially supported by the Federal Ministry of Education and Research under grant number 01UG1632B.

References

1. Agt-Rickauer , H. , Hentschel , C. , Sack , H.: Semantic annotation and automated extraction of audio-visual staging patterns in large-scale empirical lm studies . In: Proceedings of the Posters and Demos Track of the 14th International Conference on Semantic Systems (SEMANTiCS 2018 ), Vienna, Austria ( 2018 )

2. Aubert , O. , Champin , P.A. , Prie , Y. : Integration of semantic web technology in an annotation-based hypervideo system . In: SWAMM 2006 , First International Workshop on Semantic Web Annotations for Multimedia, 15th WWW Conference ( 2006 )

3. Aubert , O. , Prie , Y. : Advene: active reading through hypervideo . In: ACM Hypertext'05 . pp. 235 { 244 . Salzburg , Austria (Sep 2005 )

4. Scherer , T. , Greifenstein , S. , Kappelho , H.: Expressive movements in audiovisual media: Modulating a ective experience . In: Muller, C. , Cienki , A. , Fricke , E. , et al. (eds.) Body { Language { Communication. An international handbook on multimodality in human interaction , pp. 2081 { 2092 . De Gruyter Mouton, Berlin, New York ( 2014 )