LinkedTV at MediaEval 2014 Search and Hyperlinking Task

Abstract

The paper presents the LinkedTV approaches for the Search and Hyperlinking (S&H) task at MediaEval 2014. Our submissions aim at evaluating 2 key dimensions: temporal granularity and visual properties of the video segments. The temporal granularity of target video segments is de ned by grouping text sentences, or consecutive automatically detected shots, considering the temporal coherence, the visual similarity and the lexical cohesion among them. Visual properties are combined with text search results using multimodal fusion for re-ranking. Two alternative methods are proposed to identify which visual concepts are relevant to each query: using WordNet similarity or Google Image analysis. For Hyperlinking, relevant visual concepts are identied by analysing the video anchor.

1. INTRODUCTION

This paper describes the framework used by the LinkedTV team to tackle the problem of Search and Hyperlinking inside a video collection [ 3 ]. The applied techniques originate from the LinkedTV project1, which aims at integrating TV and Web documents, by enabling users to access additional information and media resources aggregated from diverse sources, thanks to automatic media annotation. Here follows the description of our media annotation process. Shot segmentation is performed using a variation of [ 1 ], while the selected keyframes (one per shot) are analysed by visual concept detection [ 9 ] and Optical Character Recognition (OCR) [ 11 ] techniques. For each video, keywords are extracted from the subtitles, based on the algorithm presented in [ 12 ]. Finally, video shots are grouped into longer segments (scenes) based on 2 hierarchical clustering strategies. Media annotations are indexed at 2 levels (video level and scene level) using the Apache Solr platform2. At the video level, document descriptions are limited to text (title, subtitle, keywords, etc...), while the scene level documents are characterized by both text (subtitle/transcript, keywords, ocr, etc...) and oat elds. Each oat eld corre1http://www.linkedtv.eu/ 2http://lucene.apache.org/solr/ sponding to a unique visual concept response. 1.1

Temporal Granularity

Three temporal granularities are evaluated. The rst, termed Text-Segment, consists in grouping together sentences (up to 40) from the text sources. We also propose to segment videos into scenes which consist of semantically correlated adjacent shots. Two strategies are employed to create scene level temporal segments. Visually similar adjacent shots are merged together to create Visual-scenes [ 10 ], while Topicscenes are built by jointly considering the aforementioned results of visual scene segmentation and text-based topical cohesion (exploiting text extracted from ASR transcripts or subtitles). 1.2

Visual Properties

In MediaEval S&H 2014, queries are composed of a few keywords only (visual-cues are not provided). Hence, the identi cation of relevant visual concepts is more complex than last year. We propose two alternatives to this problem. On one hand, WordNet similarity is employed to map visual concepts with query terms [ 8 ]. On the other hand, the query terms are used to perform a Google Image search. Visual concept detection (using 151 concepts from the TRECVID SIN task [ 6 ]) is performed on the rst 100 returned images and concepts obtaining the highest average score are selected. 2. 2.1

SEARCH SUB-TASK

Text-based methods

In this approach, relevant text and video segments are searched using Solr using text (TXT ) only. Two strategies are compared: one where search is performed at the text segment level directly (S ) and one where the rst 50 videos are retrieved at the video level and then the relevant video segment is locate using the scene-level index. The scenelevel index granularity is either the Visual-Scene (VS ) or the Topic-Scene (TS ). Scenes at both granularities are characterized by textual information only (either the subtitle (M ) or one of the 3 ASR transcripts ( (U ) LIUM [ 7 ], (I ) LIMSI [ 4 ], (S ) NST/She eld [ 5 ])).

Motivated by [ 8 ], visual concept scores are fused with text-based results from Solr to perform re-ranking. Relevant visual concepts, out of the 151 available, for individual queries are identi ed using either the WordNet (WN ) or the GoogleImage (GI ) strategy. For those multi-modal (MM ) runs only visual scene (VS ) segmentation is evaluated.

HYPERLINKING SUB-TASK

Pivotal to the hyperlinking task is the ability to automatically craft an e ective query from the video anchor under consideration, to search within the annotated set of media. We submitted two alternative approaches; One using the MoreLikeThis (MLT ) Solr extension, and the other using Solr's query engine. MLT is used in combination with the sentence segments (S ), using either text (MLT1 ) or text and annotations [ 2 ] (MLT2 ). When Solr is used directly, we consider text only (TXT ) or with visual concept scores of anchors (MM ) to formulate queries. Keywords appearing within the query anchor's subtitles compose the textual part of the query. Visual concepts whose scores within the query anchor exceed the 0.7 threshold are identi ed as relevant to the video anchor and added to the Solr query. Both visual (VS ) and topic scenes (TS ) granularities are evaluated in this approach. 4.1

RESULTS

Search sub-task

4.2

Hyperlinking sub-task

Table 2 shows the performance of our hyperlinking runs. Again, the approach based on subtitle only (TXT VS M ) performed best (MAP=0,25) followed by the approach using MoreLikeThis (TXT S MLT1 M ). Multi-modal approaches did not produce the expected performance improvement. We believe this is due to the signi cant duration reduction of anchors compared with last year.

CONCLUSION

The results of LinkedTV's approaches on the 2014 MediaEval S&H task show that it is di cult to improve over text based approaches when no visual cues are provided. Overall, our S&H algorithms performance on this year's dataset have decreased compared to 2013, showing that task de nition changes have made the task harder to solve. 6.

ACKNOWLEDGMENTS

This work was supported by the European Commission under contract FP7-287911 LinkedTV.

References

[1] E. Apostolidis and V. Mezaris . Fast shot segmentation combining global and local visual descriptors . In 2014 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP) , p 6583 { 6587 , Italy .

[2] M. Dojchinovski and T. Kliegr . Entityclassi er.eu: Real-Time Classi cation of Entities in Text with Wikipedia . In H. Blockeel, K. Kersting , S. Nijssen , and F. Zelezny, editors, Machine Learning and Knowledge Discovery in Databases , volume 8190 of Lecture Notes in Computer Science, pages 654 { 658 . Springer, 2013 .

[3] M. Eskevich , R. Aly , D.N. Racca , R. Ordelman , S. Chen , and G.J.F. Jones . The Search and Hyperlinking Task at MediaEval 2014 . In MediaEval 2014 Workshop, Spain.

[4] J.-L. Gauvain , L. Lamel , and G. Adda. The LIMSI broadcast news transcription system . Speech Communication , 37 ( 1 ): 89 { 108 , 2002 .

[5] T. Hain , A. El Hannani , S. N. Wrigley , and V. Wan . Automatic speech recognition for scienti c purposes-webasr . In Interspeech, Australia , pages 504 { 507 , 2008 .

[6] P. Over et al. TRECVID 2012 { An Overview of the Goals, Tasks, Data, Evaluation Mechanisms and Metrics . In Proceedings of TRECVID 2012 . NIST , USA, 2012 .

[7] A. Rousseau , P. Deleglise , and Y. Esteve . Enhancing the TED-LIUM corpus with selected data for language modeling and more TED talks . In LREC 2014 , Iceland.

[8] B. Safadi , M. Sahuguet , and B. Huet . When textual and visual information join forces for multimedia retrieval . In ACM ICMR 2014 , Glasgow, Scotland.

[9] P. Sidiropoulos , V. Mezaris , and I. Kompatsiaris. Enhancing Video concept detection with the use of tomographs . In IEEE ICIP 2013 , Australia.

[10] P. Sidiropoulos , V. Mezaris , I. Kompatsiaris, H. Meinedo , M. Bugalho , and I. Trancoso . Temporal Video Segmentation to Scenes Using High-Level Audiovisual Features . IEEE Transactions on Circuits and Systems for Video Technology , 21 ( 8 ): 1163 { 1177 , Aug . 2011 .

[11] D. Stein , S. Eickeler , R. Bardeli , E. Apostolidis , V. Mezaris , and M. Muller. Think Before You Link { Meeting Content Constraints when Linking Television to the Web . In Proc. NEM Summit , Nantes, France, Oct. 2013 .

[12] S. Tschopel and D. Schneider . A lightweight keyword and tag-cloud retrieval algorithm for automatic speech recognition transcripts . In Interspeech, Japan , 2010 .