DCU Linking Runs at MediaEval 2014: Search and Hyperlinking Task

Abstract

We describe Dublin City University (DCU)'s participation in the Hyperlinking sub-task of the Search and Hyperlinking task MediaEval 2014. The investigation focuses on how to e ciently identify target segments in a large BBC TV dataset. In our submission, Linear Discriminant Analysis is used to estimate fusion weights for multimodal features with the objective of improving the reliability of creating the potential links.

1. INTRODUCTION

For our participation in the Hyperlinking sub-task of Search and Hyperlinking of Television Content in MediaEval 2014, we developed a new strategy of target segment determination based on speaker identi cation. Linear Discriminant Analysis was used to estimate fusion weights. The paper is organized as follows: Section 2 describes a new strategy to determine target segments, Section 3 describes fusion weight determination based on Linear Discriminant Analysis, Section 4 gives our experimental results, and Section 5 concludes the paper.

SPEAKER-BASED TARGET SEGMENT

DETERMINATION

This is the third year of the MediaEval Search and Hyperlinking task [ 8, 7 ]. The previous editions showed that state-of-the-art IR techniques can be applied to multimodal hyperlinking. However, identifying e ective target segments is still an open issue. According to [ 7 ], a target segment should be moderate in length 10 to 120 seconds. In this paper, a simple and e cient target segment determination algorithm was developed by using the speaker identi cation from the LIMSI transcript [ 10 ].

Similar to searching model, the retrieval list of target segments are determined by the relevance to the query anchor. Existing researches [ 5, 3 ] use a sliding window to cover a variety of identifying target segments. We annotate that a target segment is a collection of multimodal features with time stamps, and those containing relevant information should be allocated a higher rank. As a result, we identify target segments as following. 1) Separate a video into a number of clips. The separating benchmark is de ned according to the multimodal feature distribution. Each separated clip is de ned as a seed segment. 2) Each seed segment increases the size of itself by merging adjacent segments. It stops when the length of newly merged segment reaches the target segment standard de ned in [ 7 ]. 3) Repeat step 2 until all the seed segments have been expanded. 4) Identify each expanded segment as the target segment.

The kernel of the algorithm is how to determine the seed segments in a video and how to de ne the benchmark to merge adjacent segments. Existing researches purpose that the speech transcript often plays an important role in effective hyperlinking [ 5, 3 ]. Our investigation is based on LIMSI transcripts. Identifying a seed segment involves the transcript and speaker analysis. We assume that a group of spoken words, if presented by the same speaker, is content related. LIMSI transcript provides speaker information with each sentence. De ne each transcript sentence as seni and its speaker ID as spki. Each video Vj is presented as a vector Vj = [sen1; sen2; :::seni]. If k continuous sentences [senn; senn+1; :::senn+k] satisfy spkn = spkn+1 = ::spkn+k, they are merged to create a new seed segment sdl, whose duration jsdlj is determined by the start time of senn and end time of senn+k. Our algorithm traverses all sentences to convert Vj into a new vector of seed segments [sd1; sd2; :::sdl]. We assume that two transcripts presented by di erent speakers, if close enough, are possible to relate to the same topic. As a result, our algorithm to create target segments is dened as:

Step 1: For each sdi in Vj , compare the two nearby speech transcripts. Merge the one with lower time interval to create a new target segment mgi. In this paper, the time interval threshold is set to be 10 seconds, following the minimum length of a target segment. Step 2: Continue step 1 until the length of merged segment mgi is longer than 2 minutes.

Step 3: Save the merged segment mgi and process the sdi+1 in Vj .

Step 4: Finally all merged segments [mg1; mg2; :::mgk] are regarded as potential target segments. 3.

ESTIMATE FUSION WEIGHTS USING

LINEAR DISCRIMINANT ANALYSIS

Linear Discriminant Analysis (LDA) algorithm [ 9 ]. was used to estimate linear fusion weights. The linear combination in LDA wT x can determine the fusing weights for the multimodal features. We de ne a training group as jXj and its corresponding sample groups as jY j, where yi 2 jY j could be 0 or 1, meaning that the video segment to a speci c query anchor can be either related or non-related. Therefore, the training data was separated into two classi cations: jX1j where x 2 X1 satis es p(xijyi = 0), and jX2j, where x 2 X2 satis es p(xijyi = 1). Assuming both dataset follows the normal distribution, we have the means 1, 2, the covariance 1 and 2. The ratio between them was de ned as the class variance Sb, the within class variance Sw, and w as the vector of fusing weights. Sw and Sb can be de ned as:

Sb = (w 1 w

2)2 Sw = (wT 1 w + wT 2 w)

A linear combination was calculated by maximizing the criterion of between class variance and within class variance. De ne the criterion c as following: c = wT ( 1 + 2) 2)2

According to [ 2 ], a maximum separation can be achieved by maximizing c by making the weight vector to follow : w / ( 1 + 2) 1 ( 1 2)

MAP evaluation on DCU Hyperlinking

EXPERIMENT

The experiment data is introduced in MediaEval 2014 overview paper [ 6 ]. Our experiment is constructed based on only LIMSI transcripts. Two runs are investigated to evaluate the performance of speaker-based target segment determination and fusion weight estimation using LDA. All 2 runs creates the target segments using speaker-based determination algorithm. RUN 1 involves the late fusion model on metadata and LIMSI transcripts. RUN 2 involves only LIMSI transcripts. The fusion equation is de ned as following:

FusionScore = wm

MetadataScore + wt LIMSIScore (5)

In RUN 1, the linear fusion weights wm and wm are determined by LDA algorithm. The hyperlinking retrieval on metadata and LIMSI transcripts are processed using TDIDF model on text features. Lucene 4.9.0 is applied to implement indexing and searching. The details are provided in [ 4 ]. The retrieval model in RUN 2 uses Lucene implementation as well.

Table 1 shows the Mean Average Precision (MAP) value of each runs. The evaluation metrics is de ned in [ 1 ]. The experiment reveals that LDA estimation on linear fusion work could improve the hyperlinking quality, as presented in Table 1 that RUN 1 has an overall increase on MAP, MAP bin and MAP tol. (1) (2) (3) (4) 6. 7.

CONCLUSIONS

This paper presented details of DCU's participation in the TV Data Hyperlinking task of MediaEval 2014. The evaluation shows that LDA algorithm can improve the hyperlinking performance by estimating multimodal fusion weights. Our future work will be placed on comparing the speakerbased target segment determination with other strategies described in previous MediaEval research [ 8, 7 ].

ACKNOWLEDGEMENT

This work is funded by the European Commission's Seventh Framework Programme (FP7) as part of the AXES project (ICT-269980).

References

[1] R. Aly , M. Eskevich , R. Ordelman , and G. J. F. Jones . Adapting Binary Information Retrieval Evaluation Metrics for Segment-based Retrieval tasks . 2013 .

[2] G. Balakrishnama. Linear Discriminant Analysis - A Brief Tutorial , 1998 .

[3] S. Chen , M. Eskevich , G. J. F. Jones , and N. E. O' Connor . An Investigation into Feature E ectiveness for Multimedia Hyperlinking . In MultiMedia Modeling , pages 251 { 262 . Dublin, Ireland, 2014 .

[4] S. Chen , G. J. F. Jones , and N. E. O' Connor. DCU Linking Runs at Mediaeval 2013: Search and Hyperlinking Task . In MediaEval 2013 Workshop, Barcelona, Spain, 2013 .

[5] S. Chen , K. McGuinness , R. Aly , N. O 'Connor , and F. de Jong. The AXES-lite video search engine . In Proceedings of WIAMIS 2012 , pages 1 { 4 , Dublin, Ireland, 2012 .

[6] M. Eskevich , R. Aly , R. Ordelman , D. N. Racca , S. Chen , and G. J. F. Jones . The Search and Hyperlinking Task at Mediaeval 2014. In In Proceedings of the MediaEval 2014 Multimedia Benchmark Workshop , Barcelona, Spain, 2014 .

[7] M. Eskevich , G. J. F. Jones , S. Chen , R. Aly , and R. Ordelman . Search and Hyperlinking Task at MediaEval 2013 . In MediaEval 2013 Workshop, Barcelona, Spain, 2013 .

[8] M. Eskevich , G. J. F. Jones , S. Chen , R. Aly , R. Ordelman , and M. Larson . Search and Hyperlinking Task at MediaEval 2012 . In MediaEval 2012 Workshop, Pisa, Italy, 2012 .

[9] R. A. FISHER. The Use of Multimple Measurements in Taxonomic Problems . pages 179 { 188 , 1936 .

[10] L. Lamel and J.-L. Gauvain . Speech processing for audio indexing . In Advances in Natural Language Processing (LNCS 5221) , pages 4 { 15. Springer-Verlag, 2008 .