This paper describes a modular architecture for searching and hyperlinking clips of TV programmes. The architecture aimed to unify the combination of features from di erent modalities through a common representation based on a set of probability density functions over the timeline of a programme. The core component of the system consisted of analysis of sections of transcripts based on a textual query. Results show that search is made worse by the addition of other components, whereas in hyperlinking precision is increased by the addition of visual features.
The 2013 MediaEval search an hyperlinking task [
Our overall idea was to represent each programme by a probability density function (PDF) over the timeline of the programme. The area under the PDF between two time points essentially represents the probability of that portion of the programme to being relevant to the query. By constructing PDFs for each programme with a given query, we can then locate the high-probability segments of the PDFs, which in turn tell us the beginning and end times of hits that can be returned to the user. With respect to the search and hyperlinking tasks, the primary di erence is the form of the query.
The architecture backing the approach was modular, and we constructed various modules to incorporate data from di erent data sources. We developed two forms of module; the rst was capable of using the query to generate new data points for the PDFs for a set of programmes, and the second was capable of weighting the entire PDF for a given programme (i.e. to increase or decrease the global relevance). The modules responsible for adding to the PDF worked by placing Gaussian functions at the point of interest on the timeline, with variance proportional to the length of the segment of interest. At the end, the overall PDF for a programme can be computed from summation of the Gaussians at every time point; this entire process can be viewed as a variable bandwidth kernel density estimation.
Most of the work is done by modules working on the textual data (transcripts, synopsis, program titles). This information was indexed using Lucene1 with separate elds for each source. Each module is described brie y below. 2.1
The transcript search module was the most important component of the system. The module searched for keywords (taken from the query string) across all transcripts of a certain kind (LIMSI/Vocapia, LIUM, or subtitles) using the Lucene index. Keyword matches were extracted from each transcript in turn and grouped using hierarchical agglomerative clustering over the time di erences between hits within a programme. When a cluster's separation (calculated as the temporal distance between the average values of the cluster's left and right children) fell below a specied threshold, a cluster was formed, and used to build a Gaussian whose amplitude was calculated from
jQj w2WQ = jWQj X boost(w) idf(w) where Q was the set of all possible keywords from the query, WQ was the subset of query keywords that appeared in the transcript, idf : W ! R was a function mapping each keyword on to its inverse document frequency, and boost : W ! R was a function mapping each keyword on to its Lucene query boost. Additionally, the true amplitude was scaled by the normalised score returned by Lucene when searching for transcript documents matching the query. The amplitude of the Gaussian captures the relevance of all keywords in the cluster with respect to the document, as well as how completely the cluster covers the set of all possible query terms. The Gaussians were centred on the midpoint of the range covered by the cluster, and the standard deviation of the Gaussian was chosen as one third of the temporal size of the cluster plus 60 seconds.
The concept module analysed the query text and visual cues for known concepts that could be added to timelines. The amplitude for the concept module's Gaussians was determined from the normalised con dence for each concept 1http://lucene.apache.org Run code S M Mod U M Mod I M Mod S MV ModCon U MV ModCon I MV ModCon S MV ModConLSH U MV ModConLSH I MV ModConLSH detection, and the standard deviation was a constant 5 seconds.
The visual information module worked by nding shots
that were visually similar to other shots with high con
dence. For each programme, the most stable key-frame of
each shot was extracted and SIFT features were calculated.
Each SIFT feature was hashed using locality-sensitive
hashing (LSH) and a graph was constructed where the vertices
were keyframes and edges were created if pairs of key-frames
contained colliding features [
The synopsis and title modules increased the weight of timelines belonging to programmes whose keywords matched keywords in the synopsis and title elds of the index. The channel lter module performed nave NLP on the query: if a channel was mentioned in the query, then any timelines corresponding to programmes on other channels were removed from the timeline set (i.e. setting the weight to zero).
The architecture described in the previous section was used to facilitate both the search and hyperlinking tasks. For the search task, the system was con gured to take the query and pass it directly to each module. Concepts were inferred from the query text and the visual information module was used for query expansion, through the detection of visually similar segments to high-con dence detections from the other modules. For hyperlinking, the transcript of the anchor segment was used as the query text (together with the synopsis, title and channel of the programme from which the anchor was drawn). Concepts detected in the anchor were used as input to the concept module. The visual information module was used to nd segments that were visually similar to the anchor.
The results from the search task are summarised in Table 1. Runs using the subtitles gave the best performance in each category, which is understandable due to the more accurate nature of subtitles compared with speech-to-text transcripts. It is interesting to observe that the performance of the system decreased as additional features were brought in, which may indicate that these additional modules were not scaled properly, or otherwise very noisy in the context of the query. This is surprising for concept detection, as in the search task concepts were directly picked from the query.
Table 2 shows the results for the hyperlinking task. It can be seen that the baseline results with just the subtitle information used give the best results. Adding the concept detections causes a drop in performance, indicating that similar visual concepts do not necessarily indicate relevant content. The addition of the SIFT-LSH features looks like it might slightly improve performance; this needs further veri cation without the concepts. 5.
The system performed better at the hyperlinking sub-task than the search sub-task; this was slightly unexpected as the search performance on the development data was higher. This may in part be due to fundamental limits within the transcript module: textual queries have a low bandwidth and describe many features that are not discernible from a programme's transcript, and thus a more complex approach might be required to improve performance. Additional NLP on queries, along with person detection (i.e. face recognition/veri cation), could also improve performance in the search domain.
For both tasks the addition of visual information tended to harm overall performance. Again, this was slightly unexpected, as we saw improvements in the development search queries when using this information. One possible reason for this is that the visual information is only useful for certain types of query (or anchor). It would be interesting to explore this further; a starting point for this would be to analyse the results on a per-item basis (rather than just looking at the overall averages).