We describe the LACS submission to the Search sub-task of the Search and Hyperlinking Task at MediaEval 2014. Our experiments investigate how different retrieval models interact with word stemming and stopword removal. On the development data, we segment the subtitle and Automatic Speech Recognition (ASR) transcripts into fixed length time units, and examine the effect of different retrieval models. We find that stemming provides consistent improvement; stopword removal is more sensitive to the retrieval models on the subtitles. These manipulations do not contribute to stable improvement on the ASR transcripts. Our experiments on test data focus on the subtitle. The gap in performance for different retrieval models is much less compared to the development data. We achieved 0.477 MAP on the test data.
The amount and variety of multimedia data available online is
rapidly increasing. As a result, the techniques for identifying
content relevant to a query need to improve, to effectively process
large multimedia data collections. There are existing works
utilizing multi-modality for multimedia retrieval [
Copyright is held by the author/owner(s).
We first present our results on the development (dev) data, reporting the Mean Reciprocal Rank (MRR). The dev experiment is known item retrieval. The parameter for Okapi retrieval models is k1= 1.2, b= 0.75 and k3 = 7, and the μ for LM is 2500. From Table 1 and 2, we can observe the interaction between different processing and retrieval models. Stemming and stopword removal provides persistent improvement on subtitles. On the other hand, for the ASR transcript, these appear unstable. Aside from the difference due to recognition errors, one possible factor contributing to this phenomenon is the size of vocabulary. The vocabulary size for the subtitle is 251506, while the vocabulary size for the ASR transcription is 83094, one-third of subtitle vocabulary. The lack of vocabulary, combining with stemming or stopword removal, can potentially decimate words in the transcript, hence harm the retrieval result. Another phenomenon we observed are a significant performance gap between different retrieval models. TF-IDF retrieval model outperforms LM and Okapi retrieval models, which was unexpected. Since the dev data is a known item retrieval task (For each query, there is only 1 matching speech segment), we suspect that dev data might have some bias in favor of the TF-IDF retrieval model. Another possible factor for superior performance on the TF-IDF retrieval model is smoothing. Both LM and Okapi retrieval model relies on smoothing parameters, but there is no smoothing for the TF-IDF retrieval model. If the data have a good number of exact matching between query and documents, TF-IDF may outperform other retrieval models due to the absence of smoothing.
The experiments on test data are ad-hoc retrieval task, which is no longer restricted to one result per query. Due to the time constraint, we only submitted systems based on subtitle data. Our submissions use both word stemming and stopword removal, as this setup gave the most promising result on the dev data. Results on test data are in Table 3. MAP
TF-IDF 0.477 0.747 0.673 0.578 The performance gap between retrieval models is much smaller compare to dev data. Yet the trend is still the same: TF-IDF gives the best performance compared to other retrieval models. We suspect that the absence of smoothing contributes, and can explain the superior performance on TF-IDF. In a regular retrieval task, TF-IDF is not expected to outperform Okapi and LM consistently. While processing the experiments on the test data, we noticed a difference between dev and test queries. The number of words in dev queries was greater than for test queries. Originally we thought that this might be a factor affecting performance on different retrieval models, but it does not appear to be an issue. Still, we suggest the characteristics of queries in dev and test data should be more consistent, so that the datasets are better matched.
We find that the TF-IDF retrieval model is the best of the three models tested. We believe this is because it does not do smoothing. However, generally speaking, smoothing can provide significant improvement on the standard retrieval task. We conducted experiments with LM retrieval model without smoothing; the resulting MAP on dev data is less than 0.05. So we can only assume that TF-IDF retrieval model could possibly find the correct way for processing the absent query word on our data. The possible reason for the performance gap on dev data is query text length. TF-IDF (which relies on exact word matching) is stronger than the other approaches. The test data has much shorter query length, so the gap is not as great as we observed on the dev data.
Research in the Spoken Term Detection community suggests
using context for improving retrieval performance [
We examined how different retrieval models interact with different text processing techniques such as word stemming and stopword removal, on subtitles and ASR transcript, two different forms on the dev data. We find that stemming and stopword removal can provide persistent improvement on the subtitle data, yet for the ASR transcript, these processing mostly harm performance except for stemming on the TF-IDF retrieval model. The result on test data shows that the difference on retrieval methods is not that significant when the retrieval task contains more possible targets. TF-IDF still has the best performance, which we believe is due to the absence of smoothing technique.