The aim of the paper was to support the answer to a query with a ranked list of video segments (search sub-task) and to generate possible hyperlinks (in ranked order) to other video segments in the same collection that provide information about the found segments (linking sub-task). Our solution is based on concept enrichment i.e. the set of words is extended with their synonyms or other conceptually connected words. The other contribution is the content mixing using the combination of all transcripts and manual subtitles of the videos.
Our paper is about a user who searches for di erent
segments of videos within a video collection that address
certain topic of interest expressed in a query. If the user
nds the segments that are relevant to his initial information
need, he may wish to nd additional information connected
to these segments [
We used the same collection of the BBC videos as a source
for the test set collection. Collection of BBC consists of
video keyframes, audio content, 3 sets of automatic speech
recognition (ASR) transcripts: LIMSI/Vocapia [
During the tasks we developed a small system for processing the data. Our solution is solely based on textual analysis, we only used the subtitles and ASR transcripts. It has 5 distinctive stages: data normalization (2.1), shot cutting (2.2), concept enrichment (2.3), content mixing (2.4), indexing and retrieval (2.5).
The data set was given in various forms, so the rst step was to normalize the data formats and to convert all data to the same scale. We used the time dimension as scale and csv as the common data format. 2.2
Since in the data set each le represented a whole television program and we wanted to work on 'shot' level we created a tool, that based on the provided 'scenecut' description cuts each input data into shots. Using this method we created more than 300000 small les, each representing one shot with only one metric (like LIMSI transcript).
Our main goal was to create a concept enrichmented so called 'shot-document' le for each shot with each metric, by doing this the content can be found using synonyms in the search query. For example if the search query is "dog" and there is a shot-document which has the word 'puppy' in it, the aim is to connect them and return the needed result. 2.3
Our concept enrichment stage consists of three text transformation stages. First, each word in the shot-documents is analysed by the phpMorphy1 morphology engine. This engine can create the normal form (stem) of each word using basic grammatical rules and a large dictionary. In our work we replaced every word with its stem by this engine. In this point the shot-document is only a bag of words.
After this step we ltered out the stop words, we used 702 di erent English stop words2 for that, including search term like words e.g.: less, important. This way we narrowed down the word list of a shot-document to its core concept.
For a better match we needed to enrich this list with synonyms and conceptually connected other items. For this we used the well known ConceptNet 53 system, which can give us other words / phrases connected to each word in a shot-document. We experimented with a wider and a smaller range solution: including 50 and 10 conceptually connected words for each word, respectively. In the results the (C2) notates the smaller range result. We introduced a weight for each word, the "original" words inside the shot-document's weight is 1, the weight of connected words are lower (for wide range: 0.2, for small range: 0.1). At aggregation all of the enriched words there can be duplicates (like 'home' is connected to 'school' and 'teacher' is connected to 'school'), we aggregate them by a simple weight sum. Using this method the weight represents the importance of a word in the conceptual graph (sum of all words in the shot-document). 1http://sourceforge.net/projects/phpmorphy 2http://www.ranks.nl/stopwords 3http://conceptnet5.media.mit.edu/
We created multiple shot-document types (3 transcripts and manual subtitles), furthermore a combined type, so called "All transcript and subtitle". This later case was created by taking each shot-document with word weights and put together by the same sum method explained before. This way we could represent each and every possible word in our concept le, but on the other side we added a lot of conceptual noise to the originally clean document. 2.5
For indexing and retrieval we used Apache Solr4, and each shot-document is considered in Solr as a single continuous text stream, the order of the words represented the weight in the shot-document. Important note is that during the word reordering we kept concept phrases as one entity.
In the search sub-task the we only included the following steps: stop word ltering for the query, creating the norm form for each word in the query, using the query as a search input in Solr. The result was limited to 30 retrieved items.
In the linking sub-task we used the shot-document representing the needed section as a search query, but we removed the concept enriched words from it.
The whole dataset was more than 3700 hours of video and
the evaluation was on a shot level base (sometimes less than
5 seconds). Table 1 and 2 represent the results of the search
and linking sub-tasks, where P@N are precision oriented
metrics, such as precision at various cuto s (adjusted for
this task).[
LIMSI transcripts
LIUM transcripts NST/She eld transcripts All transcripts and subtitles
Manual subtitles (C2) LIMSI transcripts (C2)
LIUM transcripts (C2) NST/She eld transcripts (C2) All transcripts and subtitles (C2) 0.1778 0.1481 0.1630 0.1769 0.1517 0.3407 0.3111 0.3704 0.2846 0.1655 0.2000 0.1667 0.1444 0.1308 0.1345 0.3074 0.2926 0.2815 0.2231 0.1586 0.1407 0.1185 0.1148 0.0981 0.1017 0.2074 0.2204 0.2204 0.1692 0.1190 In the search sub-task we reached a quite stable result for each subtitle / transcript. Using a manually written transcript is much better since it can include visual clues, non-spoken informations (e.g. texts) and it has lower error rate, on the other hand in the transcripts there can be 'misheard' sentences.
In the linking sub-task the Manual subtitles gave us the best result, but it is interesting to note that for 2 of the anchors we cannot nd any relevant items among all of our results, that is why the P@N results are so low. These anchors are anchor 22 and anchor 27.
LIMSI transcripts
LIUM transcripts NST/She eld transcripts All transcripts and subtitles
Manual subtitles (C2) LIMSI transcripts (C2)
LIUM transcripts (C2) NST/She eld transcripts (C2) All transcripts and subtitles (C2) 0.0750 0.0444 0.0533 0.0400 0.0370 0.1818 0.0500 0.0526 0.0300 0.0143 0.0500 0.0333 0.0400 0.0467 0.0407 0.1000 0.0625 0.0316 0.0350 0.0250 0.0312 0.0167 0.0200 0.0233 0.0222 0.0500 0.0375 0.0184 0.0175 0.0196 The publication was supported by the TAMOP-4.2.2.C-11/1/KONV-2012-0001 project. The project has been supported by the European Union, co- nanced by the European Social Fund. 5.