In this paper, we explore the task of tag aggregations for the video and image les. In our work, we describe recent achievements image description generation, discuss tags and tagging. We present the result of our experiments based on the lexical resources and natural language processing approaches. In our work, we use the auto-tagging program from Imagga company as the generating program. As data, we use 5 videos which were split into shots for future processing. We identi ed two subtasks for tag aggregation: 1) creation of general set tags for the whole video le and 2) creation separate sets of tags for each video shots. Our detailed tag analysis and experiment results showed that pipeline with NLP methods received good results for the whole video, but for each shot, in future work, we need to use such resources as Nasari vectors, SensEmbed vectors or word2vec. We present all our experiments with graphs, tables etc. Our future work will be related to aggregation of video descriptions.
Online social networks are providing more and more convenient services to their users. Today, social networks have grown to be one of the most important sources for people, they are involved in all aspects of their lives. Meanwhile, every online social network user is a contributor to such large amounts of information. Online users like to share their experiences and to express their opinions on virtually all events and issues. Among a large amount of online user-generated data, we are particularly interested in peoples opinions or sentiments towards specic topics and events.
In social web there appeared many aspects for exploration of large multimedia datasets that have previously been unavailable. Popular social websites, such as Flickr, Photobucket, Picasa etc. contain a massive amount of visual photographs, which have been collectively tagged and annotated by members of the respective community.
Other sources of images are di erent professional stock image marketplaces. Stock photos are made by professional or semi-professional photographers and are usually contained in search databases. They can be purchased and delivered online.
Each of these photos independent of a source should have relevant tags, sometimes these tags from human, sometimes this is auto-tags. The problems in the tags area are tag generation, tag translation/tag disambiguation, image classi cation/clusterization based on tags etc.
However, along with the growth in the number of images on the Internet, the number of video content growth also. The advances in computer and network infrastructure together with the fast evolution of multimedia data has resulted in the growth of attention to the digital videos development. The scienti c community has increased the amount of research into new technologies, with a view to improving the digital video utilization: its archiving, indexing, accessibility, acquisition, store and even its processing and usability. Image and video processing are very close to each other and should be explored in parallel because the video can be divided into slots, where each slot represents an image.
The separate trend which is related to images and video is automatic
generation of image description (full sentences). However, the problem of video
description generation has several properties that make it especially di cult.
Authors in the work [
One of our goals is the construction of a system that optimizes the number of tags describing video resources, without any loss of sense. By using the textual information, a user is facilitated on the one hand to locate a speci c video and on the other hand is able to comprehend rapidly the basic points.
The speci city of this problem lies in the fact that we have many tags from the auto-tagging program, and the keywords are necessary for the whole video and also for shots of these videos, because a user may need to nd a fragment of some video. Therefore, this task is divided into two subtasks: 1) the optimization of keywords for the whole video le; 2) the aggregation of tag sets for separate shots.
The paper is organized as follows. In Section 2, we show the most recent related works for tag aggregation and in Section 3 we describe the Imaggas auto-tagging program that we use. The general approach, our methods for tag aggregation and results we analyze in Section 4. Section 5 concludes this paper. 2
The last three years have been associated with increased interest in the eld of generating descriptions and keywords/tags for images. This task involved both large companies like Google, Microsoft and small, which are pro led on certain domain areas such as Clarifai (clarifai.com), Imagga (imagga.com) etc. Also in this area, preliminary studies and prerequisites for such intensive development have been made, for example, the emergence of special collections of images, such as ImageNet and Microsoft COCO etc. All this has allowed achieving that in 2014, research scientists on the Google Brain team trained a machine learning system to automatically produce captions that accurately describe images. Further development of that system led to its success in the Microsoft COCO 2015 image captioning challenge, a competition to compare the best algorithms for computing accurate image captions, where it tied for rst place.
In the paper \Show and Tell: Lessons learned from the 2015 MSCOCO Image
Captioning Challenge" [
Another company Microsoft also has excellent results in this eld. This company presents some results of image auto-tagging and image captioning. You can see that each image from Fig. 1 has Description eld. This eld consists of tag set and caption with the value of con dence. The Tags eld also contains important tags with the relevant value of con dence.
In work [
2. This is followed by a generation phase that turns the detector outputs into words or phrases. These are then combined to produce a natural language description of the image, using techniques from natural language generation (e.g., templates, n-grams, grammar rules).
In paper [
Quite often, when the image contains a close-up object, Imagga's platform assigns correctly the most relevant tags to the central object (Fig. 2). In the right part of Fig. 2 keywords are ordered according to their relevance score. Associating external tags imports additional keywords in the annotation of Imagga's images. 4
We used ve fragments of lms for our experiments: Batmobile, FC Barcelona, Hunger Games, Meghan Trainor, Remi Gaillard. All these lms were divided into shots. The structure of these les you can see on the Table 1. We received sets of tags for all videos with using auto-tagging program from Imagga company.
The authors in the paper [
At the rst stage, we remove duplicates. There are quite a lot of them from 68% to 92% of all video tags. Then we process phrases which are found among the keywords. For example, if we have tags such as jelly, sh and jelly sh, then we leave only jelly and sh tags, and the jelly sh tag remove from the tag set. This choice is based on the fact that single tags have a higher score, and therefore more relevant to the image. At the next step, we get delete of in ective forms, such as playing, played etc. For English, this can be done using the popular Porter stemmer. It is giving the opportunity for to remove the repeat words. At the end of this process, we delete the keywords which are characterizing the color (example, blue, red etc.). They occur quite often, but they are not needed for video tagging.
The e ect from our re nement of image tags we will describe below. The Fig. 3 shows results for les Batmobile, FC Barcelona, Hunger Games, Meghan Trainor, Remi Gaillard.
The graphs show that the preliminary processing stage is su cient to
significantly reduce the number of tags that describe the whole video. However, we
have each tag has a relevant score, which allowed us to evaluate its e ect on the
number of tags. In [
We think that our results make it possible to consider this task to be performed at a fairly good level. But if the rst task about the total number of tags for the whole video le is clear, then the solution to the task of tag aggregation for separate shots cannot be solved so simply, because we have not repetitions. Consider the results for the movie "Batmobile" (Fig. 4). The pre-processing stage showed that for the video "Batmobile" we have 1,524 tags for all shots, and after all lters, we received 1,399 tags, i.e. reduced only 8% by the total. Each color represents one shot and for the le "Batmobile" we have 24 shots.
We analyzed our results and decided that we can use only tags with the relevant score>20%, but we got the results shown in Fig. 5. This gure shows that we have 6 shots that do not have any tags with score> 20% at all. This is not good since all fragments must have keywords. Of the remaining 18 shots, 11 of them have less than 10 tags, which is also few. All this showed that in this case, we cannot use the score value as a de ning characteristic. It also makes changes to the generation of a common set of tags for the whole video, since some shots will not be represented in the nal set. The low score is primarily due to the features of the algorithm work of the program Imagga auto-tagging Fig. 3. Quantity of tags that are considered redundant after pre-processing stage.
Tag Score Tag Score Tag Score car 100,00 businessman 32,00 team 23,53 man 40,94 convertible 31,88 wheel 23,39 container 39,63 automobile 31,82 sitting 23,17 petri dish 39,46 happy 29,57 meeting 22,93 vehicle 38,84 professional 28,92 job 22,57 dish 38,46 adult 28,75 success 22,22 people 37,02 smiling 28,14 attractive 21,86 business 35,38 person 27,89 women 21,66 auto 34,56 corporate 27,45 speed 21,64 businesswoman 34,22 work 26,39 tow truck 21,10 wheeled vehicle 33,65 transportation 26,21 drive 20,82 male 33,47 limousine 26,18 beaker 20,64 truck 33,40 crockery 24,66 suit 20,32 o ce 33,19 portrait 24,45 teamwork 20,06 caucasian 32,57 businesspeople 24,02
Tag Score Tag Score Tag Score swimsuit 70,54 caucasian 29,42 model 22,69 king 49,68 structure 28,36 summer 22,50 grass 44,63 people 27,00 happy 22,19 building 43,42 player 26,51 golfer 21,56 bikini 42,30 male 25,56 architecture 21,30
eld 34,65 adult 24,84 torch 21,14 beach 34,32 person 24,79 body 21,13 man 33,45 attractive 24,60 ocean 20,79 greenhouse 32,91 smiling 24,00 silhouette 20,16 rival 32,71 sea 23,33 maillot 20,14 sexy 29,98 art 23,23
Tag Score Tag Score Tag Score wheat 100,00 rural 29,18 hair 23,51 curtain 100,00 summer 29,00 adult 23,50 blind 95,77 water 28,72 landscape 23,43 furnishing 88,92 farm 28,06 icon 23,33 shower curtain 82,75 plant 27,66 straw 22,96 cereal 71,41 minaret 27,61 structure 22,85 protective covering 67,64 portrait 27,30 obstruction 22,53 fence 58,38 face 26,71 container 22,35 picket fence 56,33 blond 26,46 sign 22,28 eld 45,64 seed 26,25 design 22,24 crystal 41,96 fountain 26,20 male 22,20 ice 40,13 building 26,10 river 21,71 covering 39,68 attractive 25,98 natural 21,53 menorah 39,44 source of illumination 25,31 solid 21,45 candle 38,91 candlestick 24,47 glass 21,43 dam 37,31 crop 24,45 sky 21,40 grain 37,13 man 24,25 symbol 21,25 barrier 34,24 person 24,20 model 21,15 clock 34,16 corn 24,18 coral fungus 21,14 photograph 32,18 people 24,11 caucasian 21,04 candelabrum 31,56 timepiece 23,97 bread 20,59 agriculture 30,35 pretty 23,74 looking 20,21 harvest 29,87 cleaning implement 23,70 mosquito net 29,86 suit 23,64
Tag Score Tag Score Tag Score art 63,63 plaything 28,47 blind 23,68 blond 56,92 happy 28,39 cute 23,47 candle 50,18 smile 28,37 body 22,93 dress 47,26 makeup 28,35 quill 22,58 toiletry 46,45 shower curtain 27,80 lady 22,44 nipple 41,64 gymnastics 27,71 eyes 22,31 attractive 36,84 pajama 27,70 nightwear 22,15 hair 34,83 source of illumination 27,41 clothing 21,98 portrait 33,98 gown 27,03 letter opener 21,68 face 32,89 silhouette 26,91 light 21,42 cap 32,28 sexy 26,32 net 20,74 model 32,22 hair spray 25,90 expression 20,52 pretty 31,89 swing 25,19 man 20,51 adult 31,36 complexion 24,88 cheerful 20,45 curtain 30,82 fashion 24,79 mechanical device 20,40 person 30,61 lipstick 24,13 sunset 20,30 people 29,73 top 24,05 human 20,19 caucasian 29,06 swimsuit 23,99 lips 20,05 texture 28,54 furnishing 23,71 pen 20,03
Tag Score Tag Score Tag Score curtain 87,58 maillot 34,72 sexy 23,96 shower curtain 75,19 seaside 33,90 coastline 23,79 swimsuit 73,50 truck 33,24 van 23,76 beach 68,46 coast 31,06 outdoors 23,75 furnishing 68,17 shepherd dog 31,03 grass 23,47 blind 67,44 sky 30,84 structure 23,18 shore 58,56 pool 30,50 minibus 22,33 swimming 47,97 fountain 29,09 adult 22,30 sea 47,63 swimming trunks 29,03 sport 22,27 protective covering 47,01 walk 29,00 road 21,93 vessel 46,64 landscape 28,85 vehicle 21,68 ship 45,68 summer 28,41 jump 21,49 car 44,25 vacation 28,33 art 21,45 boat 44,13 reboat 28,28 ball 21,30 sand 42,51 german shepherd 28,23 swing 20,98 dune 42,10 travel 27,98 bus 20,92 bikini 41,88 dog 27,50 person 20,92 screen 40,04 male 27,09 minivan 20,92 ocean 39,66 man 26,95 fun 20,70 golf 39,63 covering 26,03 holiday 20,50 garment 37,09 sur ng 25,51 people 20,36 water 36,47 sun 24,26 seashore 20,10 door 35,92 clothing 24,08 Fig. 4. Quantity of tags that are considered redundant after preprocessing stage of each shot for Batmobile lm. program. But it is getting better every year and so we think that in the future all shots will have tags with a high score.
In future, we decided to make some experiments with Nasari vectors, SensEmbed vectors or word2vec for the aggregation of keywords for a single shot. 5
In this work we i) analyzed of systems which generate tags/descriptions for video and images; ii) compared the results of related works; iii) explored methods for generating natural language descriptions for images/video; iv) created short overview of the generation of video and image descriptions and explored main problems of this task. We concentrated on the problem of tag (keywords) aggregation into a single description of the object. Multimedia collections integrate electronic text, graphics, images, sound, and video. Their objects are usually annotated by keywords which characterize, describe or refer to categories in certain classi cations. These tags help to distinguish the objects and often form folksonomies: user-generated categories for organizing digital content. In this work, we showed how works the preprocessing stage for tag optimization of keywords sets for video fragments, using NLP techniques, lexical resources to tag aggregation. We presented the statistical information about our experiments and results.