Multimedia items account for an important part of Linked Open Data (LOD), but currently most of the semantic relations between multimedia items and semantic knowledge base (KB) are based on manual semantic annotation. With the popularity of multimedia hosting websites, plentiful tagging information makes it possible to automatically generate semantic relations between multimedia items and KB. In this paper, we propose a mechanism for linking multimedia items to KB with user-generated tags, while taking into account topical semantic similarity between tags. Experimental results show the effect of our work on this task.
Multimedia LOD has attracted considerable attention, but how to share and search
multimedia items on semantic web remains a significant yet challenging research
issue. Most of the semantic relations between multimedia items and semantic
ontologies are based on manual semantic annotation, which is very time-consuming. Some
researches try to automatically tag multimedia items based on their web page text [
Online multimedia hosting systems, such as Flickr, YouTube and Last.fm, have
attracted great attention in that they enable an effective way for users to organize, tag
and share multimedia items. Researches on understanding multimedia items based on
their tags have been attached importance [
We firstly detect tags which can be unambiguously linked to ontologies in highquality public semantic KB and process some ambiguous tags with simple tags' Cooccurrence relation, and then use the multimedia “item-tag” relation matrix to train topic models, through which to calculate topical semantic relations between tags, and then detect implicit semantic links between tags and semantic ontologies. Finally, we link multimedia items to semantic KB through tag-based mediation. In the following parts of this article, we will illustrate the details of the problem definition and proposed model, and report the experimental results.
Referring to [
1) Linking Tags to Ontologies: We firstly link all unambiguous tags to KB, which means that a tag has only one matched ontology definition in KB. For example, when T(mi) contains a tag “credit_card”, we can detect an unambiguous matched ontology as “dbr:creditcard”, then we add it to O(mi). However, when T(mi) contains a tag with multiple matched ontologies in KB, such as the tag “Apple”, we design a simple tags' Co-occurrence relation based method to determine linking an ambiguous tag to which ontology or not linking it to any ontology. The formula of co-occurrence relation (!!, !!) between an ambiguous tag !! and an unambiguous tag !! is given below:
U U C(tia , tku ) * C(tku , t uj ) R(tia , t uj ) = ∑ R(tia , tku , t uj ) = ∑ k=1 k=1 F (tku ) (1) where (!!, !!, !!) means partial co-occurrence relation between !! and !! created through !!. U means number of all unambiguous tags. (!!, !!) means co-occurrence frequency of !! and !!, and (!!) means frequency of !!. Finally, a !! with the maximum (!!, !!) of !! will be detected as a vicarious tag of !!. In particular, we also detect some tag-combined ontologies for expanding the links' range. For example, if an item has both “DigitalCamera” and “Conon” as its tags, we will check if there is a semantic ontology “DigitalCamera_Conon” or “Conon_DigitalCamera” in KB.
Latent Dirichlet Allocation model (LDA), have been proved to be powerful tools for
identifying latent topical information. We utilize JGibbLDA [
Three datasets of images, videos and music are respectively prepared: 1) mirFlickr
dataset [
For evaluating the effects of different β, we set β as positive integer between 1 and 6 and compare the performance with different β, since we empirically judge that two tags with Levenshtein Distance more than 6 are with little probability to be similar tags. We randomly choose 500 tag-couple with Levenshtein Distance equals to or smaller than 6 as the dataset for evaluating different β, and we roughly set the recall as 1.0 when β = 6 considering we are unable to collect all valid tag-couples. We firstly get the recall and precision results and further the F1-value to evaluate β. Table 1 shows the results. As shown, we get the best performance when β is 2. Therefore, we set β = 2 in the following experiments. 1 https://lod4all.net/zh/index.html 2 https://labrosa.ee.columbia.edu/millionsong/lastfm Precision 0.322 0.545 0.921 0.877 1.000 1.000 Recall 1.000 0.887 0.728 0.411 0.169 0.055
F1-value 0.487 0.675 0.813 0.560 0.289 0.104
Totally 144,318 tags are unambiguously mapped to dbo or dbr, which only account for 17.59% of all tags. To evaluate the effects of different d when β = 2, we manually label 600 multimedia items. We roughly set the recall as 1.0 when d = 1 considering we are unable to collect all possible related ontologies to an item. After getting both precision and recall with different d, we also use F1-value as evaluation criterion to evaluate d. Table 2 shows the results. As shown, the best performance shows when d = 3, which indicates that when σ =1-10-3 we can get best discriminant performance. Therefore, we set d = 3 in the following experiments.
With our model, 591,850 tags are finally mapped to KB, which account for 72.18% of all tags, which makes tremendous growth compared to 17.59%. We compare our model with two baselines on the tag similarity calculation subtask:
••Word2Vec based model (W2V): Word2vec is a recently popular method for getting distributed representation of words, of which the output form is similar to LDA.
••Co-occurrence based model (Co-occur): Co-occur is another method for detecting relationship between words, which doesn't consider latent semantic information.
Flickr Co-occur Our model W2V 0.654 0.841 0.401
W2V 0.388
We randomly choose 300 items from each dataset as test datasets. For each item, we manually check the validity of every created item-ontology link, which can help getting the precision easily for each model, and we take the union of all valid results of three different models as basis for getting recall of each model. Then the F1-value can be calculated and the results are shown in Table 3.
This paper is only a preliminary work. Named entity disambiguation and named entity normalization will be considered in our next step work. Besides, predicate discriminate should be performed by considering adjacent tags as context. 4 5