The rapid rate of information propagation on social streams has proven to be an up-to-date channel of communication, which can reveal events happening in the world. However, identifying the topicality of a short messages (e.g. tweets) distributed on these streams poses new challenges in the development of accurate classification algorithms. In order to alleviate this problem we study for the first time a transfer learning setting aiming to make use of two frequently updated social knowledge source (KS) (DBpedia and Freebase) for detecting topics in tweets. In this paper we investigate the similarity (and dissimilarity) between these KS and Twitter at the lexical and conceptual(entity) level. We also evaluate the contribution of these types of features and propose various statistical measures for determining the topics which are highly similar or different in KS and tweets. Our findings can be of potential use to machine learning or domain adaptation algorithms aiming to use named entities for topic classification of tweets. These results can also be valuable in the identification of representative sets of annotated articles from the KS, which can help in building accurate topic classifiers for tweets.
The dynamic changes in both vocabulary and style pose additional challenges for supervised classification algorithms, since the collection of annotated data becomes particularly difficult. However, frequently updated social knowledge sources(KS), such as DBpedia and Freebase, present an abundant source of structured data which could potentially aid in streamed topic detection. Similar to Twitter, these sources exhibit the following characteristics: i) they are constantly edited by web users; ii) they are social and built on a collaborative manner; iii) they cover a large number of topics; and iv) they provide plentiful amount of annotated data.
In this work we present for the first time a comparative study which analyses the similarity between Twitter and two frequently updated KS including DBPedia and Freebase. This comparative study includes the analysis of various cross-domain(CD) topic classifiers built on these KSs considering different lexical and conceptual features derived from named entities. Our intuition for the conceptual features is that the mention of certain entity types could be a good indicator for a specific topic. For e.g. a tweet containing the entity “Obama” is more likely to be a trigger for the topics “Politics” and “War&Conflict” than for the topic “Entertainment”. Similarly, “Lady Gaga” is more likely to appear in tweet messages about the topics “Entertainment” or “Music”, than about the topic “Sports”.
In addition, we propose different statistical measures for quantifying the similarity and differences between these KS and tweet messages. The main research questions we investigate are the following: i) Do KSs reflect the lexical changes in Twitter?; ii) Which features make the KSs look more similar to Twitter?; iii) How similar or dissimilar are KS to Twitter; and iv) Which similarity measure does better quantify the lexical changes between KS and Twitter?
The main contributions of this paper are as follows: i) we present a methodology for building CD topic classifiers for tweets making use of KSs; and ii) we present a comparative analysis exploring the similarity between KSs and Twitter at the level of words and named entities for CD topic classification;
In the remaining of the paper we briefly describe the DBpedia and Freebase KS, we then present the state-of-the-art approaches in topic classification of Tweets, then we describe the main methodology and present the results obtained. 2
Social Knowledge Sources: an overview of DBpedia and Freebase In this section we briefly review the main features of the DBpedia and Freebase KSs, highlighting the differences and similarities between them.
DBpedia1 is a structured knowledge base derived from Wikipedia2, the largest
collaboratively maintained encyclopaedia. The latest released, DBpedia 3.7, classifies 1.83
million resources into 740,000 Wikipedia categories and 18,100,000 YAGO2 categories.
For a given Wikipedia article DBpedia provides the following information [
Similarly, Freebase3 is a huge online knowledge base which users can edit in a similar manner as Wikipedia. The latest version of Freebase 4 comprises of 85 domains, more than 20 million entities and more than 10 thousand relations across a large number of these domains. In contrast to DBpedia however, in Freebase the source of articles include Wikipedia as well as other sources such as MusicBrainz, WordNet, OurAirports, etc 5. The classification of articles in Freebase is also slightly different; for a given Freebase article: i) a domain denote the topic of the article; ii) a type define a particular kind of entity such as person or location (for e.g. “Lady Gaga” is a Person); and iii) properties describe an entity (for e.g. “Lady Gaga” has a “place of birth”). Another notable difference between the two knowledge source is the level of deepness in the hierarchy for a particular category or topic. 3
DBpedia and Freebase KSs have been important knowledge sources in many classification tasks such as topic detection and semantic linking of Twitter messages. These approaches mostly employ traditional machine learning algorithms building a classifier on Twitter dataset and deriving useful features from KSs.
To date, to the best of our knowledge, no analysis has been done in exploiting these KSs for cross-domain (CD) topic classification of tweets and also in measuring the similarity between these KSs and Twitter. In the following section we thus provide a summary of the related work using these KSs for Twitter on topic detection and semantic linking.
Related Work on using DBpedia for Topic Classification of Tweets Ferragina et al.
[
Meij et al [
Mendes et al. [
Related Work on using Freebase for Topic Classification of Tweets Kasiviswanathan
et al[
This section describes three different steps required for the analysis presented in this paper. The first step, described in Section 4.1, consists on the compilation of datasets from KSs; the second step, described in Section 4.2, consists on the use of these datasets for the development of CD topic classifiers; and the third step consists on the introduction of similarity metrics that can characterise distributional changes between datasets. 4.1
Collecting Data from KS In this section we refer to our datasets, which will be further described in Section 5. The Twitter dataset consists of a collection of tweets, which were annotated with 17 different topics using the OpenCalais services. In order to compile a set of articles relevant to each of these 17 topics, from both DBpedia and Freebase KSs, we performed two steps. In the case of DBpedia, for a given topic, we SPARQL6 queried for all resources whose categories and subcategories are similar to the topic. For the returned resources we only kept the first 500 characters from the resources’ abstracts. In the case of Freebase, we downloaded the articles using the Freebase Text Service API7. Given a topic, we collected all the articles whose domain matched the topic 8. In addition, for some of the 6 http://www.w3.org/TR/rdf-sparql-query/ 7 http://wiki.freebase.com/wiki/Text_Service 8 The collection of domains are enumerated at http://www.freebase.com/schema topics (e.g. Disaster or War), which were not defined as domains in Freebase, we looked at all the articles containing these topics in their title. While this service also allows to download the full content of an article, similarly to DBpedia, we only considered the first paragraph up to 500 characters.
The following Subsection 4.2, describes how the DBpedia and Freebase datasets are used to built three different CD classifiers for detecting topics in tweets. 4.2
Building Cross-Domain(CD) Topic Classifier of Tweets We formally describe each dataset D as a tuple (X; F; P (X)) composed of a set of instances X, a set of features F and a marginal probability distribution P (X). Each instance x 2 X is represented by a vector of features x = (f1; ::; fm); fi 2 F . The possible topics y = fcatY1 ; : : : ; catYd g for an instance x can take values from Y 2 fcat1; : : : ; catkg. The goal of the classification then is to learn a model h : X ! Y from a set of annotated training data L = f(x1; y1); : : : ; (xn; yn)jxi 2 X; yi 2 Y g, that induces a non-injective map between X and Y such that multiple class labels can be assigned to the same instance - e.g. h(x1) = fcat1; cat2g.
A CD scenario consists of a source dataset DS = (FS ; XS ; P (XS )) –on which the classifier is built– and a test dataset DT = (FT ; XT ; P (XT )) –on which the classifier is evaluated–. As illustrated in Figure 1, in this paper we consider three cases for the source dataset. The first two cases aim to investigate the usefulness of DBpedia and Freebase KSs independently, and the third case combines the contribution of both KSs. Thus, the CD scenarios studied in the paper are described as follows: Scenario I (Sc.DB) consisting of sole DBpedia articles; Scenario II (Sc.FB) consisting of sole Freebase articles; and Scenario III (Sc.DB-FB) consisting of a joint set of DBpedia and Freebase articles. The test dataset in each case is the Twitter dataset.
Sc. DB
Sc. FB
Sc. DB-FB Retrieve articles
Concept enrichment Build Crossdomain Classifier
Retrieve tweets
Concept enrichment Annotate Tweets
We used as baseline classifier an SVM classifier with linear kernel, which has been
found to perform best for transfer learning [
Feature Extraction The performance of the machine learning algorithm rely on the feature representation employed. We propose two different feature sets for the examples in both train and test datasets: – a bag-of-word(BoW) representation: This representation captures our natural intuition to utilise what we know about a particular topic, so that the features which are most indicative of a topic can be detected and the appropriate label(s) assigned. This feature consists of a collection of words weighted by TF-IDF (term frequencyinverse document frequency) in order to capture the relative importance of each word. – a bag-of-entities(BoE) feature representation. The second set of features makes use of named entities. These entities were extracted by querying OpenCalais API9 for entity extraction on each instance belonging to the Dbpedia, Freebase and Twitter datasets as presented in Figure 1. We then used the dereferenceable URI and concepts returned by the API as features for the classifier. Our intuition is that entities can be characteristic of a topic, serving as trigger words for this topic; reducing in this way the lexical differences between the source and target datasets. 4.3
Measuring Distributional Changes Between KS and Twitter
In addition to building the CD classifiers, we investigated various measures for
quantifying the similarity between KSs and Twitter. When building a machine learning
classifier, it is expected that the closer the train dataset to the test dataset the better the
performance of the classifier [
To measure the similarity between the distributions of the presented datasets, let !d represent a vector consisting of all the features occurring on a dataset. Then, !ds denotes such a vector for the train dataset and !dt for the test dataset. In light with the feature set employed, the !ds and !dt contain the TF-IDF weight for either the BoW or BoE feature sets. Then the proposed statistical measures are: – the chi-squared ( 2) test: The 2 test measures the independence between the feature sets (FS and FT ) and the train and test datasets. Given the !ds and !dt vectors , the 2 test can be computed as 2 =
X (O
E
E)2
, where O is the observed value for a feature, while E is the expected value
calculated on the basis of the joint corpus.
9 www.opencalais.com/
– the Kullback-Leibler symmetric distance (KL): Originally introduced in [
KL(!dsjj!dt ) =
X f2FS[FT (!ds(f ) !dt (f )) log !ds(f ) !dt (f ) – cosine similarity measure: The cosine similarity represents the angle that separates the train and test vectors !ds and !dt : cosine(!ds; !dt ) =
PFS[FT (!ds(fSk )
k=1 PFS[FT (!ds(fSk ))2 k=1
!ds(fTk )) PFS[FT (!dt (fTk ))2 k=1 We also note that some of these proposed functions measure actual similarity (cosine), while others measure distance KL, 2. 5
The Twitter dataset consists of tweets posted between October 2010 and January 2011,
and was originally collected by [
DBpedia one thousand articles for each topic, comprising of 9,465 articles from DBpedia and 16,915 articles from Freebase. 71%
Dbpedia multilabel frequency
Freebase multilabel frequency
Twitter multilabel frequency 88.6% 88.6% 10 Available at http://wis.ewi.tudelft.nl/umap2011/
In line with previous approaches ([
Figure 2 shows the distribution of the examples belonging to multiple topics in each dataset. The Twitter dataset contain some tweets annotated with up to six categories, with the majority of them being annotated with only one topic. In the case of the Freebase dataset, due to the nearly flat hierarchical structure of the domains, the majority of the articles belong to a single category. In the case of the DBpedia dataset the majority of the articles belong to a single category, and less than 1% of the articles are annotated with 3,4,5,6,7 or 9 topics. The size of the vocabulary for each category and dataset is presented in Figure 3. This distribution presents a variation in the vocabulary size between the different datasets. Namely, in the DBpedia dataset each category is featured by a large number of words. This is expected, as the DBpedia articles are typically longer than the Freebase articles. The richest topics in DBpedia being Religion, EntCult, TechIT . In contrast, in the Freebase dataset the topics are being described by less words. The richest topics in Freebase are Sports, TechIT , HumInt. While for the Twitter dataset these topics are Env, DisAcc, BusFi.
When looking at the frequency of the entities in Figure 4, we can observe similar trends. The DBpedia articles contain the most number of entities for each topic, on average 22:24 1:44 entities. From the full collection 69(0.72%) of the articles do not have any entity. In the case of Freebase, the average number of entities per article is %#!!!" %$$#L" %%&%K'&#"%'%"%!IL" %&'%##!%"&L%"%%%J''&"'#%"!L%I#%I"!J%"%&K&" %%#K%" %%&%K'&#"KK%"%&K&" %!JII" %!!!!" $I%&!" $KIK'"
$%&$" $JJ'" $!!!" IJ%I'&"!I"I%&#!'"&L"IIJI$K"!K"IIJ#I#"J$"I&%'!&"#J"I$K!" I$L$" I#%J" I'!!" IJ#I" I$!I" I#&#" I#J'K&"LK"IJL'" II%I" IIK#!$"#K"I'$##K"J!" 'J%&" ()*+," -,*.//" 01)" 0234)53" 026" 78953:" 7;*<=8/" 7)>?23"
D;5" =85,E,;2"
F;/?**)8"
F<;B3*"
G8/:?G" =85,E,;2"
F;/?**)8"
F<;B3*"
G8/:?G" 8:14 5:78. The percentage of articles without any entity is 19.96%(3,377 examples). Lastly, the Twitter dataset contains the smallest number of entities, on average 1:73 0:35 entities per articles. The number of articles mentioning no entity is 5,137 (41%).
The heatmap in Figure 5 demonstrates how the entity types’ frequencies differ across different datasets. The darker the color, the higher the frequency of an entity in a dataset. According to this figure, Organization and Position have a relatively high frequency across all datasets. Other entities appearing frequently on these datasets are Person, Country and Natural Feature. Entity types such as MedicalCondition, or SportsEvent appear to be more representative of particular topics such as Health and Sports. When checking the clustering by topic in Figure 5, we can notice that the Health and Edu topics present a similar entity distribution in DBpedia and Freebase; the War topic ProvinceOrState Facility City IndustryTerm Company Continent Region SportsEvent MedicalCondition Technology Organization Position Person Country NaturalFeature )B )T )B )B )B )B )B )B )B )B )B )B )B )B )B )T )T )B )T )B )B )B )B )B )B )B )B )B )T )T )T )B )B )B )B )B )B )B )B )T )T )T )B )B )T )T )T )T )T )T )T .F TG .cF .ccF .ceF .eTTD t.unHDmt.raeheDWI.cssoueSD .LawD .raboLD i.cscADD .rcsepoRD ltt.cunFESC I.(ccehTTFS It(nuTHGmltt(unTECG ltt.(cnuCD I(cehTTTG i.(scuFFBS lt.(caehFSH lt.(eahSHD i.(suFSBD .(vnSED .(aLFSw .(vnFES i.(ccsFASD .r(ccsepFSR tr(haeeTGWi(sccTADG (vnTEG .(cudESD .(cduFES t.r(sopSD .(raSDW.tI(nFSml.(oFSP t.(rsopFS ili.(cngoFS tr(spoTSG ili(neogTG l(oTPG l.(coSPD i.(cnogSD lt(haeTHG r(csepTRG i(suFTBG r(boaLTG I(scsueTG (duTEG (aLTwG (rcS r(a r(S (S (S (cS (cS (cS (cS (cS (cS (cS (cS ( S c c c c c c c c c c c
Labo WaWIssueoS traehWIch I H E oH S uH S eR R ileR oH oS has a similar entity distribution in Twitter and Freebase; while the Pol category presents a similar entity distribution in Twitter and DBpedia.
Based on the above figures on lexical richness and entity frequency, thus, we can notice that the Freebase dataset exhibits more similarity to Twitter than Dbpedia datasets. In order to get a better insight into this similarity, we will compare these datasets according to the proposed measures in the coming section.
In this section we perform a series of experiments to investigate which KS exhibits more similarity to Twitter. In the first set of experiments we compare the performance of the SVM classifiers derived for the proposed cross-domain (CD) scenarios (Subsection 4.2), with the SVM classifier built on Twitter data only. These classifiers were trained using the different BoW and BoE features(Section 6.1) in each scenario. Therefore in this first set of experiments we address the questions of which KS reflects better the lexical variation in Twitter? and what feature makes the KSs look more similar to Twitter?.
The second set of experiments, consists on computing the correlation between the proposed statistical measures (Section 6.2) and the accuracy of the CD classifiers. In this correlation analysis we investigate which statistical measure presents the highest correlations with the accuracy of a CD classifier? providing the most reliable estimate for the quality of KSs in topic classification of tweets. 6.1
Comparison of the Different Feature Sets in Cross-Domain Scenarios The SVM classifiers derived from the CD scenarios –Sc.DB, Sc.FB and Sc.DB-FB– were evaluated based on their performance when trained using BoW and BoE features. The TGT SVM classifier –based on Twitter data only– was built on 80% of the Twitter data, and evaluated on 20% of the twitter data over five independent runs.
Figure 6 shows the results obtained using BoW and BoE features for the different CD scenarios. Based on the average performance in F1 measure, we can observe, that among the three CD scenarios, the best average performance was obtained by the Sc.DB-FB SVM classifier using BoW features, which is followed by the Sc.FB and Sc.DB SVM classifiers also using BoW features.
Using both feature sets, we found that for the Sc.DB-FB scenario the topics which presented a performance closer to the one obtained by the TGT classifier were the Weather and Edu. For the Sc.FB scenario these topics were the Edu, Weather, Labor. Finally for the Sc.DB scenario these topics were the Edu, Health. The topics for which the performance was higher using BoE features were the BusFi, Env, Pol, SocIssue, Sports. For Labor the performance was the same for both features .
A slightly different trend can be observe for the TGT classifier, where the best average F1 measure was achieved using BoE features. There were 10 topics for which BoE features were useful: DisAcc, EntCult , Env, Health, HospRec, HumInt, Religion, TechIT, War and Weather.
Overall, our results indicate that Sc.FB KS is more similar to Twitter than Sc.DB. Furthermore, combining the contribution of the Sc.DB and Sc.FB is beneficial for detecting topics in Tweets, since the Sc.DB-FB scenario achieves the best overall results. With regard to the features, we found that in 11 out of 17 cases the results obtained using BoW features were better, and in 5 out of 17 cases the BoE features were found more effective.
We also compared the performance of the Twitter classifier against the three CD classifiers over the full learning curve, by gradually increasing the number of tweets used to train the classifier. Our analysis revealed that in the majority of the cases the CD classifiers worked relatively well. That is, a sufficient amount of annotated tweets were needed to significantly outperform the three CD classifiers over the full learning curve. The number of annotations needed for each topic is summarised in Table 2. For e.g. for more than 9 out of 17 topics the necessary amount of annotated tweets need to exceed 900. However, in a real-world scenario annotating tweets is an expensive task.
Comparison of Statistical Measures in Topic Classification of Tweets In this second set of experiments we aimed to investigate our research question of how similar or dissimilar are social knowledge sources to Twitter posts; and which similarity measure does better reflect the lexical changes between KSs and Twitter posts?. We thus performed a comparison between the proposed KL divergence, cosine similarity and 2 test by measuring the correlation of these values with the performance of a CD classifier using Sc.DB, Sc.FB and Sc.DB-FB scenarios.
Each CD classifier was evaluated on 20% of the Twitter data, and the performance was averaged over five independent runs. The obtained F1 measures for the CD classifiers were then compared with the values obtained for the different statistical measures, and the Pearson correlation was computed.
Figure 7 show the correlations obtained using KL, cosine and 2 values. A positive correlation indicates that the performance of the CD classifiers increases as the divergence decreases (the distribution are more similar); while a negative correlation indicates that the performance increases as the divergence increases (the distributions are less similar). As we can notice, for the KL scores, there are 24 cases in which the correlation scores are higher than 70% in absolute terms. In the case of Cosine similarity these cases sum up to 25. While in the case of 2 values for a total of 32 cases were the correlation values higher than 70%.
Chi−TGT(BoE) Chi−Sc.DB−FB(BoE)
Chi−Sc.FB(BoE) Chi−Sc.DB(BoE)
Chi−TGT(BoW) Chi−Sc.DB−FB(BoW)
Chi−Sc.FB(BoW) Chi−Sc.DB(BoW)
Cosine−TGT(BoE) Cosine−Sc.DB−FB(BoE)
Cosine−Sc.FB(BoE) Cosine−Sc.DB(BoE)
Cosine−TGT(BoW) Cosine−Sc.DB−FB(BoW)
Cosine−Sc.FB(BoW) Cosine−Sc.DB(BoW)
KL−TGT(BoE) KL−Sc.DB−FB(BoE)
KL−Sc.FB(BoE) KL−Sc.DB(BoE)
KL−TGT(BoW) KL−Sc.DB−FB(BoW)
KL−Sc.FB(BoW) KL−Sc.DB(BoW)
Based on these results, we found the 2 to provide the best correlation scores for the usefulness of the KSs data. The second best score was for the cosine similarity, which was followed by the KL measure.
Figure 8 shows the pairwise similarity obtained for the source and target datasets according to ( 2) 1 similarity measure.11 As expected the closest datasets to the test Twitter dataset is the training set for the Twitter classifier (ChiSc.TGT). The second closest dataset according to 2 is the Sc.Fb dataset. The Sc.DB and Sc.DB-FB are then the less similar datasets to the test dataset. 7
In this paper we presented a first attempt towards understanding the usefulness of DBpedia and Freebase KSs in CD topic classification of tweets. We presented an analysis 11 As 2 measure distance rather than similarity we inverted its value to present the similarity between topics better. 8 6 4 2 between these data sources focusing on various lexical features (BoW) and entity features(BoE).
For a total of 17 topics we compiled a gold standard for each individual KS, and for the joint set of these sources. From the resulted datasets we then built three CD classifiers which we evaluated against a Twitter classifier using the different features.
Our analysis revealed that from the two KSs, Freebase topics seem to be much closer to the Twitter topics than the DBpedia topics due to the much restricted vocabulary used in Freebase. Furthermore, we found that the two KSs contain complementary information, i.e.; the joint dataset was found more useful than the individual KS datasets. With regard to the feature sets, we found that for the three CD classifiers on average the results obtained using BoW were better than those obtained with BoE in 5 out of 17 cases.
When comparing the results of these CD classifiers to the Twitter classifier we found
that for some of the topics the Twitter classifier required a large number of annotations
to outperform these classifiers, indicating that in the absent of any annotated tweets,
applying these CD classifiers is still beneficial. Previous research on transfer
learning has also shown, that outperforming the target (Twitter) classifier is extremely
difficult for many tasks including sentiment classification ([
Finally, we also looked at various statistical measures for predicting the usefulness
of the data gathered from these KSs. These experiments revealed the 2 test as
being the best measure for quantifying the distributional differences among between KSs
and Twitter. Our future work in this direction will focus in investigating more accurate
measures for quantifying this difference for e.g. by taking into account the special
vocabulary (abbreviations, misspellings, shortening) used in Twitter, and normalise this to
standard English terms ([