We enhance the accuracy of the currently available semantic hashtag clustering method, which leverages hashtag semantics extracted from dictionaries such as Wordnet and Wikipedia. While immune to the uncontrolled and often sparse usage of hashtags, the current method distinguishes hashtag semantics only at the word level. Unfortunately, a word can have multiple senses representing the exact semantics of a word, and, therefore, word-level semantic clustering fails to disambiguate the true sense-level semantics of hashtags and, as a result, may generate incorrect clusters. This paper shows how this problem can be overcome through sense-level clustering and demonstrates its impacts on clustering behavior and accuracy.
1 Introduction Hashtags are used in major social media (e.g., Twitter, Facebook, Tumblr, Instagram, Google+, Pinterest) for various purposes – to tell jokes, follow topics, put advertisements, collect consumer feedback, etc. For instance, McDonald’s created a hashtag #Mcdstories to collect consumer feedback; #OccupyWallStreet, #ShareaCoke and #NationalFriedChickenDay are only a few examples of many successful hashtag campaigns.
Twitter is the first social media platform that introduced hashtags, and is used as the representative social media in this paper. Most tweets contain one or more hashtags in their texts of up to 140 characters.
Clustering is commonly used as a text classification technique, and clustering of hashtags is the first step in the classification of tweets given that hashtags are used to index those tweets. Therefore, admittedly, classification of tweets benefits from accurate clustering of hashtags.
Social media is arguably the best source of
timely information. On Twitter alone, for
example, an average of 6000 micro-messages are
posted per second
The popular approach to hashtag clustering has
been to leverage the tweet texts accompanying
hashtags
In this regard, the focus of this paper is on leveraging dictionary metadata to identify the semantics of hashtags. We adopt the pioneering work done by Vicient and Moreno (2014). Their approach identifies the “lexical” semantics of hashtags from external resources (e.g., Wordnet, Wikipedia) independent of the tweet messages themselves. To the best of our knowledge, their work is the only one that uses this metadata-based approach. This approach has the advantage of being immune to the sparsity and poor linguistic quality of tweet messages, and the results of their work demonstrate it.
On the other hand, their work has a major drawback, in that it makes clustering decisions at the word level while the correct decision can be made at the sense (or “concept”) level. It goes without saying that the correct use of metadata is critical to the performance of any metadata-based approach, and indeed clustering hashtags based on their word-level semantics has been shown to erroneously putting hashtags of different senses in the same cluster (more on this in Section 4).
In this paper, we devise a more accurate senselevel metadata-based semantic clustering algorithm. The critical area of improvement is in the construction of similarity matrix between pairs of hashtags, which then is input to a clustering algorithm. The immediate benefits are shown in the accuracy of resulting clusters, and we demonstrate it using a toy example. Experimental results using gold standard testing show a 26% gain of clustering accuracy in terms of the weighted average pairwise maximum f-score (Equation 5), where the weight is the size of a ground truth cluster. Despite the gain in the clustering accuracy, we were able to keep the run-time and space overheads for similarity matrix construction within a constant factor (e.g., 5 to 10) through a careful implementation scheme.
The remainder of this paper is organized as follows. Section 2 provides some background knowledge. Section 3 describes the semantic hashtag clustering algorithm designed by Vicient and Moreno (2014). Section 4 discusses the proposed sense-level semantic enhancement to the clustering algorithm, and Section 5 presents its evaluation against the word-level semantic clustering. Section 6 discusses other work related to the semantic hashtag clustering. Section 7 summarizes the paper and suggests future work.
Concept Meaning desert.n.01 arid land with little or no vegetation abandon.v.05 leave someone who needs or counts on you; leave in the lurch defect.v.01 desert (a cause, a country or an army), often in order to join the opposing cause, country, or army desert.v.03 leave behind
Table 1: Example synset for the word “desert”. 2 2.1
Background
Wordnet – synset hierarchy and
similarity measure
Wordnet is a free and publicly available lexical
database of English language. It groups English
words into sets of synonyms called synsets. Each
word in Wordnet must point to at least one synset,
and each synset must point to at least one word.
Hence, there is a many-to-many relationship
between synsets and words
Table 1 shows an example synset. The synset contains 4 different concepts, where a concept is a specific sense of a word – e.g., “desert” meaning “arid land with little or no vegetation”, “desert” meaning “to leave someone who needs or counts on you”.
All of these concepts are linked to each other using the semantic and lexical relationships mentioned. For example “oasis.n.01”(meaning “a fertile tract in a desert”) is a meronym of “desert.n.01” i.e, “oasis.n.01” is a part of “desert.n.01”.
Given this network of relationships, Wordnet is frequently used in automatic text analysis through the application program interface (API). There are different API functions that allow for the calculation of semantic similarity between synsets, and the Wu-Palmer similarity measure (Wu and Palmer, 1994) is used in this paper in order to stay consistent with the baseline algorithm by Vicient and Moreno (2014). In a lexical database like Wordnet synset database, where concepts are organized in a hierarchical structure, the Wu-Palmer similarity between two concepts C1 and C2, denoted as simW P (C1, C2), is defined as simW P (C1, C2) = 2 · depth(LCS(C1, C2)) depth(C1) + depth(C2) (1) where LCS(C1, C2) is the least common subsumer of C1 and C2 in the hierarchy of synsets.
This Wordnet functionality is used to calculate the semantic similarity between hashtags in this paper, that is, by grounding hashtags to specific concepts (called “semantic grounding”) and calculating the similarity between the concepts. 2.2
Wikipedia is by far the most popular crowdsourced encyclopedia. Not all hashtags can be grounded semantically using Wordnet because many of them are simply not legitimate terms found in Wordnet (e.g. #Honda). This situation is where Wikipedia can be used to look up those hashtags. Wikipedia provides auxiliary categories for each article. For example, when Wikipedia is queried for categories related to the page titled “Honda”, it returns the following auxiliary categories. [Automotive companies of Japan’,
Companies based in Tokyo’, Boat builders’, Truck manufacturers’, ... ]
Auxiliary categories can be thought of as categories the page belongs to. In this example, if we are unable to look up the word “Honda” on Wordnet, then, through the help of these auxiliary categories, we can relate the term to Japan, Automotive, Company, etc. There are several open source Wikipedia APIs available to achieve this purpose – for example, the Python library “wikipedia”. 2.3
Hierarchical clustering is a viable approach to cluster analysis, and is particularly suitable for the purpose of hashtag clustering in this paper for a few reasons. First, the approach does not require apriori information about the number of clusters. (The number of outputs clusters is not known in most real applications.) Second, it is suited to the taxonomic nature of language semantics. Third, it facilitates a fair comparison with the algorithm by Vicient and Moreno (2014), which also uses hierarchical clustering.
There are two popular strategies for hierarchical
clustering – bottom-up (or agglomerative) and
topdown (or divisive). In bottom-up strategy, each
element starts in its own cluster and two clusters are
merged to form one larger cluster as the
clustering process moves up the hierarchy. In top-down
strategy, all elements start in one cluster and one
cluster is split into two smaller clusters as the
clustering process moves down the hierarchy.
Bottomup strategy is used in this paper because it is
conceptually simpler than top-down
For bottom-up strategy, several distance measurement methods are available to provide linkage criteria for building up a hierarchy of clusters. Among them, single-linkage method and unweighted pair group method with arithmetic mean (UPGMA) are used most commonly, and are used in this paper. Single-linkage method calculates the distance between two clusters Cu and Cv as d(Cu, Cv) =
min ui2 Cu ^ vj2 Cv dist(ui, vj ) and UPGMA calculates the distance as d(Cu, Cv) =
X
d(ui, vj ) ui2 Cu,vj2 Cv |Cu| ⇥ | Cv| where |Cu and |Cv| denote the number of elements in clusters Cu and Cv, respectively.
To generate output clusters, “flat clusters” are
extracted from the hierarchy. There are multiple
possible criteria to do that
Semantic Hashtag Clustering
We adopted the semantic clustering approach
proposed by Vicient and Moreno (2014) specifically
for hashtags. This approach uses Wordnet and
Wikipedia as the metadata for identifying the
lexical semantics of a hashtag. Source codes of their
algorithms were not available, and therefore we
implemented the approach described in Vicient’s
PhD dissertation
There are three major steps in their semantic clustering algorithm: (a) semantic grounding, (b) similarity matrix construction, and (c) semantic clustering. Algorithm 1 summarizes the steps.
In the first stage (i.e., semantic grounding), each (2) (3) Input: list H of hashtags Output: clusters Stage 1 (Semantic grounding): Step 1: For each hashtag h 2 H perform Step 1a.
Step 1a: Look up h from Wordnet. If h is found then append the synset of h to a list (LCh). Otherwise segment h into multiple words and drop the leftmost word and then try Step 1a again using the reduced h until either a match is found from
Wordnet or no more word is left in h.
Step 2: For each h 2 H that has an empty list LCh, look up h in Wikipedia. If an article matching h is found in Wikipedia, acquire the list of auxiliary categories for the article, extract main nouns from the auxiliary categories, and then, for each main noun extracted, go to Step 1a using the main noun as h.
Stage 2 (Similarity matrix construction): Discard any hashtag h that has an empty LCh. Calculate the maximum pairwise similarity between each pair of lists LChi and LChj (i 6= j) using any ontology-based similarity measure.
Stage3 (Clustering): Perform clustering on
the distance matrix (1’s complement of the
similarity matrix) resulting from Stage 2.
Algorithm 1: Semantic hashtag clustering
For each hashtag that was not found from Wordnet in Step 1 (i.e., of which the LCh is empty), it is looked up in Wikipedia. If a match is found in Wikipedia, the auxiliary categories (see Section 2.2) of the article are acquired. Main nouns from the auxiliary categories are then looked up in Wordnet, and if a match is found, we save the concepts by appending them to the list LCh; this step is repeated for each main noun.
In the second stage (i.e, similarity matrix construction), first, hashtags associated with an empty list of concepts are discarded; in other words, hashtags that did not match any Wordnet entry, either by themselves or by using word segmentation technique, and also had no entry found in Wikipedia are discarded. Then, using the remaining hashtags (each of whose LCh contains at least one concept in it), semantic similarity is calculated between each pair of them. Any ontologybased measure can be used, and Wu-Palmer measure (see Section 2.1) has been used in our work to stay consistent with the original work by Vicient and Moreno (2014).
Specifically, the similarity between two hashtags, hi and hj , is calculated as the maximum pairwise similarity (based on the Wu-Palmer measure) between one set of concepts in LChi and another set of concepts in LChj . Calculating the similarity this way is expected to find the correct sense of hashtag (among all the sense/concepts in LCh).
Finally, in the third stage (i.e., clustering), any clustering algorithm can be used to cluster hashtags based on the similarity matrix obtained in the second stage. As mentioned earlier, in this paper we use hierarchical clustering which was used in the original work by Vicient and Moreno (2014). 4
Sense-Level Semantic Hashtag
Clustering In this section, we describe the enhancement made to the word-level semantic hashtag clustering and showcase its positive impact using a toy example. Both Stage 1 (i.e, semantic grounding) and Stage 3 (i.e, clustering) of the sense level semantic clustering algorithm are essentially the same as those in the word-level semantic clustering algorithm (see Algorithm 1 in Section 3). So, here, we discuss only Stage 2 (i.e, similarity matrix construction) of the algorithm, with a focus on the difference in the calculation of maximum pairwise similarity. 4.1 Similarity matrix construction 4.1.1
Word-level versus sense-level similarity matrix As mentioned in Section 3, the similarity between two hashtags hi and hj is defined as the maximum pairwise similarity between one set of senses in LChi and another set of senses in LChj . (Recall that LCh denotes a list of senses retrieved from Wordnet to semantically ground a hashtag h.) This maximum pairwise similarity is an effective choice for disambiguating the sense of a hashtag and was used to achieve a positive effect in the word-based approach by Vicient and Moreno (2014).
However, we have observed many instances where a hashtag word has multiple senses and it introduces an error in the clustering result. That is, the word-level algorithm does not distinguish among different senses of the same word when constructing a similarity matrix and, as a result, two hashtags are misjudged to be semantically similar (because they are similar to a third hashtag in two different senses) and are included in the same cluster. Moreover, a false triangle that violates the triangular inequality property may be formed at the word level. (Note this property is required of any distance metric like Wu-Palmer.) See Figure 1 for an illustration. As its side effect, we have observed that a cluster tends to be formed centered around a hashtag that takes on multiple senses.
(a) Sense level. (b) Word level. (Edge weights denote similarity values (= 1 distance). Assume the minimum similarity threshold is 0.5. Then, at the sense level (a), two clusters ({H1, H2}, {H1, H3}) should formed because H2 and H3 are not similar (note 0.1 < 0.5), but, at the word level (b), one cluster {H1, H2, H3} is formed because it appears as if H2 and H3 were similar via H1. Moreover, the false triangle that appears to be formed at the word level violates the triangular inequality property because dist(H1, H2) + dist(H1, H3) < dist(H2, H3).) Figure 1: An illustration of clustering at the word level versus sense level.
Thus, we chose to explicitly record the sense in which a hashtag is close to another hashtag when constructing a similarity matrix. This sense-level handling of hashtag semantic distance helps us ensure that the incorrect clustering problem of wordlevel clustering does not happen. Accordingly, it avoids the formation of clusters that are centered around a hashtag that has multiple senses.
Algorithm 2 outlines the steps of calculating maximum pairwise similarity between hashtags in the word-level algorithm. One maximum pairwise similarity value is calculated for each pair of hashtags semantically grounded in the previous stage (i.e., Stage 1) and is entered into the similarity matrix. The similarity matrix size is |H|2, where H is the number of hashtags that have at least one sense (i.e., nonempty LCh). Note that the pairwise similarity comparison is still done at the sense level, considering all senses of the hashtags that are compared.
Input: set H of hashtags h with nonempty LCh.
Output: pairwise hashtag similarity matrix. 1 Initialize an empty similarity matrix
M[|H|, |H|]. 2 Initialize maxSim to 0. 3 for each pair (hi, hj ) of hashtags in H do 4 // Calculate the maximum pairwise similarity between hi and hj. 5 for each sp 2 LChi do 6 for each sq 2 LChj do 7 Calculate the similarity sim between sp and sq. if sim > maxSim then
Update maxSim to sim . 8 9 10 end 11 end 12 end 13 Enter maxSim into M[i, j]. 14 end
Algorithm 2: Word-level construction of semantic similarity matrix.
Algorithm 3 outlines the steps of constructing a similarity matrix at the sense-level algorithm. Unlike the case of the word-level algorithm, entries in the similarity matrix are between senses that make maximum similarity pairs between a pair of hashtags. Since these senses are not known until the maximum pairwise similarity calculations are completed, the construction of the similarity matrix is deferred until then. In the first phase (Lines 2⇠ 16), for each pair of hashtags, the alInput: set H of hashtags h with nonempty LCh.
Output: pairwise hashtag similarity matrix. 9 10 11 1 Create an empty list LHs of (hashtag sense
pair, pairwise maximum similarity). 2 for each pair (hi, hj ) of hashtags in H do 3 // Calculate the maximum pairwise
similarity between hi and hj. 4 Initialize maxSim to 0. 5 Initialize maxSimPair to (null, null). 6 for each sp 2 LChi do 7 for each sq 2 LChj do 8 Calculate the similarity sim between sp and sq. if sim > maxSim then
Update maxSim to sim .
Update maxSimPair to (hi.sp, hj .sq).
end 12 13 end 14 end 15 Add (maxSimPair, maxSim) to LHs. 16 end 17 // Construct the similarity matrix. 18 Count the number |Sˆ| of distinct hashtag
senses in LHs. 19 Initialize a similarity matrix M[|Sˆ|, |Sˆ|] as a 0 matrix. 20 for each triplet (hi.sp, hj .sq, maxSim) in LHs do 21
Update the M[m, n] to maxSim, where (m, n) is the matrix index for (hi.sp, hj .sq) . 22 end
Algorithm 3: Sense-level construction of semantic similarity matrix. gorithm saves the pair of senses (hi.sp, hj .sq) in the maximum similarity pair and the maximum similarity value in the list LHs. Then, in the second phase (Lines 18⇠ 22), for each triplet element (hi.sp, hj .sq, maxSim) in LHs, the algorithm enters the maximum similarity value maxSim at the matrix index corresponding to the pair of senses (hi.sp, hj .sq).
This two-phase construction of similarity
matrix brings two advantages. First, it enables the
algorithm to use exactly the needed number of
matrix entries for those senses that are distinct among
all senses that constitute pairwise maximum
similarities between hashtags. The size of the
matrix, therefore, is |Sˆ|2, where Sˆ is the set of
distinct senses in LHs (see Lines 18⇠ 19). Second,
it enables the algorithm to add exactly the needed
number of entries, that is, |H|2 entries (i.e., one
for each pair of hashtags (see Lines 20⇠ 22)) into
a matrix of size |Sˆ|2, where |Sˆ|2 > |H|2. (The
remaining entries are initialized to 0 and remain 0,
as they are for pairs of senses that do not represent
maximum similarity pair between any hashtags.)
Our observation is that the ratio |Sˆ|/|H| is limited
from 5 to 10 for most individual hashtags, which is
consistent with Vicient’s statement
Since what is clustered are hashtags, although their similarities are measured at the sense level, a number of interesting points hold. First, we do not need to add similarities between all pairs of senses in the similarity matrix. Second, a hashtag may appear in multiple clusters, where each cluster is formed based on distinct senses of the hashtag, and therefore the resulting clusters are overlapping. 4.2 A toy example To demonstrate the merit of clustering at the sense level as opposed to the word level, we made a toy set of hashtags and ran the metadata-based semantic clustering algorithm at both the word level and the sense level. The hashtags used are #date, #august, #tree, and #fruit. From Wordnet, we found that there were 3 senses associated with the word august, 13 senses with date, 5 senses with fruit, and 7 senses with tree.
Using the Wu-Palmer similarity measure (explained in Section 2.1) at the word level, we obtained the distance matrix shown below.
Hashtag august august 0.000 date 0.200 fruit 0.500 tree 0.667
Then, to perform clustering using the wordlevel distance matrix as the input, we used both the single-linkage and UPGMA (see Section 2.3) as the measure to calculate distance between newly formed clusters and the distance threshold for extracting flat clusters from hierarchical clusters was set to 0.5.
Table 2 shows the clusters obtained using the word-level clustering. We see that #august, #date, and #fruit are included in the same cluster in both cases of the distance measures. This example demonstrates a case in which #date takes on multiple sense identities and glues together #august and #fruit in the same cluster at the word level although these two are not similar at the sense level, as shown next.
Now, using the sense-level clustering, out of a total of 28 senses associated with the four hashtags, the algorithm picked 10 senses shown in Table 3. These 10 senses were picked as a result of maximum pairwise similarity calculations between two sets of senses belonging to each pair of hashtags. (With 4 hashtags, there are a maximum of 12 senses that can be obtained for 6 (= C(4, 2)) maximum similarity pairs, and in this example case, there were duplicate senses, consequently giving 10 distinct senses.) As mentioned earlier, each of these senses represents the semantics of the hashtag word it belongs to, and thus makes an entry into the similarity (or distance) matrix input to the hierarchical clustering algorithm.
The distance matrix obtained from the 10 senses is shown in Figure 2. The numbers in bold face are the maximum similarity values entered. Note that distance 1.000 means similarity 0.000.
Table 4 shows the resulting cluster assignments. (The outcome is the same for both distance measures, which we believe is coincidental.) We see that #august and #date are together in the same cluster and so are #date and #fruit but, unlike the word-level clustering result, the three of #august, #date, and #fruit are not altogether in the same cluster. This separation is because, at the sense level, #date can no longer take on multiple identities as it did at the word level. 5
Evaluation In this evaluation, all algorithms were implemented in Python and the experiments were performed on a computer with OS X operating system, 2.6 GHz Intel Core i5 processor, and 8 GB Sense Semantics august.n.01 the month following July and preceding September august.a.01 of or befitting a lord corner.v.02 force a person or animal into a position from which he can not escape date.n.02 a participant in a date date.n.06 the particular day, month, or year (usually according to Gregorian calendar) that an even occurred date.n.08 sweet edible fruit of the date palm with single long woody seed fruit.n.01 the ripened reproductive body of a seed plant fruit.v.01 cause to bear fruit tree.n.01 a tall perennial woody plant having a main trunk and branches forming a distinct elevated crown; includes both gymnosperms and angiosperms yield.n.03 an amount of product (‘n’ stands for noun, ‘v’ for verb and ‘a’ for adjective.) Table 3: Senses and their semantics (source: Wordnet).
Hashtag date tree fruit fruit august tree fruit date august date
Cluster Hashtag sense using single
linkage date.n.02 1 tree.n.01 1 yield.n.03 2 fruit.v.01 3 august.a.01 3 corner.v.02 4 fruit.n.01 5 date.n.08 5 august.n.01 6 date.n.06 6
Cluster using UPGMA 1 1 2 3 3 4 5 5 6 6 Evaluating clustering output is known to be a “black art” (Jain and Dubes, 1988) with no objective accuracy criterion. It is particularly challenging for the semantic hashtag clustering addressed in this paper. Sense-level clustering generates more items to be clustered than word-level and the output clusters are overlapping. Therefore, internal measures (e.g., Silhouette coefficient, SSE) are not desirable because they simply consider the cohesion and separation among the output clusters without regard to the semantic accuracy of the august.n.01 august.a.01 corner.v.02 date.n.02 date.n.06 date.n.08 fruit.n.01 fruit.v.01 tree.n.01 yield.n.03 items clustered. For this reason, our evaluation is done with an external “standard” (i.e., gold standard test). To this end, we use f-score, which is commonly used in conjunction with recall and precision to evaluate clusters in reference to ground truth clusters, as the accuracy metric. In our evaluation, the f-score is calculated for each pair of a cluster in the ground truth cluster set and a cluster in the evaluated algorithm’s output cluster set. Then, the final f-score resulting from the comparison of the two cluster sets is obtained in two different ways, depending on the purpose of the evaluation. For the purpose of evaluating individual output clusters, the pairwise maximum (i.e., “best match”) f-score, denoted as fm-score, is used as the final score. Given a ground truth cluster Gi matched against an output cluster set C, the fmscore is obtained as fm-score(Gi, C) =
max Cj2 C ^ f-score(Gi,Cj)>0 f-score(Gi, Cj ) (4) they match those used in the evaluation of wordlevel clustering by Vicient and Moreno (2014). They used tweet messages from the Symplur website, and so we did.
We manually gathered a tweet dataset from the
Symplur web site (http://www.symplur.com). The
dataset consists of 1,010 unique hashtags that are
included in 2,910 tweets. The median of the
number of tweets per hashtag was only two.
(Distribution of the number of tweet messages per hashtag
generally follows the power law
On the other hand, for comparing overall accuracy of the entire set of clusters, the weighted average of pairwise maximum f-scores, denoted as fascore, is used instead. Given a ground truth cluster set G and an output cluster set C, the fa-score is calculated as fa-score(G, C) =
PGi2 G(fm-score(Gi, C) ⇥ |
PGi2 G |Gi| 5.1.2 Dataset With the focus of evaluation on comparing between the sense-level and the word-level of the same clustering algorithm, deliberate choices were made in the selection of the datasets and the number of hashtags used in the experiments so that
The distance threshold for determining flat clusGi|) ters in hierarchical clustering was set using the “best result” approach. That is, we tried both dis(5) tance measures (i.e., single-linkage and UPGMA) and different distance threshold values and picked the measure and value that produced the best result based on the weighted average f-score measure.
Figure 4 shows the accuracies achieved by the metadata-based semantic clustering at the wordlevel and the sense-level. Table 5 shows more details, including precision and recall for individual clusters. From the results we see that every senselevel cluster outperforms the word-level counterpart (except cluster 1 due to rounding-off difference). Particularly, the fm-scores are zero for word-level clusters 6, 14, and 15, thus bringing the performance gain to “infinity”. (Word-level clustering did not generate any cluster of size 3 or larger and with the best match f-score to clusters 6, 14, and 15 greater than 0.1.) Further, when all 15 clusters are considered together, the weighted average of maximum pairwise f-scores, fa-score, is 0.43 for sense-level clustering and 0.34 for wordlevel clustering – a 26% gain.
There are several works on semantic clustering of
hashtags that focused on the contextual semantics
of hashtags
Lacking the ability to form lexical semantic sense-level clusters of hashtag has been a major shortcoming of the current approaches. To the best our knowledge, the work by Vicient and Moreno (2014) is the only one that opened research in this direction. They used Wordnet and Wikipedia as the metadata source for clustering hashtags at a word-level.
In this paper, we enhanced the current metadatabased semantic hashtag clustering algorithm by determining the semantic similarity between hashtags at the sense level as opposed to the word level. This sense-level decision on clustering avoids incorrectly putting hashtags of different senses in the same cluster. The result was significantly higher accuracy of semantic clusters without increasing the complexities of the algorithm in practice. A gold standard test showed that the senselevel algorithm produced significantly more accurate clusters than the word-level algorithm, with an overall gain of 26% in the weighted average of maximum pairwise f-scores.
For the future work, new metadata sources can be added to provide the metadata-based semantic hashtag clustering algorithm with more abilities. For example, to understand hashtags of a different language, online translation services like Google Translate (https://translate.google.com) can be a good source since empirical evidences suggest that it can be very effective in identifying spelling errors, abbreviations, etc. Additionally, crowdsourced websites like Urban Dictionary (www.urbandictionary.com) that specializes in informal human communication can be a helpful metadata source for decoding lexical semantics of hashtags. Internet search engines also provide rich information on the semantics of hashtags.
This project was supported by a Fulbright Program grant sponsored by the Bureau of Educational and Cultural Affairs of the United States Department of State and administered by the Institute of International Education.
Recall 0.63 0.35 0.39 0.91 0.41 0.21 0.64 0.25 0.82 0.18 0.40 0.11 0.22 0.13 0.17 fa-score (weighted average of fm-scores) is 0.43 for sense-level clusters and 0.34 for word-level clusters.
Table 5: Details of gold standard test results.
Zhibiao Wu and Martha Palmer. 1994. Verbs semantics and lexical selection. In Proceedings of the 32nd Annual Meeting on Association for Computational
Linguistics, pages 133–138.