We address the problem of enhancing the classical bag-ofwords representation of texts by designing and engineering Tagme, the rst system that performs an accurate and on-the- y semantic annotation of short texts via Wikipedia as knowledge base. Several experiments show that Tagme outperforms state-of-the-art algorithms when they are adapted to work on short texts and it results fast and competitive on long ones. This leads us to argue favorably about Tagme's application to clustering, classi cation and retrieval systems on challenging scenarios like web-snippets, tweets, news, ads, etc..
The typical IR-approach to indexing, clustering, classi cation and retrieval, just
to name a few, is that based on the bag-of-words paradigm. In recent years a
good deal of work attempted to go beyond this paradigm with the goal of
improving the search experience on (unstructured) textual data. In our work we
are concerned with the task of adding structure to unstructured data, consisting
of the identi cation of sequences of terms (aka spots) in the input text and their
annotation with a Wikipedia page. This annotation process provides a stunning
contextualization of the input text so that each subsequent IR-task could be
improved by leveraging the huge semantic network provided by Wikipedia.
Recently several works (see e.g. [
We add to this ow of work the specialty that the input texts to be annotated are short, namely, they are composed of few tens of terms. The context of use we have in mind is the annotation of either the snippets of search-engine results, or the tweets of a Twitter channel, or the items of a news feed, or the posts of a blog, or the advertisement messages, etc.. It is easy to argue that these poorly composed texts pose new challenges in terms of e ciency and e ectiveness of the annotation process, which (1) should be very fast, because in those contexts data may be retrieved at query time and thus cannot be pre-processed, and (2) should be designed properly, because the input texts are so short that it is di cult to mine signi cant statistics that are rather available when texts are long. To address these issues, we have designed and implemented Tagme the rst software system that, on-the- y and with high precision/recall, annotates short texts with pertinent hyper-links to Wikipedia pages.
As an example, let us consider the following news: \Diego Maradona won against Mexico". Our goal is to detect \Diego Maradona" and \Mexico" as spots, and then hyper-link them with the Wikipedia pages which deal with the ex Argentina's coach and the football team of Mexico. Tagme uses as spots (to be annotated) the sequences of terms composing the anchor texts which occur in the Wikipedia pages, and it uses as possible senses for each spot the (possibly many) pages pointed in Wikipedia by that spot/anchor. Tagme selects among the potentially many available mappings (spot-to-page)1 the most pertinent ones by nding a collective agreement among them via new scoring functions which are fast to be computed and accurate in the nally produced annotation.
A preliminary description of Tagme has been published as poster in Procs
ACM CIKM 2010. Tagme is available for test at http://tagme.di.unipi.it.
What follows is a sketch of the main ideas and experimental results concerning
with Tagme, the interested reader is invited to read [
The annotation process of Tagme is composed by two main phases: Anchor disambiguation and Anchor pruning.
Anchor disambiguation. This is the task that judiciously cross-references
each anchor a 2 AT found in the input text T with one pertinent page pa of
Wikipedia. Tagme selects the best association a 7! pa by computing a score for
each possible page pa linked to a in Wikipedia (we call Pg(a) this set) that is
based on a new notion of \collective agreement" between the page pa and the
pages that can be associated to all other anchors detected in T , i.e. the anchors
in AT . This agreement is evaluated by means of a voting scheme that computes
for each other anchor b 2 AT n fag its vote to the annotation a 7! pa. Given
that b may be linked to many pages in Wikipedia (i.e. jPg(b)j > 1) we compute
this vote as the average relatedness between each page pb, potentially linked to
b, and the sense pa we wish to associate to a. However we argue that not all
possible pages of b have the same (statistical) signi cance, so we weight each
relatedness with the commonness of the page pb with respect to b (denoted as
Pr(pbjb) and computed as the prior probability that b points to pb over all links of
b in Wikipedia). Hence the voting given by anchor b to the annotation a 7! pa is
vb(pa) = jPg1(b)j Ppb2Pg(b) rel(pb; pa) Pr(pbjb) where the relatedness rel(pb; pa)
between the two Wikipedia pages pa and pb is computed as suggested in [
Anchor pruning. This step detects and possibly prunes some of the candidate annotations produced by the Disambiguation phase, if they are considered to be not meaningful. These \bad annotations" are detected via a simple, yet effective, scoring function that takes into account only two features: (1) the link 1 \Diego Maradona" is the name of two persons and \Mexico" points to 154 di erent pages in Wikipedia. probability lp(a) of the anchor a, computed as the number of times that a is used as anchor divided by the occurrences of a in Wikipedia (as anchor or not) and (2) the coherence between its candidate annotation a 7! pa (assigned by the Disambiguation Phase) and the candidate annotations of the other anchors in T , computed with the relatedness function presented above. This pruning score, say (a 7! p), is then compared against a properly set threshold na, so that if (a 7! p) < na then that annotation for a is discarded. The parameter na allows to balance recall vs precision of the annotation process and we deeply investigated its impact in our experiments. 3
We evaluated Tagme over a set of short texts randomly drawn from Wikipedia,
composed by about 20 terms (like web-snippets), and containing an average of
about 10 spots. We compared the annotation produced by Tagme against the
links attached by Wikipedia editors. We also evaluated Tagme on the dataset
proposed in [
F-measure of about 78% with an absolute improvement of more than 8%; { on long texts, Tagme resulted competitive (if not superior) with respect to M&W's and Chakrabarti's systems even if our \shift-based" approach clearly gives advantages to our competitors that deploy the full input text; { a deep evaluation on the scalability and time e ciency of Tagme showed that it is able to annotate a short text of about 10 spots in 18ms, while M&W's and Chakrabarti's systems take about 95 ms and > 2 sec, respectively, on comparable hardware. 4
We believe that Tagme has implications which go far beyond the enrichment of a text with explanatory links. The most interesting bene t of this annotation process is the structured knowledge attached to textual fragments that let us to leverage the semantic network provided by Wikipedia to improve IR-tasks which nowadays are mainly addressed with the bag-of-words paradigm, with all its well-known limitations.
We are currently investigating the impact of Tagme's annotation onto several
application domains:
2 A deeper evaluation is presented in our technical report [
Finally, we are studying the impact of other tools and information sources
in Tagme to better relate and/or assign pages/senses to text anchors (e.g.
Natural Language Processing tools, other by-products of Wikipedia| such as
DBpedia.org or YAGO| or the huge amount of structured informations provided
by the W3C SWEO Linking Open Data project [