This paper presents a method for identifying concepts in microposts and classifying them into a predefined set of categories. The method relies on the DBpedia knowledge base to identify the types of the concepts detected in the messages. For those concepts that are not classified in the ontology we infer their types via the ontology properties which characterise the type.
The concept spotting stage analyses the micropost for extracting the keywords that are candidates for being concepts, or can serve as context for disambiguating the concepts. The stage is executed in three steps, namely:
The text normalisation step converts the text of the micropost, that often
includes metalanguage elements, to a syntax more similar to the usual natural
language. Previous results demonstrate that this normalisation step improves
the accuracy of the part-of-speech tagger [
After normalising the micropost text, we execute the part-of-speech analysis of
the normalised text. For doing so, we make use of Freeling [
Once the part-of-speech tagging is obtained, the keyword selection step is executed. For each sentence within the micropost text we extract all the possible n-grams. In this case a gram is a word in the sentence. After that we select only the n-grams that satisfy the following criteria: – The n-gram contains at least one noun. – The n-gram is not contained in a set of stop words. – If the number of words included in the n-gram is greater than one, the n-gram is included in the set of Wikipedia article titles. – The n-gram is not contained in another n-gram that has been added to the keyword set (longer n-grams prevail).
To speed-up the process of querying the millions of Wikipedia article titles
we have uploaded the list of titles (available at [
To identify the semantics of the keywords we tap into the DBpedia knowledge
base [
Our process starts by identifying for each keyword the DBpedia resource which represents its intended meaning. Once we have the corresponding resource we query in DBpedia its classes, whenever they are available, or infer them through the identification of specific resource properties, so that we can identify the types defined in the challenge. 3.1
First we query DBpedia for a resource with a label matching the keyword. We use exact string matching between the resource label and the keyword which has been previously modified to fit the style of article titles in Wikipedia. The output resource of this query represents the most frequent meaning of the keyword defined by Wikipedia editors. We call this resource default sense of a keyword. If the resource is not related to a disambiguation resource, we consider that the term is not ambiguous and therefore we use the default sense as the one representing the keyword meaning.
In case we do not find a match between the keyword and a resource label we use an spelling service that suggests similar titles of Wikipedia articles. This spelling service3 compares the n-grams based on characters of both keywords and article titles, and takes into account the popularity of the articles in Wikipedia (i.e., the times that an article has been linked from other articles) when producing the final ranking of suggestions. We use the most similar suggestion, above a given threshold, for searching for the DBpedia resource.
If the resource is related to a disambiguation resource, then we have to select the proper sense among the candidates. To do so we leverage the correspondence of DBpedia resources with Wikipedia articles to obtain textual descriptions of each resource. Thus, we calculate similarity between each resource and the term by comparing the resource textual description with the term context. The most similar resource is selected as the resource representing the term meaning. By context we mean the set of keywords identified in the same sentence.
To calculate similarity between the keyword context and the resource
description we use a vector space model. The components of the vectors are the
most frequent terms of the Wikipedia articles related to each candidate DBpedia
resource. To populate the vectors representing resources we use term frequency
3 The spelling service was built upon Lucene [
In short for ambiguous keywords if there is not context and there is a default sense we select the DBpedia resource corresponding to the default sense. If there is context and default sense, and the context do not overlap with any of the candidate vectors we use the DBpedia resource corresponding to the default sense too. If there is overlap between the context and candidate vectors we use the most similar candidate. 3.2
We manually select the classes from DBpedia and linked ontologies that allow us to identify the types of the concepts defined in the challenge. For instance, – dbpedia-owl:Person and foaf:Person are the classes for People; – dbpedia-owl:Place is the class for Location; – dbpedia-owl:Organisation, dbpedia-owl:Company, and umbel:Organization are the classes for Organizations; – dbpedia-owl:ProgrammingLanguage, umbel:SoftwareObject, dbpedia-owl:Film, dbpedia-owl:TelevisionShow, dbpedia-owl:Award, and dbpedia-owl:SportsEvent, are the classes for the Miscellaneous type.
Therefore, for each DBpedia resource we obtain its class from the ontology and classify it according to the challenge types.
However, many DBpedia resources are not classified in the ontology. For those resources we infer its type from certain properties which are characteristic of the type. For instance, from a triple <subject> dbpedia-owl:birthPlace <object>
we can infer that object is a location given that it is the birth place of the subject described in the triple. The same rationale can be used with predicates such as dbpedia-owl:hometown and dbpedia-owl:location. Similarly, from a triple <subject> dbpprop:mvp <object>
we can infer that the subject is an sport event since it has a most valuable player. Other predicates used for identifying sport events include dbpprop:menDraw, dbpprop:teams, dbpprop:sport, and dbpprop:referee.
Finally, in case we cannot identify the concept type using DBpedia, we use a list of concepts and their types which have been collected from the training data set. From this list we take the first type associated with that concept.
We have not included evaluation results in this extended abstract since the only available source of annotated data for the evaluation in this challenge was the the training data set (the test data set was not annotated). Given that our approach uses a list of concepts gathered from the training set is not fair to report evaluation results on this data set.
This research is partially supported by the Spanish Centre for the Development of Industrial Technology under the CENIT program, project CEN-20101037, “Social Media” (http://www.cenitsocialmedia.es).