Traditional Information Retrieval (IR) systems are based on bag-of-words representation. This approach retrieves relevant documents by lexical matching between query and document terms. Due to synonymy and polysemy, lexical methods produce imprecise or incomplete results. In this paper we present how named entities are integrated in SENSE (SEmantic N-levels Search Engine). SENSE is an IR system that tries to overcome the limitations of the ranked keyword approach, by introducing semantic levels which integrate (and not simply replace) the lexical level represented by
In recent years a lot of attention has been invested on
Named Entities (NE), and their informative and
discriminative power within documents. Due to the importance
of research on NE, several sub-areas arose, such as entity
detection and extraction, entity disambiguation and entity
ranking. The typical information extraction task involving
NE is Named Entity Recognition (NER). This task has been
defined for the first time during the Message Understanding
Conference (MUC) [
Named entities are phrases that contain the names of
persons, organizations, locations and, more generally, entities
that can be identified by proper names. In order to
identify named entities in a text, several methods can be applied
such as Rule-based, Dictionary-based or Statistical ones. We
adopted a statistical method exploiting YamCha2, a generic
open source text chunker useful for a lot of NLP tasks.
YamCha adopts a state-of-the-art machine learning
algorithm called Support Vector Machines (SVMs), introduced
by Vapnik in 1995. We trained YamCha using the dataset
provided by CoNLL-2003 organization during the
SharedTask 2003 [
Building the entity level requires three steps:
1. pre-processing and entity extraction: XML files
2http://chasen.org/ taku/software/YamCha/
provided by CLEF 2008 organizers are processed in
order to extract entities. Named entities are stored
in IOB2 format. In IOB2, words outside the Named
Entity are tagged with O, while the first word in the
entity is tagged with B-k (to begin class k), and further
words receive the I-k tag, indicating that these words
are inside the entity;
2. entity indexing: entities extracted in the previous
step are stored into an index using Lucene. The entity
extraction procedure allows to obtain an entity-based
vector space representation, called bag-of-entities (BoE).
In this model an entity vector, rather than a word
vector, corresponds to a document.
3. Semantic Vector building: in this step semantic
vectors are built by exploiting the Lucene index. The
main idea behind models based on Semantic Vectors [
For the evaluation of the system effectiveness, we used the CLEF Ad Hoc WSD-Robust dataset derived from the English CLEF data, which comprises corpora from “Los Angeles Times” and “Glasgow Herald”, amounting to 166, 726 documents and 160 topics in English and Spanish. The relevance judgments were taken from CLEF. The goal of the evaluation was to prove that the combination of three indexing levels outperforms a single level. In particular, that adding the entity level increases the effectiveness of the search with respect to the keyword and meaning levels. To evaluate system effectiveness, different runs were performed by exploiting a single level at a time, or a combination of two or more levels. Each experiment is identified by the names of the used levels. To measure retrieval performance, we adopted Mean-Average-Precision (MAP) and Geometric-Mean-Average-Precision (GMAP) calculated by trec eval 0.8.1, a simple program supplied by the Text REtrieval Conference organizers3, on the basis of 1,000 retrieved items per request. Table 1 shows the results for each run, with an overview on the exploited features.
The results confirm our hypothesis: named entity recognition, in conjunction with an IR model capable of expressing semantics, can greatly improve the retrieval performance. If evaluated individually, the entity level does not yield to satisfactory results. This result is due to the presence of topics in which no entity was recognized. Conversely, when search is performed by making use of multiple levels, the entity level is able to improve performance even on those (difficult) topics for which few relevant documents are returned. This result suggests that named entities play a key role in increasing the number of retrieved relevant results previously ignored. Specifically, considering the experiment K+M+E where we used all three levels, an improvement of 14.5% in the MAP and 33.3% in the GMAP was observed. Generally speaking, we noted an overall improvement in all the experiments that used the entity level, compared to the equivalent experiments in which that level was not exploited. 4.