Search engines traditionally su er drawbacks from ambiguities of natural language, which users often solve via query re nement. In contrast to web search, querying document collections of limited size (e.g. blogs, multimedia collections, or libraries) can quickly lead to empty result sets because the wrong choice of
Documents
2 Named Entity
Linking manual y verified
Query
Textual 1 Preprocessing
Stemming, Stopwords, etc.
Index Index Index
Traditional Retrieval Model Connectedness 4b Retrieval Model
Taxonomic 4a Retrieval Model
The search process of our Semantic Search system is depicted in Fig. 1. It rst
preprocesses documents and queries using state-of-the-art indexing methods (
With the goal to increase recall, the taxonomic approach determines documents containing entities that are not explicitly mentioned in, but strongly related to the query, e. g. if a document entity belongs to the same class as a query entity. We exploit these relationships to also identify documents that can serve as helpful recommendations if none or only few directly relevant documents exist, which is a frequent scenario when searching on limited document collections. The term vectors ti are constructed from the entity vector ei of the entity it represents and the set of classes c(ei) the entity is member of: ti = eei + c jvviij , with vi =
X w(cj ; ei) cj :
(
The ei and cj are pairwise orthogonal vectors with n dimensions, where each dimension stands for either an entity or a class. Since not every shared class means the same level of relatedness between two entities, not all classes should contribute equally strong to the similarity score. Assigning weights w(cj ; ei) that express the relevance of the class cj to the entity ei achieves this e ect. Traditional semantic similarity measures are most suited for this purpose if they consider class speci city, so we have used the measure proposed by Resnik [2]. The factors e and c de ne the contribution of entities compared to their classes and should incorporate normalization to keep unit length for the term vectors. With larger e, a document with few occurrences of the queried entity will be preferred over a document with frequent occurrence of related entities. The text index integrates into this model by appending the traditional document vector to the entity-based document vectors.
The second approach addresses the issue that term frequency is not always the most appropriate indicator of the relevance of a term (or entity) within a document. We propose to use connectedness weighting instead, which de nes an entity's relevance within a document based on how strongly it is connected within the document subgraph D. D includes all entities that are linked within the document as well as all entities from the knowledge base that connect at least two entities from the document. To obtain an undirected graph as required for connectedness calculation, the function rel(ei; ej ) is applied. It returns true 1 http://www.mpi-inf.mpg.de/departments/databases-and-informationsystems/research/yago-naga/yago/ i there exists a relation from ei to ej or from ej to ei. Each entity ei 2 D has a set Ei of directly connected entities and a set Fi of indirectly connected entities:
Ei = fe 2 Djrel(e; ei)g and Fi = fe 2 Dj9x : rel(e; x) ^ rel(x; ei)g Based on these sets, connectedness is calculated as follows: cn(ei; d) = 1 + (jEij + jFij) jDj , where nd = X jEj j + jFj j: nd ej2D
Entities may have no connections to any other entities in the document subgraph. Since they are nevertheless relevant to the document, we add 1 to all scores. The score is normalized by the average number of connected entities over all e 2 D (jDj=nd) to create comparability between di erent documents. This is otherwise lacking because entities are more likely to be connected to other entities in documents with more annotations. In addition, a single connection to another entity is more signi cant in a sparse document subgraph than in a dense one.
Whether or not a word or entity has a large power to distinguish relevant
from non-relevant documents depends on the corpus. Therefore, we keep the
traditional Inverse Document Frequency (IDF) to calculate a term's distinctness.
The entity vectors' values are thus cn-idf values, i.e. w(ei; d) = cn(ei; d) idf (ei).
(
There are currently no datasets that provide semantically annotated documents and queries with relevance assessments, so we have compiled a new dataset from 331 texts. They have an average length of 570 words, with 3 to 255 manually revised annotations. We also assembled and manually annotated a set of 35 queries.
For every query, the top 10 ranked documents from text-, class- , and
connectedness search were presented to users in random order. The users were asked
to assign every document to one of the ve categories based on its relation to
the query: Document is relevant (corresponding to a score of 5), parts are
relevant (
Tab. 1 shows that the inclusion of semantic annotations and similarities clearly improves retrieval performance compared to the text search baseline. The taxonomic approach not only increases recall, but also improves the ranking quality, measured by Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (NDCG).
The connectedness approach performs better than text search, but worse than the other semantic methods, including the simple \Concept+Text" approach, 2 The ground truth dataset is published at: http://s16a.org/node/14 where entities are treated as regular index terms within Lucene's default model. This is surprising because when we let the users directly compare the rankings produced by the three approaches, connectedness performed best. The evaluators had to identify the best (2.0) and second-best (1.0) rankings, which resulted in an average score of 1.09 for connectedness, followed by 1.01 for the taxonomic approach and 0.90 for the baseline. This seeming contradiction hints at a difference between information retrieval evaluation measures and user perception of ranking quality. The evaluators seem to have judged mainly by the very top few documents. Connectedness outperforms the other approaches in this respect, as shown by the reciprocal rank and precision@1 (Tab. 1). Also, connectedness performs best when related documents are not considered relevant. Combining the connectedness measure with tf weights leads to clear improvements. 4
Both proposed methods seem to achieve improvements over traditional text retrieval. Open questions, which are to be answered in future work, include how well the models would perform with other knowledge bases (e.g. Wikidata), what other semantic relations between entities are valuable for document retrieval, and how the semantic similarity can be given more in uence. The annotation of queries with classes may improve the retrieval, and so could the combination of the two proposed methods. Furthermore, the main ideas could be transferred to an adapted Language or Probabilistic Retrieval Model.