In this article we describe the Oslo University College's participation in the INEX 2013 endeavor. This year we participate in the Book Track's "Prove it" (as in 2012) and Social search tasks, as well as the Linked Data track's Ad-hoc task. In 2011 and 2012, the OUC submitted retrieval results for the "Prove It" task with traditional relevance detection combined with detection of confrmation based on speci city detected through the Wordnet concept hierarchy, as well as some rudimentary experiments based on named entity recognition . In line with our belief that proving or refuting facts are di erent semantic aware actions of speech, we have this year attempted to incorporate some semantic support based on Named entity recognition. For the Social search task, we wish, at this point, to examine the potential utility of Amazon assigned browse nodes, a hierarchy of tags assigned to item for sale purposes. For the Linked data task, we wish to explore the possibility of using categories to detect relevant articles.
On the surface, this may be seen as simple retrieval, but proving a fact is more than nding relevant documents. This type of retrieval requires from a passage to "make a statement about" rather than "be relevant to" a claim, which traditional retrieval is about. The questions we posed in 2010 were: { what is the di erence between simply being relevant to a claim and expressing support for a claim { how do we modify traditional retrieval to reveal support or refutal of a claim?
We have also made the claim that "Prove It!" sorts within the (admittedly not very well-de ned) category "semantic-aware retrieval", which, for the time being will be de ned by us as retrieval that goes beyond simple string matching, and is aware of the meaning (semantics) of text.
Those questions, being rhetorical in part, may be augmented by the question: { How can one detect the meaning of texts (words, sentences and passages) and incorporate those in the retrieval process to attain semantic-aware retrieval and consequently Finally { can one exploit technologies developed within the semantic web to improve semantic-aware retrieval { can we build a mathematical or statistical model that captures the proving properties of a document in relation to a statement?
The semantic-web path is not directly addressed in this paper, but we claim that the techniques used here point in that direction. 1.1
The prove-it task is currently being subjected to changes, one of which is the exclusion of pages from inappropriate books from the relevant sets. This change need some rounds of crowdsourced relevance assessments, which have not take place at the time of writing. For the present paper therefore, results are based on older relevance assessment.
This user scenario is a natural point of departure as it is in the tradition of information retrieval and facilitates the development of the task by using existing knowledge. As a future strategy, it may be argued that this user scenario is gradually modi ed, as ranking in the context of proving is a highly complex process, and, in the context where Prove-it algorithms are most likely to be used, arguably super uous.
What constitutes a proof is well de ned in elds like mathematics and computer science. In connection with a claim or a statement of fact, it is less obvious what demands a passage of text should satisfy in order to be considered proof of the claim. Obviously, we are looking for a passage which expresses a relevant truth about the claim, but what are the characteristics which signal a su cient degree of relevance and truthfulness? We might want to identify a trustworthy passage, which in turn might be identi ed by considering the source of the passage, the degree to which the passage agreed with other passages treating the same claim or fact, or the centrality of the claim to the main content of the text. We might want to identify a concentrated passage, a passage where the largest amount of elements contained in the claim were represented or where they were by some measure most heavily represented. We might look for a de nitional passage, which typographically or linguistically showed the characteristics of a de nition. Or we might try to identify a "proof" by linguistic characteristics, mostly semantic, which might be of di erent kinds: certain typical words might be relatively consistently used to speak about a fact or claim in a "proving" manner, writing in a "proving" mode might entail using terms on a certain level of speci city, etc. These latter aspects are orthogonal to the statement or claim itself in the sense that they (at least ideally) apply equally to whatever claim being the subject of proving / con rming. 1.3
A statement considered as a "proof" (or con rmation) may be characterized semantically by several indicators: { the phenomenon to be supported may be introduced or denoted by speci c terms, for instance verbs indicating a de nition: "is", "constitutes", "comprises" etc. { terms describing the phenomenon may belong to a speci c semantic category { nouns describing the phenomenon may be on a certain level of speci city { named entities of di erent kinds are heavily used { verbs describing the phenomenon may denote a certain type of action or state Deciding which speci city level or which semantic categories, will depend on the semantic content and the relationship between the terms of the original claim. Without recourse to the necessary semantic analysis, we assume that in general, terms indicating a proof / con rmation will be on a relatively high level of speci city. It will in some way constitute a treatment of one or more aspects of the claim at a certain level of detail, which we expect to be re ected in the terminology which is applied.
In 2010 we were investigating the extent to which the distribution of certain words, in ections of "to be" and "to have" in various tenses, could be an indicator of the page being a proof / con rmation of a statement.
In 2011, we were investigating whether terms, in our case nouns, found on a
page indicated as a potential source of proof diverge in a signi cant way from
other texts in terms of level of speci city. We determined the level of noun
speci city through their mean position along trajectories in the WordNet([
During the autumn of 2012 a new round of crowdsourced assessments was applied to the collection, using pooled results from all the participants, presenting AMT-workers1 with pages to assess.
In the present experiments, we proceed along the same lines, this time involving the use of named entity recognition.
Con rmation or proofs will often be about subjects identi able by a name. Gradually, we rst need to nd the limits of current detection of named entities, how easy it is to adapt it to a relatively diverse text mass that the (English part of) our text collection is, and then the approach's e ectiveness in detecting proving pages. The two main possibilities in taking NED into use are: { Detecting of named entities in general: pages that mention many named entities are candidates for being "con rming of something". Other methods are used to nd the speci c subject of proof. this means we only detect named entities in the book pages. { Detecting the named entity being the subject of the statement to be proved.
This means detecting named entities in the query, and matching them to named entities in the page.
Even though the latter possibility looks obvious, it entails some problems, like polymorphism in identi cation of entities, which must be approached. This is the main rationale for starting out with the former possibility. 1.4
In this paper we are still following the two-step strategy of rst nding pages relevant to the claim, and from those pages trying to identify pages that are likely to prove the claim2. The rst step is naturally done using current strategies for ranked retrieval. The second stage identi es among relevant documents those which prove / con rm the statement. Rank order is not necessarily preserved in this process: if document A comprises a better string-wise match with the claim than does document B, document B can still be more e cient at proving the claim than document A is. Not all elements that make a document relevant also make it a good prover
Another issue is the context in which proveIt! is used. One example is the writing of a paper. A writer is (again, arguably) more likely to evaluate a greater number of sources for proof of a claim than he or she would in a context of pure fact nding. Additionally, di erent contexts would arguably invite di erent proof 1 The crowdsourcing uses Amazon Mechanical Turk, and "workers" is the term used for people performing Human Intelligence Tasks (HIT) 2 We see refutal as a totally di erent type of task and will not address it in this paper. emphases. All this advocates for use of other strategies of presenting proving results than ranked lists, but we need to conform to the standard evaluation procedure which demands ranked lists.
The point of departure of the strategies discussed have been that con rming or
refuting a statement is a simple action of speech that does not require from the
book (the context of the retrieved page) to be about the topic covering the fact. In
that way, the "Prove It" task was di erent than e.g. the one referred to in [
This means that for each encountered page, the system needs to know (or be able to nd) the appropriateness of the book. The simplest strategy is that the index holds (for each page) some metadata about the book. A more elaborate strategy uses the existing metadata as basic information to use when resorting to external sources for nding appropriateness information.
Indexing In line with the above, indexing should facilitate the following main aspects at retrieval time: (a) identifying relevant pages, (b)excluding those that come from a book inappropriate for the proving the statement (c) identifying the extent of usage of named entities in a page, and (d) identifying the named entities themselves. The rst aspect, (a) is catered for creating a simple index of all the words in the corpus, page by page. The pages are treated as separate documents regardless of the book in which they appear, for (b), each page is also tagged by the subject headings of the book extracted from its bibliographic record. (c) is catered for by detecting and counting named entity occurrences in each page, and tagging each page by the extent of its use of named entities, and (d) is catered for by tagging and exposing each named entity in a eld tag.
The index also includes tags for con rming words and speci city ([
Named entity discovery Named entity discovery is a natural language processing (NLP) activity. There are several publicly available NED tools. As in 2012, we are still using the opennlp package of the Apache project. The package was used with default settings (no special training to adapt to our collection), with the assumption that the big diversity of the book collection is not apt to any signi cant improvement with respect to the default settings. 1.6
For our group it is still the case that we see the task of refuting as distinct from (and much more complicated than) the task of con rming, and we concentrate on con rming. This means that in addition to using the o cial qrels (where con rmation and refutal are merged) we will also use "pure con rming" qrels, from which refutal assessments are separated and ltered away. Identifying a baseline Even though our documents are all book-pages they may vary in length, and other language attributes meaning that identifying the smoothing that gives the best baseline for the documents as a whole requires some trial and error. As in all our experiments reported in this paper, the best baseline would be identi ed by running a basic edition of our query types against our index using a scan of (JM constants: 0.1 : 0.1 : 0.9). The optimal baseline parameters are then used in the semantic variation we do on baseline retrieval when testing its e ect.
Named entities: occurrence rate of named entities in the page As already indicated, rudimentary experimentation has been done with named entity recognition in this year's experimentation. The main parameter is the number of occurrence of named entities in the page. This is a very simple measure, insensitive to the length of the page itself. But as an initial gauge of the merit of name entities in detecting con rmation, we feel it is su cient, particularly when the property will be used in combination with other properties. We simply count the number of times a location or a personal name occurs in the page, and have nine di erent tags to stand for 1 - 8 occurrences, and nine or more occurrences, respectively. The various runs give extra weight to pages within di erent range of occurrence rates.
Referring to runs weighting named entity pages in isolation, it is di cult to pin-point a level of occurrence of named entity that stands out in performance. Named entity presence in a page does not seem to be a very good indicator of proving when used as the sole indicator.
A more topic-speci c approach than the former is searching the documents for occurrences of the same named entity identi ed in the query. Here it is not enough to match the named entity as it appears in the query, since a matching entity in a document may appear in a di erent form. This calls for expanding the query with as many as possible of the known forms of the entities found in it, with the hope that one of them will match the entity appearing in the page. This expansion, needless to say, is done at retrieval time. In the current experiments, we are using the MediaWiki API (ref) query and backlink facilities (ref). This is done in two steps: (a)querying Wikipedia for each of the named entities identi ed in the topic and identifying its preferred form and (b) applying backlinks to the preferred form to nd all the forms that redirect to (a). All those forms are then used (as synonym groups for each named entity) to expand the query, placing a higher weight on retrieving documents with named entities. This means that documents (pages) having matching named entities are assumed to have a higher con rming probability with respect to the statement in question.
ment New to 2013, a path of research we wish to pursue is the appropriateness of the containing book for con rming a claim, so that pages from inappropriate books are not counted. As our books have MARC-records associated with them, it is natural, to this end, to extract subject heading information from the MARC records. 21500 of the book-records contain the 650 (subject added entry) tag. About 28000 records have a 65X tag (including the former), and about 35000 (including both former groups include any 6XX tag. 082 (Dewey Decimal Classi cation) is rarely used in these records, and is therefore of a lesser value to this task). We see particularly the 650 tags as potentially helpful for identifying the genre or main topic of a book, and including it in an index, so that each page has the subject of its containing book in an appropriate tag. Using these tags is still a problem. Books that do not have this tag, need a less obvious treatment so that their appropriateness can be assessed.
There still remains the challenge to automatically deduce from the statement which subjects are appropriate to con rm it. 1.7
The results above indicate that there is more potential in treating occurrences of named entities occurring in the query rather than the general occurrence rate of named entities in documents. Still, more experiments are necessary.
The experimental research into the prove It! task requires, in our opinion, gradual advance in several fronts. Diverting from traditional IR, two main activities need to be re ned in a semantic direction: the semantic aware search algorithms and the intellectual "relevance assessment", here understood as rening the preparation of the crowdsourcing activities. The goal must be to arrive at models that incorporate the di erent aspects of proving/con rming/substantiating discussed above that may predict the proving power of a document with respect to a statement.
At this stage we are assessing the potential of each of the aspects of named entity discovery, to explore how each of them (and a combination of them) can contribute to a linear model that predicts the con rming power of a document.
A baseline as ours (created with smoothed language modeling algorithms taken from traditional IR) is di cult to beat, but it is nevertheless interesting to measure how closely we approach the results by above mentioned combinations. 2 2.1
The social book search features two representations of books: the social data, which is a mixture of "Amazon data" (descriptive and social data to facilitate book sale via Amazon) and social encounters as recorded in the libraryThing fora on one hand, and, on the other hand, traditional library data (MARC records) entered by professional catalogers. The main purpose is to nd out the relative utility of each of these representations when it comes to automatic book recommendation.
[
As the case is for the prove It! research discussed above, also here we need to
advance gradually. Traditional IR is content-aware. Social search does not always
originate in a knowledge (read:content) anomaly ([
The collection has been loaded into a database where all types of data about each book are associated with the book's ISBN. We create an Indri index that contains two representations of each book as two documents, one containing the browseNodes and the other containing the rest of the tags under the <book> hierarchy - all the tagged XML information that is extracted from both amazon and the LT fora. The result list will be a mixture of documents of both representations, and they are collated so each original document takes the rank of the best of its representations in the list. <DOCNO>0836921283 N</DOCNO> <TEXT> <browseNodes><browseNode i d ="48">Home & ; Garden</ browseNode><browseNode i d ="1000"> S u b j e c t s </browseNode ><browseNode i d ="5241"> Gardening & ; H o r t i c u l t u r e </ browseNode><browseNode i d ="5276"> General </browseNode>< browseNode i d ="283155">Books</browseNode><browseNode i d ="713339011"> General AAS</browseNode></browseNodes> </TEXT> </DOC>
At retrieval time, doubling the weight of browseNodes can be done <query>
<number>530</number> <t e x t > #combine ( #weight ( 1 . 0 #combine ( J e s u s . book Why. book s c h o l a r l y . book p e r s p e c t i v e . book h i s t o r i c a l . book From . book ) 2 . 0 #combine ( J e s u s . browsenodes
Why. browsenodes s c h o l a r l y . browsenodes p e r s p e c t i v e . browsenodes h i s t o r i c a l . browsenodes From . browsenodes ) ) ) </t e x t > </query>
After retrieval, the ranked lists are collated as the original book document takes the place of its higher ranked representative. In this way we can more easily isolate the contribution of the di erent parts of the document to the retrieval performance.
Preliminary runs were performed in accordance with the description in Section 2.2. Figure 3 summarizes the results. As for now, the queries are quite "naive" as related to the Browsenodes, and no expansion is taking place. a natural path to consider is expanding the browsenodes with synonyms before indexing them, so that the queries are not dependent on having the exact wording of the browsenode for matching a document containing it. Some work needs also to be done to nd the exact average net relative weight of the document representations in order to 3
The purpose of the linked data track is to nd out how techniques from semantic web / linked data can be used to improve and enhance retrieval of Wikipedia articles. The data collection is an XML'i ed version of a Wikipedia subset (about 4.1M articles), where RDF-properties from DBPedia and YAGO, are combined with the article text.
The idea behind our experiment is based on a two-stage approach. The initial search is in an index built from the entire corpus. Here each article is only represented by a small selection of the le's contents. For each topic, categories found in the articles from the initial search result (1000 articles) are extracted and matched against the topic. If the similarity is above a given threshold the articles related to the category are inserted at the top of the nal result list.
Wikipedia categories have been used to enhance ad hoc retrieval [
Stage one In an e ort to reduce noise, les that are not considered articles, such as metadata on images and deleted articles are ignored. The original Wikipedia articles are reduced to skeleton documents consisting of a selection of the semantic markup. In our 2012 run only title, headings and categories where indexing. In 2013 the extraction process has been expanded to include the following six elds: { the title (TITLE) { the infobox type (INFO-BOX) { the title of other entities that are linked to (out links) from the article text (ENTITY) { the Dublin Core subjects from the DBPedia properties (DC-SUBJECT) { the RDF types from the DBPedia properties (RDF-TYPE) { the RDFS labels from the Yago properties (ALT)
As an example, the Wikipedia article on Edvard Munch, ready for indexing by Indri (please note that the example is abridged due to space limitations): <DOC> <DOCNO>1x1axf2x9779</DOCNO> <TEXT><TITLE>Edvard Munch</TITLE> <ENTITY>mona lisa</ENTITY> <ENTITY>degenerate art</ENTITY> <ENTITY>auguste rodin</ENTITY> <DC-SUBJECT>expressionist painters</DC-SUBJECT> <RDF-TYPE>symbolist painters</RDF-TYPE> <DC-SUBJECT>norwegian artists</DC-SUBJECT> <ALT>edvvard munch</ALT> <ALT>edward munch</ALT> <ALT>edvard</ALT></TEXT> </DOC>
The ALT eld combines alternative forms of the article title (re-directs), e.g.
"edvvard munch" and "edward munch", and disambiguations, e.g. "edvard".
DBPedia and Yago properties are also generated from the same structured parts
of the Wikipedia article [
An unweighted search for a query generated from the unique words combined from the topic title and description using two-stage smoothing with a lambda value of 0.9 creates the initial ranked list which forms the baseline for stage two. Stage two Each document from the result list is fetched and the subjects (Wikipedia categories) are extracted. The category name is matched against the topic title. Prior to the matching the topic title is expanded with synonyms from Wordnet and a small list of stopwords (in, on, of, the, and, by, to) are removed. The Levershtein edit distance measure is used to nd the lexical similarity between topic and category. The best category word is found for each synonym group and the total is normalised by the category length. Only categories with a similarity score above a 0.75 threshold result in the insertion of associated articles. Articles from the original ranked list, that are not assigned to the selected category are pushed down the list.
Only six of the 144 topics resulted in categories that are above the chosen threshold (compared to six of the 30 evaluated topics in 2012) which suggests that the 2013 topics don't have the same broad nature of the previous year.
We are awaiting results for our 2013 run. 3.3
Future work is required to examine why only such a small number of the topics found an appropriate category. The quality of category assignment in Wikipedia is a further issue that could be investigated and the possibility of assigning categories automatically. Exploiting the hierarchical nature of the Wikipedia category structure and interest.
nding links to additional subject descriptions are of 4
At the same time that the book world becomes more and more digital,as old books are being digitized and new books are increasingly published digitally, information not published in book format becomes more and more "semantic" in the sense that data pieces (as opposed to exclusively documents in the web's rst years) are linked together and made available. These two parallel development entail great opportunities in the exploitation of book material for di erent purposes, of which the topic of this paper is one example.
This paper provides an example of the possibilities and the challenges. Whereas
"WordNet speci city"([
To judge by the results of the runs presented here, this path of research, though promising, still requires a lot of modi cation and calibration.
Exploring the semantics of a page in a basically statistical manner may be
seen as a superposition of independent components or treatments. Particularly
in the ProveIt! task, we wish to pursue the treatments from [
Utilizing digital books poses new challenges on information retrieval. The mere size of the book text poses both storage, performance and content related challenges as compared to texts of more moderate size. But the challenges are even greater if books are to be exploited not only for nding facts, but also to support exploitation of knowledge, identifying and analyzing ideas, a.s.o.
This article represents work in progress. We explore techniques gradually in an increasing degree of complexity, trying to adapt and calibrate them.
Even though such activities may be developed and re ned using techniques
from e.g. Question Answering[
One obstacle particular to this research is the test collection. Modern
ontologies code facts that are closely connected to the modern world. For example
the Yago2 [
This article is a further step in a discussion about semantics-aware retrieval in the context of the INEX book track. Proving (or con rmation or support) of factual statements is discussed in light of some rudimental retrieval experiments incorporating semantics. We also discuss the task of proving statement, raising the question whether it is classi able as a semantics-aware retrieval task. Results are highly inconclusive.