This is the working note of our participation in the 2020 Causality-driven Adhoc Information Retrieval Task. We aim to retrieve news documents that indicate possible causes of a query event. Our proposed method involves query expansion based on causality relation detection from a primarily retrieved collection. We compare our method with two baselines and analyze the performance of expansion terms on selected topics. Our analysis shows that although the naive causality detection method recognizes some critical terms, precision should be improved to outperform the baselines.
Automatic connecting events based on causality relation provides a promising application for
The core idea of our method is query expansion based on causality detection. We define a postevent as the event we want to know about its cause (e.g., Telecom minister A. Raja resignation ), and a pre-event as one possible event that leads to the post-event. Thus, we treat the task as detecting possible pre-events given the post-event, which is the query, and retrieving documents Initial Query Telecom minister A.
Raja resignation
Event Term resignation
Document
Set_1
Pre-event Post-event accusation scam resignation …… Query Terms accusation scam …… telecom minister A. Raja Document
Set_2
that describe the pre-events. To achieve this, we assume that potential pre-events can be found in the documents that describe the post-event. Once retrieve these kinds of documents, we can conduct causal relation detection to find potential pre-events and use them to do a second retrieval to get the final desired documents. We organize our approach’s procedure to 5 steps, as shown in figure 2.1, and describe them in detail in the following paragraphs.
Event extraction: The first step is event extraction, in which we extract the syntactic head word of the sentence (usually the verb) in the query title to represent the post-event. The intention is to get more information from similar events.
First retrieval: We use the resulting event term above to retrieve a document collection for extracting potential pre-events.
Causal relation detection: We implement a simplified version of [
Query expansion: We use the extracted pre-event terms above to expand the original query and remove the original event term. We presumably choose the top 5 sorted according to its frequency mentioned above. Further investigation of the number of terms to use can be explored in the future.
Second retrieval: The document set retrieved using the expanded query is our final result.
Following the work of [
Documents retrieved should contain information about the accusations against A. Raja in the 2G Spectrum scam. Documents that talk about his criticisms for allocating 4.4MHz of spectrum a scarce resource at the 2001 price of Rs 1,651 crore through a first come, first served policy rather than auctions; by compromising on the market value of spectrum, the government is estimated to have lost revenues worth around Rs 50,000 crore; new licensees are signing equity deals with foreign players in which the value of spectrum has risen sevenfold, would be considered as relevant. as the queries. For similarity function, we use LMJelineMercer similarity with the smoothing parameter set to 0.7. 5 top pre-event terms expanded by our method are used and removed if the expanded term is identical to the original event term. We retrieve up to 1, 000 documents for each topic as our final results.
The CAIR dataset is a collection of 303,291 Telegraph India news documents from 2001 to 2010
[
We compare our method with two baselines that directly use the query title and query narratives, respectively, with the same setting. We use title to represent the baseline using query title and narratives to represent the one using query narratives. We use title-expansion to denote our method.
Table 2 shows the experimental results. We can find that our method has not been competitive with the two baselines. narratives outperforms the other methods significantly, which is not surprising as the content of query narratives have already contained the pre-events desired. In Contrast, the results show that the expanded terms in title-expansion do not help improve the original query title in the retrieval task. We then provide further analysis in section 4.
We choose 3 topics from the testing set to conduct the analysis. The top terms as potential pre-events extracted by our model are listed in table 3. For the event term ‘assassination,’ our model efectively extracted terms like ‘attack’ and ‘deathkilled,’ which is of quite a small portion. Most extracted terms are rather irrelevant. For the event term ‘accused,’ more terms for possible pre-events are detected (‘court’, ‘police’, ‘case’, ‘arrest’, ‘murder’, etc.) We notice that the accuracy of our extraction method is afected by the topic property. For the event ‘assassination,’ proper nouns are identified (‘Gandhis,’ ’Musharraf’) as well as other terms like ‘President,’ ‘Minister,’ and ‘government’ that might be too specific for some particular events that involve assassination. The extracted terms for the post-event ’resignation’ follow a similar pattern. In contrast, the extracted terms for the ‘accused’ event are rather general and can be applied to diferent events that involve accusations. We infer that the diference comes from the frequency of some particular events being reported on the news. For example, If there is a lot of news that reports the assassination of a particular person, the terms specifically related to this single event are more likely to be extracted by our model. Overall, the analysis shows that the extracted terms of our current method contain some useful information yet not accurate enough and highly depends on the topic.
In this work, we implement a retrieval model based on query expansion and naive causality detection to approach the CAIR task. We also compare our model with baselines that only use query title and query narratives. Experimental results show that our method has not been able to surpass the baselines. Further analysis of selected topics indicates that the causality detection method needs to be improved to enhance the extracted terms’ precision. While our naive approach has not distinguished the causal sentences that are actually relevant to the topic from the non-relevant ones, possible approaches like embedding method to jointly model the topic and the causal sentences may be considered in future works to improve the performance.
This work is supported by the IRKM lab in University of California, Santa Cruz.