In this paper we present our participation as SINAI research group from the Universidad de Jaen at Task 3 \Consumer Health Search" speci cally in sub-task 1 \Ad-hoc Search". The main objective of the task is to provide relevant information to people seeking health advice on the web. We apply the query expansion technique using the most famous search engine at the moment: Google. We search additional information related to the query using the search engine. We identify the medical concepts in Google results using cTAKES. This recognizer provides UMLS concepts from a given text. In this way, we avoid introducing noise with words that are not related to the user query. Our system improves NDCG@10 measurement by 48% over the previous year. We also signi cantly reduces the response time of the Information Retrieval System (IRS) by 90% compared to previous years.
CLEF 2018 [15] consists of an independent peer-reviewed conference on a broad range of issues in the elds of multilingual and multimodal information access evaluation, and a set of labs and workshops designed to test di erent aspects of mono and cross-language Information retrieval system.
This year we have participated in the Information Retrieval (IR) task and
we will continue exploring the same problems and issues identi ed in 2014 [
This task deals the problem of retrieving relevant information in the biomedicine domain. People are constantly looking for information by querying the most important search engines. With these searches, the user tries to obtain more information about diseases, illnesses, treatments, etc.
Our research group SINAI has a large experience participating in several
tasks of other editions of CLEF eHealth. In previous years, we have participated
in this competition making use the Google search engine[
For our approach, we use the clinical Text Analysis and Knowledge Extraction System (cTAKES)1. cTAKES is an open-source Natural Language Processing (NLP) system for information extraction from electronic medical record clinical free-text [13] including types of clinical named entities mapped to various biomedical terminologies/ontologies such as the Uni ed Medical Language System (UMLS).
This paper is organized as follows: In the next section, we introduce the resources provided by the organizers. Our approach is described in Section 3. In Section 4 we include the results obtained and nally, we expose the conclusions and future work. 2
2.1
For task 3 of CLEF 2018 the corpus of documents consists of web passages acquired from CommonCrawl2. For the creation, an initial list of websites was selected, this list was constructed by sending the queries to Microsoft Bing APIs and discarding the unreliable websites.
The collection consists of 1,903 domains including some of the most famous such as facebook.com, answer.com and health.com. Each web page is in the original format as crawled from CommonCrawl, thus it may be html, xhtml, xml, etc. It is important to have a global vision for this type of task, as each website can provide relevant information from every point of view. 2.2
The organization also provided several indexes created due to the large size of the collection because there are groups that do not have the possibility of dealing with a corpus of this size. This makes it easier to compare the systems between the participants.
After uncompressed, each document of the collection was extended with the traditional TREC format following the Figure 1.
The collection was indexed as shown in the Figure 2.
2 http://commoncrawl.org/2018/03/february-2018-crawl-archive-now-available/ (last
visited: May 31, 2018)
<doc> <docno> DOC ID </docno> ORIGINAL CONTENT <doc>
The query set for 2018 consists of 50 queries issued by the general public to
the HON3 (Health On the Net) and TRIP4 search services. The queries and the
process to obtain them are described in [
We found out that entering everything provided by Google, it is included a lot of noise to the nal query and for this reason, we decided to use a biomedical entity recognizer.
The Figure 4 detail the procedure followed. The IRS (explained in section 2.2) uses an information retrieval model described in [14]. The new document collection contains web pages from di erent websites. We consider a good idea to simulated a typical web search and so, we have tried to integrate the knowledge from the most popular web search engine: Google.
We have performed a search for each query using the Google API for the extraction of titles and snippets. The query was cleared of punctuation marks. A total of 10 results have been taken into account and added to the query the titles and snippets of those results.
The average word count of the expanded query with 10 titles and 10 snippets is 330 words, out of which 29% are stop-words. 3.2
cTAKES: recogniser of medical entities To select the terms used to expand the query, we use cTAKES, a tool for the recognition of medical entities. This system identi es UMLS concepts in the query expanded by Google. We create a new query only with terms detected with cTAKES.
To build the nal query, the concepts returned by cTAKES will be assigned a weight according to the number of occurrences obtained in Google, and nally, we add those words existing in the original query and that have not been detected with cTAKES with a xed weight W , thus informing the system that these words are also relevant to the query. In this case, the stop-words of original query are not included.
To determine the value of W , several experiments have been performed using the 2017 relevant judgments and queries. In these experiments we observe that the most optimal value is equal to the number of documents selected in Google. In our case W = 10.
Figure 5 shows the result of expanding the query following the process in Figure 3. We have used the Indri query language5, where #combine allows us to use multiple words as a unique term and the #weight operator assign varying weights to each expressions. <query> <type>indri</type> <number>195001</number> <text>#weight( 1 #combine(bacteria) 15 #combine(treatment) 4 #combine(treatments) 1 #combine(microscopy) 1 #combine(direct microscopy) 1 #combine(test) 1 #combine(tips) 1 #combine(prevention) 3 #combine(therapy) 3 #combine(oxygen therapy) 3 #combine(hyperbaric oxygen therapy) 1 #combine(sick) 1 #combine(diagnosis) 1 #combine(illness) 20 #combine(disease) 20 #combine(lyme disease) 1 #combine(infection) 1 #combine(fibromyalgia) 1 #combine(conditions) 5 #combine(antibiotics) 6 #combine(antibiotic) 2 #combine(rise) 3 #combine(oxygen) 10 #combine(affective) 10 #combine(chronic) 10 #combine(lyme) ) </text> </query> 5 https://www.lemurproject.org/lemur/IndriQueryLanguage.php
This year, we have a new corpus for this task, therefore, we do not have the results obtained for our system. But in the Table 4 we can see the results of the experiments carried out in this year using the 2017 collection and the 2017 relevant judgments.
Run Baseline Google Google + cTAKES Compared to the experiment of the last year (Google) where we didn't use cTAKES to lter, this new experiment (Google + cTAKES ) improves NDCG@10 value by almost 48%, whereas the BPref and RBP values only decreases almost 11% and 22% respectively. We are analyzing the obtain results to understand why these values are lower.
In addition, the time required by the IRS to process the query is signi cantly reduced, we improve in 90% of the total time invested. A query with Google expansion without cTAKES lter contains 390 words on average, of which 23% are stop-words. However, a ltered query with cTAKES contains about 30 weighted terms. 5
In this paper, we have presented a method to expand queries with the most popular search engine at the moment, adding more information to the user's query. We have observed that this method introduces a lot of noise into the nal query, so we have tried to lter and keep the words most related to the biomedical domain.
For this reason, we use the medical entity recognizer cTAKES to lter Google results and we obtain a nal query focused on the biomedical domain, and we also use the weighted words, giving more importance to some identi ed terms.
In the future, we will continue this work to improve our systems by adding
knowledge to them. We will study di erent medical ontologies for the expansion
of queries [
This work has been partially supported by a grant from Fondo Europeo de Desarrollo Regional (FEDER) and REDES project (TIN2015-65136-C2-1-R) from the Spanish Government. 13. Savova, G.K., Masanz, J.J., Ogren, P.V., Zheng, J., Sohn, S., Kipper-Schuler, K.C., Chute, C.G.: Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications. Journal of the American Medical Informatics Association 17(5), 507{513 (2010) 14. Strohman, T., Metzler, D., Turtle, H., Croft, W.B.: Indri: A language model-based search engine for complex queries. In: Proceedings of the International Conference on Intelligent Analysis. vol. 2, pp. 2{6. Citeseer (2005) 15. Suominen, H., Kelly, L., Goeuriot, L., Kanoulas, E., Azzopardi, L., Spijker, R., Li, D., Neveol, A., Ramadier, L., Robert, A., Palotti, J., Jimmy, Zuccon, G.: Overview of the clef ehealth evaluation lab 2018. In: CLEF 2018 - 8th Conference and Labs of the Evaluation Forum, Lecture Notes in Computer Science (LNCS). Springer (2018)