We present a framework based on Statistical Topics Models, Language Models, Information Extraction, and Ontology Analysis to retrieve healthcare related documents for the CLEF eHealth 2013 Task 3. In this framework we add global information based on latent topics from the documents to improve the document retrieval. We perform six different experiments which consist of a baseline and six variants of the model. Preliminary results show that the use of Language Models with a bag of words scheme results better estimates. However model tunning in the Topic Based model is required to achieve optimal results.
In these notes we describe the experimental framework presented in the CLEF eHealth 2013 Task 3 which consist of retrieving relevant health care documents. Our main approach consist of providing a solution that does not require any manual input besides the query.
To achieve this goal, we captured global context from the documents in a unsupervised form using Statistical Topic Models. Our main hypothesis is that semantic content may not be correctly represented with single terms alone. In addition, we present a variant of the model where we extract noun phrases and incorporate them in the query. We claim that using the words independently may change the meaning of the query specially when the user is searching documents from a particular disease that is composed by two or more terms. For instance, cardiac disease might not be correctly detected as a noun phrase and the independent term disease would largely increase the documents retrieved with other types of diseases.
This paper is divided as follows: Section 2 shows the methodology that we follow and how we obtain the different components of the propose model. In section 3 we describe the steps we follow to pre-process the corpus, and we describe the different experiments that we performed. Finally the results and conclusion are discussed. such as the extraction of topics using Statistical Topic Models and the noun phrases extraction. In addition, we detail the incorporation and processing of the discharge summaries in the model. 2.1
Statistical Topic Models is an unsupervised learning technique that allows us
to extract latent topics from a corpus of documents. The fundamental idea is
that these topics provide a global context, which can not be achieved using only
independent words as in the standard bag-of-words modeling. For the current
problem, we use Latent Dirichlet Allocation (LDA) [
Where and w is the word, z is the topic label and α and β are the parameters of the prior distribution for the topic mixture and vocabulary mixture respectively. To fit the LDA model with the corpus of healthcare documents we need to select the number of topics. The two parameter vectors that are fitted in the model are the prior distribution vectors for the probability of topics in a document, and the global probability of words given a topic. With these parameters we obtain the topic mixture for each document and save it as metadata that is used in the document retrieval. 2.2
Noun phrases provide relevant information about diseases and treatments
usually. This is because several diseases are often identified by two or more terms. In
order to select accurately the noun phrases, we use the CTAKES [
We select the extracted nouns with two or more words since these nouns are not detected by the bag-of-words model. Note that some of the resulting noun phrases are combinations of two of more shorter phrases. We select only those with the smallest number of terms. Then, we analyze if a long noun phrase can be decomposed as a combination of the shortest noun phrases. If this is the case then we ignore this phrase. If this is not the case, we keep the full noun phrase. We remove the phrases which do not include any medical content. To determine if the medical content of the phrases, we compare the extracted nouns with the SNOMED ontologies in order to keep the ones inside the ontology. We obtain 3, 075 noun phrases using 10, 000 discharge summaries from patients with different diseases. 2.3
We include the discharge summaries information in three of the six experiments we perform. However, when we reviewed the content of the discharge summaries, some of them have little of not relevance with the query. To overcome this problem. We only take into account those paragraphs with significant relation to the query.
We determine this relationship in a unsupervised form by comparing the query terms with the content of the paragraph. We found that only a small percentage of the discharge summaries contains information related to the query, which ranges from 1 to 3 paragraphs, compared to the average discharge summary length of 10 to 20 paragraphs. In some cases only a sentence in the entire summary has a relationship with the query. The extracted information from these summaries is combined with the orignal query in order to create a single expanded query. 2.4
We use Language Models to retrieve the documents. In this framework, we estimate the probability that the query is generated by the document. This probability is represented by the following formula: (2) (3) (4) m P (Q|θD) = Y p(qi|D)
i P (qi|D) = c(qi, D) |D| where P (qi|D) is the probability that the query term is generated by the document, and this is defined as: here c(qi, D) is the frequency of the query word in the document and |D| is the total number of terms in the document. In this method, we might find a zero probability for P (qi|D) when we have a term in the query that is not inside the document. To overcome this problem, we incorporate a smooth factor in the probability P (qi|D) that is defined as follows: c(qi|D)
|D|
P (qi|D) = λ
+ (1 − λ)P (qi|C)
where P (qi|C) is the probability that the query word is in the collection. The
value of λ ranges from [
We include the topic information by using the framework introduced in [
K Plda(qi|D, θD, φ) = X p(w|z, φ)p(z|θ, D) z=1 (5) where K is the number of topics, φ is the posterior probability estimate of the word mixture for each topic, and θ is the topic mixture for the document. Then the language model is defined as follows: P (qi|D) = λ(
Nd c(qi|D) μ + Nd |D| + (1 −
Nd μ + Nd )P (qi|C)) + (1 − λ)(Plda(qi|D, θD, φ)) (6) The value of μ is defined as 1 and Nd is the number of documents. The value of λ is defined to be 0.6. The noun phrases are included in the model as any other word. Consequently, the frequency of those phrases in a document is used to represent the documents. 3 3.1
We use the corpus of medical-related documents provided by the Khresmoi
project. Details of the dataset are found in [
After the data has been cleaned, we indexed all the documents using Indri
index engine from Lemur Project [
We have 1628500 documents and a vocabulary size of 98031 after removing documents with less than 20 terms, removing stop words and stemming when we use the bag of words approach. When we use the noun phrases, we replace the noun phrase found in the document with a term code used as a word, and then we index the documents. With this variant we have a vocabulary size of 99538 and 1628500 documents
Once we have indexed all the data, we fit the model using 10% of the documents in the corpus and 75 topics. We select those documents randomly. This allow us to model the corpus more accurate. Once we estimate the prior distribution parameters, we extract the topic content for the remaining documents by sampling the labels for each word in the document using Gibbs Sampling and the prior distribution of the word mixture. Once we have estimated the label for each word, we calculate the topic mixture for each document and save it a metadata in the document. Therefore we have two sets of topic mixtures: one only taking into account the single terms and other that takes into account the noun phrases.
We analyze the queries by parsing them using the constructed index and we remove the stop words. If we use noun phrases, we first replace them with a term code and then we parsed with the index from the document and remove the stop words. 3.2
We submit six different runnings to the task: one baseline and five variants of the model. In this section we describe each of the variants of the model.
We use as baseline the language models described in equation 4 with only the single terms, and without any noun phrases. This method is considered to be the baseline for the other variants of the model.
We have 5 variants of the model: 1. Including the information from the Discharge summaries and noun phrases using Probabilistic Topic Modeling and Language Models defined in equation 6. 2. We use noun phrases with the queries that contain the information of the
Discharge summaries and using only Language Models 3. Language Models and the information extracted from Discharge Summaries without any noun phrases 4. Language models with noun phrases and topic information using only the queries 5. Language models with noun phrases without any information of the discharge summaries
The main idea is to test how the different components contribute to the global result. All the variants are fast in the retrieval process. The main constrain is the offline process that they require. 4
In this section we will discuss the results of all the variants of the model that we run. Table 1 shows the results of the Precision at 10 (P@10), Mean Average Precision (MAP) and the Normalised Discounted Cumulative Gain (NDCG) for the baseline and the five variants of the model
As we can observe the baseline results in better results than the variants. This can be explained by the nature of the query. In this case the use of noun phrases (Variant 5) may help in some queries but in others affect the result. We observe that the use of Topics does not improve the performance of the method, which may be cause because the topics may add noise to the query instead of clarifying the query. This phenomena could be caused by the number of topics, which are not sufficient to describe correctly the corpus. Other explanation is that some results retrieved by the method may not be labeled. This may result in a poor performance of the methods.
Figure 1 shows the precision results for all the queries using the best method (baseline). Here we can observe that the precision of some queries is negative. This means that the baseline have lower performance that the median of the other groups. However some other queries has a comparable performance of the best groups. 5
In this paper we presented a method to retrieve relevant health care related document. We believe that our approach have promising results. Further research and model tunning may be required in order to make our approach competitive and suitable to be applied in the healthcare context.