Nowadays, consumers often search online to seek medical and health care information that they need. To improve this access, the ShARe/CLEF eHealth Evaluation Lab (SHEL) organized a shared task on information retrieval for Medical Disorders in 2013. This paper describes our participation in this task. In order to detect latent semantic relevance between queries and webpages about disorders, a semantic vector model based on distributional semantics is used as the information retrieval model. Specifically, variants of random indexing are employed to generate document and term representations. In addition, to reduce the lexical lap between different clinical expressions of the same concept, query expansion is also conducted using the UMLS. A baseline information retrieval method using the vector space model (VSM) and semantic vector models with different random indexing building procedures were developed and evaluated with or without query expansion in the shared task. The best performance was achieved by VSM, with MAP of 0.1480, P@10 of 0.3700 and nDCG@10 of 0.3363. Experimental results indicate that VSM and semantic vector model are complementary, and suggest combining these methods may further improve performance.
Nowadays consumers increasingly access electronically available medical and health
care information. The rapid development and wide use of the Web has significantly
altered the way people find medical information. Nearly 6% of Internet users on an
average day search for medical information on the Web [
Previous work has been attempting to solve this problem in multiple ways. Health
information search behavior, information needs and contexts were analyzed based on
query logs and social communities [
The latter sections are arranged as follows: Section 2 describes the information retrieval methods for medical disorders in detail. Section 3 presents the experiments and results. Section 4 discusses the experimental results and Section 5 is the conclusion. 2 2.1
This paper employs two different information retrieval models: one is the vector space model, as one of the state-of-the-art benchmarks in information retrieval [21]. Another is the semantic vector model [16], which has been attracting research attention for effectively revealing latent semantic relations between terms and documents (unlike the VSM, which will only retrieve documents containing at least one of the query terms). 2.2.1
Represent the document and the query as vectors, != ,, … , , and = ,, … , , in an n-dimensional vector space [22]. Each dimension corresponds to a unigram, with tf.idf as the value. The cosine similarity between and is used for relevance ranking of documents. It is defined as: !, = !∙ ∙ (1)
Methods of distributional semantics [
One key issue of semantic vector model is to reduce dimensions to improve processing performance, and in some cases measures of semantic relatedness. Random Indexing [18-19] offers an efficient and effective method for reducing the dimensionality of the semantic space. The process of random indexing is as follows: 1) Generate an m-dimensional index vector for each term. The term index vectors are generated using random projection [24], which projects the n-dimensional (m << n) term vector in to a lower dimensional subspace. Each index vector is a sparse vector with a small number of +1 and -1 values like 0, 0, 0, 1, 0 … − 1, 0, −1, 0, 0 . Two arbitrary vectors are nearly orthogonal to each other, so that the distance between original vectors can be preserved [25]. Terms that co-occur directly will have similar vector representations 2) Sum index vectors of terms contained in a document to generate the document index vector.
Reflection: 3) Sum index vectors of documents containing a term to re-generate the term index vector. 4) Iterate between 2) and 3) to reflectively generate index vectors, so that to reveal higher-order relations between terms. 2.3
Previous work of query expansion [26] using the Unified Medical Language System (UMLS) [20] has achieved performance enhancement for information retrieval. We used content words extracted from queries as input and use UMLS API [27] to obtain possible Concept Unique Identifiers (CUI). Terms belonging to the top ranked CUI of the content word were used as expanded queries. 3 3.1
The open source toolkit Lucene2 is used for index building and information retrieval by VSM. Another open source toolkit, Semantic Vectors3, is used for semantic vector index building and information retrieval [28-29]. Based on the manual annotations provided by the task organizer, performance on the training set is evaluated by MAP and P@10. Performance on the test set is evaluated by MAP, P@10 and nDCG@10.
MAP, as in equation 2, is abbreviated for mean average precision, which is the mean of the average precision scores for each query q in a query set Q. P@10 is the number of relevant results on the top ten search results. DCG, as in equation 3, uses a graded relevance scale reli to evaluate the usefulness of a document based on its position in the result list. DCG assumes highly relevant documents appearing lower in a search result list should be penalized. NDCG (Normalized Cumulative Gain) is the normalization of DCG value of the ideal ranking at rank n.
MAP = !!!! !"#$%#&'(')*(!) (2)
!
DCG! = ! + !!!! !"!"##!!! (3) The following methods are compared in our experiment: • VSM (UTHealth_CCB.1.3.noadd): Results are retrieved from Lucene using
VSM, with content words from title and description as queries. • SemVec (UTHealth_CCB.5.3.noadd): Results are retrieved from 4000dimensional semantic vector based index, with content words from title, description as queries. The index is built without reflection. • VSM&UMLS (UTHealth_CCB.6.3.noadd): Results are retrieved from Lucene using VSM, with content words from title, description, narrative, and expanded terms from the UMLS as queries. • SemVec&UMLS (UTHealth_CCB.7.3.noadd): Results are retrieved from 2000-dimensional semantic vector based index, with content words from title, description, narrative, and terms from the UMLS as queries. The index is built with one turn of reflection. 3.2
Table 1 shows the performance of the employed methods on the training queries for the ShARe/CLEF eHealth 2013 shared task 3. VSM and SemVec obtained comparable results. Query expansion using UMLS achieved enhancements both for VSM and SemVec, especially in P@10, with an improvement of 100% for VSM, and an improvement of 75% for SemVec. The MAP also increased about 50% for VSM. Table 2 displays the performance of our methods on the test queries. The best performance was achieved by the baseline method, i.e., VSM, with MAP of 0.1480, P@10 of 0.3700 and nDCG@10 of 0.3363. Nevertheless, performance of the other three methods dropped severely on all the three evaluation criteria compared with VSM. As demonstrated in Table 1 and 2, performance on the training and test queries differed in several aspects. All the results enhanced significantly on test queries compared to the training set, except for SemVec with query expansion using the UMLS. Besides, the performance of SemVec was lower than that of VSM on the test set, instead of a comparable performance between the two on the training set. Furthermore, in contrast to a performance increase using query expansion on the training set, the test set illustrated a performance decrease. One possible reason of these differences could be the different pooling sets used for evaluation between the two sets. The pooling set for training was built from results from two information retrieval models, VSM and Okapi BM25 [30]. In contrast, the pooling set for test was built from the results submitted from all participants. Another reason could be the different information needs contained in the two sets. In addition to questions asking for disorder definition and treatment in the training set, relational questions asking for connections between other entities and disorders account for a large proportion in the test set, such as “<desc> can chest pain hinder the transplantation of liver? </desc>” and ““<desc> what is the connection between acidosis and metastasic adeno carcinoma</desc>”. The employed methods in our experiment may not be suitable for retrieving relevant information for such kind of information needs.
As illustrated by the P@10 plots provided by the task organizer, both VSM and SemVec contributed best P@10 for several different queries. Looking into the results of VSM and SemVec, it was found that they retrieved different relevant webpages from the index. Take test query 12 “<desc>is clots in jugular in connection with HIV</desc>” as an example, VSM mainly retrieved webpages informative of HIV. Nevertheless, SemVec obtained more navigational webpages containing links to diverse aspects of HIV, with each link leading to an informative webpage focusing a specific aspect of HIV. Figure 2 shows a navigational webpage of HIV retrieved by SemVec.
Furthermore, relevant webpages with different topics of the query were retrieved. As an example, for test query 10 “<desc>is there a connection between multiple sclerosis and dysplasia in oesophagus</desc>”, VSM obtained three relevant webpages about “dysplasia in oesophagus” and one about “multiple sclerosis”, while SemVec found eight relevant webpages about “multiple sclerosis”. Those observations demonstrate that these two methods are complementary to each other and may be combined to produce more diverse and relevant results. 5
This paper describes our participation in the task 3 in ShARe/CLEF eHealth 2013 challenge. Different information retrieval models, namely vector space model and semantic vector models based on the distributional semantics theory were employed. The experimental results demonstrate that both models are complementary to each other. The next step in our future work would be exploiting the combination of semantic vector models with other information retrieval models for further performance improvement. A. Peñas, V. Petras, and D. Santos, pp. 601–606. Springer Berlin Heidelberg (2008) 13. W. R. Hersh, Information retrieval: a health and biomedical perspective.
Springer (2009) 14. J. Lin and W. J. Wilbur: PubMed related articles: a probabilistic topic-based model for content similarity. BMC bioinformatics, vol. 8, no. 1, pp. 423. (2007) 15. Hanna Suominen, Sanna Salanter"a, Sumithra Velupillai, Wendy W. Chapman, Guergana Savova, Noemie Elhadad, Danielle Mowery, Lorraine Goeuriot, Liadh Kelly, David Martinez, and Guido Zuccon: Overview of the ShARe/CLEF eHealth Evaluation Lab 2013. (2013) 16. T. Cohen and D. Widdows: Empirical distributional semantics: Methods and biomedical applications. Journal of Biomedical Informatics, vol. 42, no. 2, pp. 390. (2009) 17. T. Cohen, R. Schvaneveldt, and D. Widdows: Reflective Random Indexing and indirect inference: A scalable method for discovery of implicit connections.
Journal of Biomedical Informatics, vol. 43, no. 2, pp. 240–256. (2010) 18. P. Kanerva, J. Kristofersson, and A. Holst: Random indexing of text samples for latent semantic analysis. In: Proceedings of the 22nd annual conference of the cognitive science society, vol. 1036. (2000) 19. M. Sahlgren: Vector-based semantic analysis: Representing word meanings based on random labels. In: ESSLI Workshop on Semantic Knowledge Acquistion and Categorization. (2001) 20. “Unified Medical Language System (UMLS) - Home.” [Online]. Available: http://www.nlm.nih.gov/research/umls/. [Accessed: 22-May-2013]. 21. C. D. Manning, P. Raghavan, and H. Schütze: Introduction to information retrieval, vol. 1. Cambridge University Press Cambridge (2008) 22. G. Salton, A. Wong, and C.-S. Yang: A vector space model for automatic indexing. Communications of the ACM, vol. 18, no. 11, pp. 613–620. (1975) 23. M. Sahlgren: An introduction to random indexing. In: Methods and Applications of Semantic Indexing Workshop at the 7th International Conference on Terminology and Knowledge Engineering, TKE, vol. 5. (2005) 24. E. Bingham and H. Mannila: Random projection in dimensionality reduction: applications to image and text data. In: Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 245–250. (2001) 25. P. Frankl and H. Maehara: The Johnson-Lindenstrauss lemma and the sphericity of some graphs. Journal of Combinatorial Theory, Series B, vol. 44, no. 3, pp. 355–362. (1988) 26. A. R. Aronson and T. C. Rindflesch: Query expansion using the UMLS Metathesaurus. In: Proceedings of the AMIA Annual Fall Symposium, pp. 485. (1997) 27. B. T. McInnes, T. Pedersen, and S. V. Pakhomov: UMLS-Interface and UMLSSimilarity: open source software for measuring paths and semantic similarity.
In: AMIA Annual Symposium Proceedings, vol. 2009, pp. 431. (2009)
D. Widdows and K. Ferraro: Semantic vectors: a scalable open source package and online technology management application. In: Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), pp. 1183–1190. (2008) D. Widdows and T. Cohen: The semantic vectors package: New algorithms and public tools for distributional semantics. In: IEEE Fourth International Conference on Semantic Computing (ICSC), pp. 9–15. (2010) S. E. Robertson, S. Walker, and M. Beaulieu: Experimentation as a way of life: Okapi at TREC. Information Processing & Management, vol. 36, no. 1, pp. 95– 108. (2000)