Based on previous experience from working on a task-based search engine, we present a list of suggestions and ideas for an Information Retrieval (IR) framework that could inform the development of next generation professional search systems. The specific task that we start from is the clinicians' information need in finding rare disease diagnostic hypotheses at the time and place where medical decisions are made. Our experience from the development of a search engine focused on supporting clinicians in completing this task has provided us valuable insights in what aspects should be considered by the developers of vertical search engines.
In task-based search scenarios, general search engines might not su ce in satisfying user information needs and information seeking behaviour. Searching for rare disease diagnostic hypotheses is one such case where a task-oriented IR system has been developed to adapt to the task-specific user needs.
The diagnostic process is a complex, often non-linear sequence of actions taken by the clinician towards reaching the final diagnosis. However, simply viewed, the process of finding the correct disease consists of generating several diagnostic hypotheses matching the patient case, followed by an iterative process of selection, testing and elimination, after which the final diagnosis is made and corresponding treatment is identified. The clinicians select up to around 6-7 diagnostic hypotheses based on pattern matching the patient data with their medical knowledge and experience [Cam87]. Studies have shown that having a good list of diagnostic hypotheses is key to reaching correct diagnosis. It was further suggested that in many of the cases of misdiagnosis the correct disease was not included in the list of potential diagnoses [KDM08].
The particular di culty of diagnosing rare diseases stems from their low prevalence (less than 1 in 2000 people being a↵ ected), large number (between 5000-8000 distinct diseases), and non-specific symptoms. Therefore, clinicians seldom encounter patients su↵ ering from rare diseases, and considering the rate of biomedical publishing1, might not be familiar with the latest findings. It was shown that around 40% of the rare disease patients are misdiagnosed and 25% experience diagnostic delays between 5-30 years [EUR04]. Furthermore, statistics show that there are around 30 million European citizens a↵ ected by a rare disease, and treatment is critical in many of the cases [EUR04]. This makes the diagnosing of rare diseases a di cult task for clinicians and at the same time it provides a promising ground for IR research. 1.3
For the medical field, several professional IR systems have been developed over the years with the goal of helping medical personnel in completing their tasks at the time and place where clinical decisions are made. However, the acceptance rate of such systems is low and some studies suggest that medical personnel would rather use a familiar web search engine instead. [CF12]
Existing IR systems that are focused on rare disease retrieval have several shortcomings. General purpose search engines, such as Google, although popular amongst clinicians, cannot model the specific task, and maybe more damaging, include low quality data in their indexes. Specialised medical systems, such as Phenomizer2 or Orphanet3, restrict the input to a limited set of medical concepts (ICD codes or MEDLINE terminology), usually selected from a drop-down list. Such limitations could make it di cult and time consuming for clinicians to accurately input patient data. Specialised information databases with search interfaces such as PubMed4 exhibit similar limitations, requiring the use of complex queries with boolean operators.
This work asks what design considerations are needed for the development of an IR system to support clinicians in the process of diagnosing rare diseases. To address this question, the current work looks at previous research done by the authors towards the task of diagnosing rare diseases with the help of IR technologies, and based on the findings, provides suggestions on developing similar systems for di↵ erent tasks.
The rest of the paper is organised as follows. Section 2 provides an overview of previous work related to the described task. Section 3 discusses considerations for developing vertical search engines based on the authors’ experience. Section 4 summarises and concludes this work. 2
Previous work conducted by the authors consisted in the design, development and evaluation of an IR system specialised on retrieving rare and genetic diseases5 information based on queries consisting of patient data, with the goal of helping clinicians in finding diagnostic hypotheses for di cult to diagnose cases [DPL+11].
The system, called FindZebra6, receives symptoms, test results, or any textual data as input, and ranks medical documents based on their estimated relevance to the query. The medical documents are indexed and retrieved using the Indri open-source search engine7. The system provides a unified interface for searching medical documents crawled from 10 di↵ erent content sources which were chosen based on specific criteria: medical articles discussing rare or genetic diseases that are curated and maintained by medical professionals or institutions.
Further work [DPLW12], intended to optimise the IR systems for the task of rare disease diagnosis, focused on mapping entities from UMLS Metathesaurus8 medical ontologies and classifications to the indexed medical articles. With this annotation in place, the search engine has received the additional functionalities of grouping and ranking diseases rather than documents, where diseases are concepts in the ontologies of the UMLS Metathesaurus.
A task-oriented evaluation strategy has been considered. On queries consisting of patient symptoms extracted from real medical cases, we have experimentally evaluated FindZebra against the search results provided by PubMed and Google. Our findings showed that FindZebra is more suited for the task-specific requirements in terms of precision and time spent searching [DPL+11]. Google is not optimised for the characteristics of this task, and the quality of some of the content it indexes is a potential issue in professional search. On the other 1Around 2000-4000 medical references are published daily on MEDLINE, www.nlm.nih.gov/pubs/factsheets/medline.html 2compbio.charite.de/phenomizer 3www.orpha.net 4www.ncbi.nlm.nih.gov/pubmed 5The motivation behind including articles about genetic diseases is that around 80% of the rare diseases are of genetic origin. 6Zebra is sometimes used in medicine to denote a surprising diagnosis. FindZebra can be accessed at findzebra.com 7lemurproject.org/indri 8www.nlm.nih.gov/research/umls hand, PubMed contains high-quality articles, but the interaction design is poor when it comes to solving the task of diagnosing rare diseases.
The conclusion from previous work on FindZebra is that rare disease retrieval can be seen as a distinct IR task, with its specific characteristics. (i) Clinicians’ queries can consist of a long list of patient symptoms, usually longer than the average length of queries sent to general web search engines. (ii) Sometimes symptoms specific to a disease might be missing from the query, or symptoms in the queries might not be specific to the correct disease. (iii) Popularity based metrics and index pruning do not necessarily benefit the retrieval of rare disease articles.
To optimise the retrieval of rare disease information, previous work has consisted mainly in the analysis and selection of high-quality information, integration of the di↵ erent content sources in one location, the design of a new vertical search engine, the analysis and selection of existing medical ontologies, the fusion of medical articles with medical concepts from ontologies, and the proposal of a task oriented evaluation. 3
Considerations in developing vertical search engines for knowledge workers improve task performance. For example, clinicians specialising on rare diseases might be interested in limiting the retrieval to sources that only cover rare or genetic disorders. 4
Diagnosing rare diseases is a task that can be potentially improved with the support of a professional IR system, and can have an important impact on the outcome of patients. Curated, high-quality information for this specific case is available. The need of a well-designed user interface is still an open research area for IR, humancomputer interaction and related fields. This work however has focused on design considerations for integrating IR technologies that support the diagnostic task.
There are certainly other verticals, apart from the medical domain, that can benefit from a similar approach to the one taken for this vertical. There is an increasing amount of information made digitally available for various professions, but o↵ -the-shelf search engines are not optimised to the specific needs of each knowledge worker. Vertical search engines can be adapted to satisfy specific information needs and support di cult tasks.
Thus, we see the potential of a framework that could simplify and speed up the development of such specialised solutions. A framework providing reusable components for vertical search engines could lower the barrier for deploying customised task-based search engines, helping knowledge workers to cope with the complexity of finding and analysing domain-specific information. [CF12]
R. Chisholm and J.T. Finnell. Emergency department physician internet use during clinical encounters. In AMIA Annual Symposium Proceedings, volume 2012, page 1176. American Medical Informatics Association, 2012.
H.C.H. Coumou and F.J. Meijman. How do primary care physicians seek answers to clinical questions? a literature review. Journal of the Medical Library Association, 94(1):55, 2006. [DPLW12] R. Dragusin, P. Petcu, C. Lioma, and O. Winther. Zebra: Searching for rare diseases a case of task-based search in the medical domain. Proceedings of the ECIR 2012 Workshop on Task-Based and Aggregated Search (TBAS2012), page 36, 2012. [EUR04]
EURORDIS. Eurordiscare2: Survey of diagnostic delays, 8 diseases, Europe, 2004.