<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>A New Method to Retrieve, Cluster And Annotate Clinical Literature Related To Electronic Health Records</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Izaskun Fernandez</string-name>
          <email>ifernandez@tekniker.es</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Ana Jimenez-Castellanos</string-name>
          <email>ajimenez@infomed.dia.fi.upm.es</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Xabier Garc´ıa de Kortazar</string-name>
          <email>xkortazar@tekniker.es</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>David Perez-Rey</string-name>
          <email>dperez@infomed.dia.fi.upm.es</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Dept. Int. Artificial, Facultad de Inform ́atica, Universidad Polit ́ecnica de Madrid</institution>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Tekniker-IK4</institution>
        </aff>
      </contrib-group>
      <fpage>19</fpage>
      <lpage>26</lpage>
      <abstract>
        <p>The access to medical literature collections such as PubMed, MedScape or Cochrane has been increased notably in the last years by the web-based tools that provide instant access to the information. However, more sophisticated methodologies are needed to exploit efficiently all that information. The lack of advanced search methods in clinical domain produce that even using well-defined questions for a particular disease, clinicians receive too many results. Since no information analysis is applied afterwards, some relevant results which are not presented in the top of the resultant collection could be ignored by the expert causing an important loose of information. In this work we present a new method to improve scientific article search using patient information for query generation. Using federated search strategy, it is able to simultaneously search in different resources and present a unique relevant literature collection. And applying NLP techniques it presents semantically similar publications together, facilitating the identification of relevant information to clinicians. This method aims to be the foundation of a collaborative environment for sharing clinical knowledge related to patients and scientific publications.</p>
      </abstract>
      <kwd-group>
        <kwd>Electronic Health Record</kwd>
        <kwd>search engines</kwd>
        <kwd>literature retrieval</kwd>
        <kwd>integration</kwd>
        <kwd>federated search</kwd>
        <kwd>collaborative environment</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>Web technologies have produced an explosion in the production and availability
of clinical publications. Access to massive amounts of information by the
practitioners, has led to include literature search tools to support patient diagnosis
and treatment. However, clinical publications are focused on population groups
rather than specific patients. So in order to find relevant publications that fit
with specific characteristics of a particular patient, clinicians must generalize
the characteristics of the Electronic Health Record (EHR) to meet the generic
patterns of the relevant literature.</p>
    </sec>
    <sec id="sec-2">
      <title>Cli2nical LitFeerrantuarnedRezeleatt.eadl.To Electronic Health Records</title>
      <p>Clinicians accessing PubMed , MedScape or Cochrane for searching
publications related to patients frequently get too many results which they should
revise in order to recover relevant information. In addition, biomedical
information is nowadays distributed across different repositories, so they have to execute
multiple queries in the distributed resources to retrieve publications regarding
a specific patient and pathology. In other areas, meta-search engines and other
tools have been widely used facilitating the integration of information from
multiple sources. However, in clinical practice, there are few trials exploiting the
potential of such cutting-edge technologies.</p>
      <p>
        In this work we propose a method aiming to provide a robust environment for
searching literature in the daily clinical practice of the physicians. The proposed
method is based on previous works regarding EHR-based literature retrieval [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ],
improving the integration of searching results and extending its functionality
with publications clustering and annotation strategies.
      </p>
      <p>The article is structured as follows: the next section presents related works; in
section 3 the method for retrieving and analysing biomedical literature based on
EHRs and clusterization techniques is described; section 4 presents the obtained
results and evaluation within the Tratamiento 2.0 project environment; to finish,
conclusions and future works are explained.
2</p>
      <sec id="sec-2-1">
        <title>Background</title>
        <p>
          In recent years there has been an increasing interest in extracting and enriching
the content of EHR standards ([
          <xref ref-type="bibr" rid="ref2">2</xref>
          ], [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ]) such as: Health Level 7 Clinical
Document Architecture (HL7-CDA) [
          <xref ref-type="bibr" rid="ref4">4</xref>
          ] and openEHR [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ]. Since the majority of the
content of EHR and clinical data is free text instead of structured information,
Natural Language Processing techniques are frequently required. For instance,
the method presented in [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ] is based on GATE (General Architecture for Text
Engineering) and the UMLS standard vocabulary to extract specific information
from pathology reports.
        </p>
        <p>
          Previous works also exist on discovering biomedical relationships from
semantic annotations [
          <xref ref-type="bibr" rid="ref8">8</xref>
          ] [
          <xref ref-type="bibr" rid="ref9">9</xref>
          ], but they are limited to present just PubMed abstract
collections as result. In [
          <xref ref-type="bibr" rid="ref1">1</xref>
          ], the authors presented a system to support clinicians
in the literature search process using EHR information. For literature retrieval,
the federated search methodology [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ] was used for integrating results obtained
from different information sources, pre-selected by the user. Search engines like
Sphinx [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ] help in the development of this kind of systems, indexing the
information and providing a fast information retrieval.
        </p>
        <p>
          In information retrieval systems, the query formulation is a crucial step to
obtain suitable results. The EBM (Evidence-Based Medicine Working Group
1992) recommends to create clinical questions using the PICO frame. Improved
results were obtained using the PICO query format [
          <xref ref-type="bibr" rid="ref11">11</xref>
          ] [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ] but, the creation of
this queries is not a trivial task as it is shown in [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ].
        </p>
        <p>The literature in the area describes numerous efforts on both, extracting
information from EHR and improving accuracy and efficiency of clinical searches,
but new tools are required to provide support to clinicians.
3</p>
      </sec>
      <sec id="sec-2-2">
        <title>Method</title>
        <p>We propose a method with a modular architecture to address the requirements
of an EHR-based literature process. The main modules are: a query generation
module, a federated-search module, an automatic literature processing process
and finally a publication annotation module. The architecture is graphically
presented at Fig.1</p>
        <p>The EHR-Based Query Generation module deals with electronic health record
information, extracting the most relevant features regarding specific pathologies.
These characteristics are used to give context to the searching criteria specified
by the expert for construct the final query, which will be used in the searching
phase to query different repositories in the second module. The method
manages every independent result set obtained from each repository, presenting the
result as a unique and integral literature collection, semantically annotated and
clustered in the third module. Within the fourth module, clinicians may also
annotate and comment the publications of the collection.
The aim of the first module is to support the clinician on literature query
generation for a certain criterion regarding an individual patient and disease,
formulating the query based on PICO model. The method combines both the
relevant information of the individual patient for the particular disease and the
theme about the clinician wants to search about. A PICO query is composed
by four main elements, P,I,C and O: P represents the problem/population; I
intervention’s information; C the comparison; and finally O the outcome. When
constructing PICO queries not always all the elements are necessary.</p>
        <p>Let’s assume that a clinician wants to search about if there is any
contraindication to prescribe ECA inhibitors (described as current criterion) for Patient A
that is a chronic diabetic patient (previously mentioned as pathology). In PICO</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>Cli4nical LitFeerrantuarnedRezeleatt.eadl.To Electronic Health Records</title>
      <p>format, the P element should be represented by the most relevant characteristics
of Patient A about his/her chronic diabetes; the I element by ECA inhibitors
drug; and the O element by contraindication, which is the outcome the clinician
wants to know about. For this query, since no comparison criterion is used, the
C element would be empty.</p>
      <p>
        Since the aim of this module is to automatize the process of extracting and
assigning values to PICO components, it applies different strategies to the input
parameters. For P definition, the module uses a a web service that extracts the
relevant characteristics from the input EHR about a certain pathology. As it is
detailed in [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ], the web service uses a knowledge base (KB) which defines the
relevant parameters and their values. And applying the transformation rules in
the KB it translated the structured or numeric values into concepts for P, such
as high glucose.
      </p>
      <p>For completing I, C and O components in a efficient way, minimizing the
non relevant information expressed on the clinician’s expressions concerning to
the current practice to search about, the module applies a NLP process. A NLP
process which takes as input a Spanish or English free text and extracts the most
relevant terminology. For that purpose it uses a GATE3 application which
combines a tokenizer, sentence splitter, part of speech tagger, stemmer and finally
an ontology based gazetteer tagger, and tags all the expressions in the text
referring to any term in the ontology. The stemmer application is a crucial step in
order to identify not only the perfect matching expressions but also the inflected
mentions of the ontology vocabulary, such as potassium ion levels, the pluralized
form of potassium ion level ontology concept. The ontology for the gazetteer
tagger is parametrizable, which makes the module portable for different pathologies
and domains.</p>
      <p>The EHR-Based Query Generation interacts with the clinician, presenting
the automatically extracted terminology and he/she must select the most
relevant ones for each PICO field.
3.2</p>
      <sec id="sec-3-1">
        <title>Federated Search</title>
        <p>Federated search module allows searching different repositories simultaneously
with the same query. In this module clinicians can not only select the desired
resources from a predefined list, but also they can weight the selected resources
establishing preferences among them. The influence of these preferences in results
will be sown at 3.3 section.</p>
        <p>The query defined in the previous module is used to search every selected
source. Since each resource should manage the information in a different way, a
configuration file is associated to each source defining the way to ask and how
to get results from it. Concretely, the configuration file specifies how the PICO
query should be transformed for asking the current resource and also, how to
3 General Architecture for Text Engineering is a open source software capable of
solving almost any text processing problem. More information at http://gate.ac.uk/
interpret the searching output. The configurations files are implemented using
xslt, a declarative language designed for defining XML file transformations.</p>
        <p>Using xslt files for specifying source dependent characteristic instead of
including them in the module code, makes it flexible. Flexible in the sense that
defining a xslt configuration file is enough for including a new source in the
module.</p>
        <p>So the module accesses to the configuration files of the selected sources
transforming the PICO query in the corresponding formats, asks the sources and
again, using the configuration files, it interprets each output and stores and
merges all the publications result sets in the server database as a unique
collection. For each publication this module gets the title (required), and a short
description, authors and the publication date when they are available. Since some
sources can share literature, the method eliminates duplicated publications using
title information.
3.3</p>
      </sec>
      <sec id="sec-3-2">
        <title>Literature Processing</title>
        <p>Literature collections without duplications are used in the following steps to
present not only a list of relevant publications, but also an enriched set of
scientific literature. In the server database for each publication among others, there
is available at least the title and a short description if it is extracted at searching
phase. Applying to that literature content the NLP terminology extractor
process described at Section 3.1, this module obtains the relevant terminology on
each publication. Since each publication is annotated separately, the method can
access to the terminology of each item on the collection and present the entire
collection as: (i) a content based relevance ranked list, (ii) a list of publications
ordered by both content and source relevance or (iii) a semantically clustered
collection.</p>
        <p>In the first two options, using Sphinx index engine4 the method compares
the extracted literature terminology with the query terminology. It ranks the
publications measuring the shared terminology in both, query and publication,
and it adjusts the measure with the resource weight for the (ii) modality. This
way in (ii) option publications are ordered by their the relevance respect to the
query, but also by the clinician resource preferences using the belonging resource
weights defined in the federated search module (described at 3.2 section). In the
last option, a clustering strategy is applied to relate and group publications
based on their semantic similarity. The terminology of different publications is
compared, clustering the publications that shares similar terms. This way the
method represent semantically similar publications grouped together.
3.4</p>
      </sec>
      <sec id="sec-3-3">
        <title>Publication Annotation</title>
        <p>Finally the proposed method allows the clinicians to annotate the results with
two main functionalities: creating notes to remark any commentary about a
4 http://sphinxsearch.com/</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>Cli6nical LitFeerrantuarnedRezeleatt.eadl.To Electronic Health Records</title>
      <p>certain publication; and voting each publication relevance respect to the current
query with like/dislike notations.</p>
      <p>Actually all the notes are stored in the server database, in order to give
the opportunity to clinicians to refer their already revised collections without
repeating the entire process and showing the annotations they did in the past.
4</p>
      <sec id="sec-4-1">
        <title>Method Validation</title>
        <p>Two use cases have been developed within the environment of the Tratamiento
2.0 5 project to test the proposed method: diabetes and arterial hypertension.
For each pathology, we have implemented a knowledge base with the relevant
parameters. Based on these parameters, the service automatically extracts from
EHR the characteristic of current patient, and translates it to the defined
vocabulary. Concretely we have integrated English and Spanish version of the MeSH
ontology at the NLP process for this validation. An example rule from the
implemented rule-set is the one treating the systolic pressure (Sp) value: if Sp &lt;=
90: then ”systolic hypotension”.</p>
        <p>A preliminary test set of ten patient data and fifty queries has been used to
check the functionality of the method. The system has extracted correctly the
characteristic for all the patients and it has correctly identified the 90% of the
relevant papers, according to assessment of experts of the project.</p>
        <p>In the context of the project OSI+6, a user interface has been developed to
exploit the modules defined above. With patient health record’s characteristics
and the current query, the searching criterion is defined. At this point the
clinician has also to select which resources wants to use for searching literature and
provide weights if a different importance among resources are identified.
Nowadays, the Cochrane library, PubMed , Fisterra and Ikere are accessible resources
from this implementation. The first two sources contain English literature while
the last two mentioned resources store Spanish publications.</p>
        <p>Based on the defined parametrization, the system searches for literature, and
presents the results to the clinician. Previously launched searching results can be
accessed, since the system maintains a cache of this information —not only the
retrieved literature, but also the notes and the valuation the clinician has done for
each publication. When previous searching results are consulted, this information
is also presented to the clinician. The interface permits to search collection, not
only using free text, but also exploiting the annotated terminology.</p>
        <p>Currently we are working on integrating the clustered results visualization
which is already running as service, but it is not exploited in the interface.
5 http://www.tratamiento20.es/inicio.html
6 http://www.i3b.ibermatica.com/i3b/noticias/2009/
osi-el-hospital-extendido</p>
      </sec>
      <sec id="sec-4-2">
        <title>Conclusions and Future Lines</title>
        <p>In this paper we have presented a method to integrate biomedical literature
repositories with patient EHRs. This method provides a platform for retrieving
literature about a certain patient practice with minimal effort. The PICO query
generation is supported, automatically extracting characteristics from patient
EHR and applying NLP techniques for relevant terminology identification.
Federated search strategy is used to access simultaneously to different information
sources and to present a unified collection. With this method clinicians can edit
the resultant collection, to record their considerations and to evaluate the
relevance of each publication. All the information would be available afterwards
to be accessed by the user without repeating the entire process. This method is
also being standardized by using EHR formats such as HL7-CDA instead of a
proprietary platform format.</p>
        <p>Nowadays only access is provided to all the information generated by the
users regarding the publications —i.e. agreement about the publications’
relevance, tags and notes. But analysing that information, user profiles can be
generated to facilitate model and adjustment of the results based on users’
preferences. So, the method is intended to be the foundation of further development
of a collaborative environments where clinicians could share their searches, notes
and all the knowledge generated around a search. It would became a new
resource of know-how that combines scientific studies with the daily experience of
clinicians themselves.</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Jimenez-Castellanos</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fernandez</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Perez</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Viejo</surname>
            , E., D´ıez,
            <given-names>F.J.</given-names>
          </string-name>
          , Garc´ıa de Kortazar,
          <string-name>
            <surname>X.</surname>
          </string-name>
          , Garc´ıa,
          <string-name>
            <given-names>M.</given-names>
            ,
            <surname>Maojo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            ,
            <surname>Cobo</surname>
          </string-name>
          ,
          <string-name>
            <surname>A</surname>
          </string-name>
          :
          <article-title>Patient based literature retrieval and integration: A use case for diabetes and arterial hypertension</article-title>
          .
          <source>Proceedings of the International Conference on Health Informatics</source>
          ,
          <fpage>33</fpage>
          -
          <lpage>41</lpage>
          (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Meystre</surname>
            ,
            <given-names>S.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Savova</surname>
            ,
            <given-names>G.K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kipper-Schuler</surname>
            ,
            <given-names>K.C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hurdle</surname>
            ,
            <given-names>J.E.</given-names>
          </string-name>
          ,
          <article-title>Extracting information from textual documents in the electronic health record: a review of recent research</article-title>
          .
          <source>IMIA Year-book of Medical Informatics, Methods Inf Med</source>
          <year>2008</year>
          ,
          <volume>47</volume>
          <issue>Suppl 1</issue>
          ,
          <fpage>138</fpage>
          -
          <lpage>154</lpage>
          (
          <year>2008</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Eichelberg</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Aden</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Riesmeier</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dogac</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Laleci</surname>
            ,
            <given-names>G. B.</given-names>
          </string-name>
          :
          <article-title>Electronic health record standards A brief overview</article-title>
          . Information &amp; Communications
          <string-name>
            <surname>Technology (ICICT06) ITI4</surname>
          </string-name>
          (
          <year>2006</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Dolin</surname>
            ,
            <given-names>R.H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Alschuler</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Boyer</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Beebe</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Behlen</surname>
            ,
            <given-names>F.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Biron</surname>
            ,
            <given-names>P.V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Shabo</surname>
            <given-names>Shvo</given-names>
          </string-name>
          ,
          <string-name>
            <surname>A.</surname>
          </string-name>
          :
          <article-title>HL7 Clinical Document Architecture, Release 2</article-title>
          .
          <source>J. Am. Med</source>
          . Inform. Assoc., vol.
          <volume>13</volume>
          (
          <issue>1</issue>
          ),
          <fpage>30</fpage>
          -
          <lpage>39</lpage>
          (
          <year>2006</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Kalra</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Beale</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Heard</surname>
            ,
            <given-names>S.:</given-names>
          </string-name>
          <article-title>The openEHR Foundation</article-title>
          .
          <source>Regional Health Economies and ICT Services: The PICNIC Experience</source>
          ,
          <volume>115</volume>
          /
          <year>2005</year>
          ,
          <fpage>153</fpage>
          -
          <lpage>173</lpage>
          (
          <year>2005</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Lee</surname>
            ,
            <given-names>K. F.</given-names>
          </string-name>
          :
          <article-title>Automatic speech recognition: The development of the SPHINX system</article-title>
          . Kluwer Academic Pub (
          <year>1989</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Liu</surname>
            , K., Mitchell,
            <given-names>K.J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chapman</surname>
            ,
            <given-names>W.W.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Crowley</surname>
            ,
            <given-names>R.S.:</given-names>
          </string-name>
          <article-title>Automating tissue bank annotation from pathology reports - comparison to a gold standard expert annotation set</article-title>
          .
          <source>Proceedings of the AMIA Annu Symp</source>
          ,
          <fpage>460</fpage>
          -
          <lpage>464</lpage>
          (
          <year>2005</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Tsuruoka</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tsujii</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ananiadou</surname>
            <given-names>S.:</given-names>
          </string-name>
          <article-title>FACTA: a text search engine for finding associated biomedical concepts</article-title>
          .
          <source>Bioinformatics</source>
          vol.
          <volume>24</volume>
          (
          <issue>21</issue>
          ),
          <fpage>2559</fpage>
          -
          <lpage>2560</lpage>
          (
          <year>2008</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Kim</surname>
            ,
            <given-names>J.J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pezik</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rebholz-Schuhmann</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          : Medevi:
          <article-title>Retrieving textual evidence of relations between biomedical concepts from medline</article-title>
          .
          <source>Bioinformatics</source>
          ,
          <volume>24</volume>
          (
          <issue>11</issue>
          ),
          <fpage>1410</fpage>
          -
          <lpage>1412</lpage>
          (
          <year>2008</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Jacso</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Thoughts about federated searching</article-title>
          .
          <source>Information Today</source>
          ,
          <volume>21</volume>
          (
          <issue>9</issue>
          ),
          <fpage>17</fpage>
          -
          <lpage>20</lpage>
          (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Schardt</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Adams</surname>
            ,
            <given-names>M. B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Owens</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Keitz</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fontelo</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Utilization of the PICO framework to improve searching PubMed for clinical questions</article-title>
          .
          <source>BMC Med Inform Decis Mak</source>
          ,
          <fpage>7</fpage>
          -
          <lpage>16</lpage>
          (
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Meats</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Brassey</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Heneghan</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Glasziou</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Using the Turning Research Into Practice (TRIP) database: how do clinicians really search?</article-title>
          .
          <source>J. Med Libr Assoc</source>
          <volume>156</volume>
          -
          <fpage>163</fpage>
          (
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Huang</surname>
            ,
            <given-names>X.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lin</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Demner-Fushman</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>Evaluation of PICO as a knowledge representation for clinical questions</article-title>
          .
          <source>AMIA Annual Symposium proceedings</source>
          ,
          <fpage>359</fpage>
          -
          <lpage>363</lpage>
          (
          <year>2006</year>
          )
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>