<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>How can heat maps of indexing vocabularies be utilized for information seeking purposes?</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Peter Mutschke</string-name>
          <email>peter.mutschke@gesis.org</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Karima Haddou ou Moussa</string-name>
          <email>karima.haddououmoussa@gesis.org</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>GESIS - Leibniz Institute for the Social Sciences</institution>
          ,
          <addr-line>Cologne</addr-line>
          ,
          <country country="DE">Germany</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>The ability to browse an information space in a structured way by exploiting similarities and dissimilarities between information objects is crucial for knowledge discovery. Knowledge maps use visualizations to gain insights into the structure of large-scale information spaces, but are still far away from being applicable for searching. The paper proposes a use case for enhancing search term recommendations by heat map visualizations of co-word relationships taken from indexing vocabulary. By contrasting areas of different “heat” the user is enabled to indicate mainstream areas of the field in question more easily.</p>
      </abstract>
      <kwd-group>
        <kwd />
        <kwd>Knowledge Maps</kwd>
        <kwd>Interactive Information Retrieval</kwd>
        <kwd>Heat Maps</kwd>
        <kwd>Information Seeking</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Introduction</title>
      <p>
        During the past decade interactive search interfaces [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ] have emerged as important in
Information Retrieval (IR) research. The insight that the success of information
seeking mainly depends on the ability of an information system to properly support
interaction between user and system has led to the establishment of Interactive
Information Retrieval (IIR) as a specific discipline within IR research. Corresponding to
this, whole-session retrieval issues [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] as well as search interfaces [
        <xref ref-type="bibr" rid="ref3 ref4 ref5">3,4,5</xref>
        ] became a
focal point in research, in particular with respect to exploratory searching [
        <xref ref-type="bibr" rid="ref4 ref6">4,6</xref>
        ]. Due
to the enormous increase of information spaces bibliometric enhanced IR models
addressing non-textual attributes of the domain under study became also more and
more important at the same time. This is particular true for the case of scholarly
searching [
        <xref ref-type="bibr" rid="ref7 ref8">7,8</xref>
        ]. Moreover, studies in interactive information seeking behavior have
confirmed that the ability to browse an information space in a structured way by
exploiting similarities and dissimilarities between information objects is crucial for
knowledge discovery [
        <xref ref-type="bibr" rid="ref10 ref9">9,10</xref>
        ].
      </p>
      <p>
        Knowledge maps, on the other hand, use visualizations to gain insights into the
structure of large-scale information spaces. They can take very different forms such as
network visualizations, heat maps, tree maps or geographic map like arrangements of
information spaces [
        <xref ref-type="bibr" rid="ref11 ref12 ref13 ref14 ref15 ref16 ref17">11-17</xref>
        ]. However, knowledge maps are still far away from being
applicable as search interfaces for Digital Libraries. Most maps are static
visualizations made for special purposes, and hence neither interactive [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ] nor dynamic, i.e.
they do not adapt to the change of user perspective to an information space during
interaction. Thus, combining knowledge mapping with IR is still a challenging
research issue. There are just a few examples where visual concepts also used in
knowledge mapping have been applied to information systems, such as DANSEasy1,
which visualizes an archive’s category structure and its content in form of a
dendrogram and a tree map to be used as a navigation tool through the information space.
Another example is PepBank2 which uses heat maps for visualizing and refining
search results.
      </p>
      <p>
        Interface studies have shown that a simple spatial interface layout performs better
than complex ones [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. Heat maps are simple visualizations of data in a color-coded
2-dimensional matrix where cells have a particular color index indicating remarkable
values of the matrix. To the best of our knowledge, heat maps have not been used for
the query formulation process so far. This position paper discusses a use case for
using heat map visualizations of relationships between indexing terms to enhance search
term recommendations.
2
      </p>
    </sec>
    <sec id="sec-2">
      <title>Using Heat Maps of Co-Word Relationships for Searching</title>
      <p>
        A particular point of failure of current information systems is the vagueness between
user search terms and the terms used for indexing the documents to be retrieved, i.e.
the indexing terms which are usually based on a controlled vocabulary such as a
thesaurus [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ]. Search term recommenders (STRs) provide models that map a user’s
search term to more appropriate terms. The model proposed by [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ], for instance, maps
search terms to indexing terms on the basis of a co-word analysis and recommends
indexing terms that strongly co-occur with the search term. The expectation here is
that retrieval quality will increase when indexing terms are used for searching.
Indeed, a retrieval evaluation showed that the use of STRs relying on controlled
vocabulary has a great potential to improve the precision of a search significantly [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ].
      </p>
      <p>
        However, search term recommendations usually appear in forms which do not
assist the user in locating the information need within the wider information space. The
STR provided by the Social Science literature portal sowiport3, for instance, displays
recommended indexing terms in a drill down menu (see Fig. 1, cp. [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]). From that list
the user can select more appropriate terms for searching. The major problem however
is that the user’s choice is not facilitated by further potentially helpful information, in
particular structural information about the semantic contexts in which a recommended
term appears.
1 http://www.drasticdata.nl/ProjectDANSEasy/indexMultipleAssignments.htm
2 http://pepbank.mgh.harvard.edu/
3 http://sowiport.gesis.org/
      </p>
      <p>Fig. 1. Search Term Recommendations in Sowiport. Recommended indexing terms strongly
co-occurring with the search term are displayed in a one-dimensional drilldown menu4.
Our approach therefore is to extend the initial list of term suggestions by further
indexing terms that frequently co-occur with the search term as well as with the initially
suggested terms. This yields a two-dimensional space of co-word relationships where
the initially recommended terms represent one dimension (first order terms) and the
indexing terms co-occurring with the search term as well as with first order terms
represent the other dimension (second order terms). A natural way to visualize such a
matrix of co-word relationships is a heat map visualization (see Fig. 2) which displays
the strength of each relationship (its “heat”) by a color on a scale ranging from red
(indicating high values) to blue (indicating low values). To keep it simple just the
frequency of co-occurrences was taken to indicate the strength of a co-word
relationship (see numbers in the cells of Figure 25). The background color code of the cells of
the map is calculated according to the ratio of the individual frequencies to the
maximum and minimum values of the matrix. By this color-coded visualization the heat
map is divided into “hot” (red), “warm” (yellow, green) and “cold” (blue) areas. Thus,
red cells point to topic combinations which appear most frequently, blue cells
correspond to areas which appear least frequently in the map. Red cells therefore represent
issues which are more heavily discussed in the research field (“hot” topics of the
fields). Accordingly, blue cells represent issues which are less heavily discussed
(compared to “hot” fields). Hence, for red areas the user can expect to find more
documents than for blue areas.
4 Here, the search term “violence” is mapped to the indexing terms “Gewalt” (“Violence”),
“Gewaltbereitschaft” (“Propensity to violence”), “Jugendlicher” (“Adolescent”), “Krieg”
(“War”), and “Rechtsradikalismus” (“Right-wing radicalism”), based on the thesaurus for
the Social Sciences provided by GESIS.
5 The map displays the frequency of documents containing the respective term combination
(see numbers in the cells as well as numbers next to the terms). Thus, the total number of
documents containing the search term “Violence” is 9718; the total number of documents
containing the search term “Violence” and the first order term “Adolescent” is 1934; the
total number of documents containing the search term “Violence” and the second order term
“Right-wing radicalism” is 846; and the total number of documents containing the search
term “Violence”, the first order term “Adolescent” as well as the second order term
“Rightwing radicalism”is 405.</p>
      <p>The goal of the heat map visualization is to enable the user to indicate mainstream
areas of the research field more easily. This is relevant for the case where a user starts
with a search term and needs an overview of main issues of the field in question.
Figure 2 displays, for the example from Figure 1, a heat map of indexing terms that are
closely related to the search term “Violence”. The column headings of the heat map
show the 10 first order terms that most frequently co-occur with the search term. The
terms are displayed in descending order of the frequency of their co-occurrence with
the search term, i.e. “Adolescent”, followed by “Developing country”, “Propensity to
violence” and so on. The row headings of the heat map display the top three second
order terms, i.e. the three indexing terms that most frequently co-occur with the
respective first order term as well as with the search term6. By this, the heat map
visualizes tuples of indexing terms as well as their “heat” with respect to the search term.</p>
      <p>The heat map displayed in Figure 2 clearly indicates the combination “Developing
country”+“Asia” and “Developing country”+“Africa” as “hot” areas of research in the
context of „Violence“, followed by “Adolescent”+“Right-wing radicalism”,
“Adolescent”+“Child”, “Developing country”+“Latin America”, “Right-wing
radicalism”+“Xenophobia” and “Propensity to violence”+”Adolescent”. By clicking a term
or into a cell the user can browse the documents related to topic combination in
question.
6 The list of second order terms is reduced to a disjoint set. For this example, this yields a list of
19 terms (instead of 10x3=30 terms).</p>
      <p>It is important to note that instead of visualizing the entire information space by a
heat map, the approach here is to reduce the information space to the fraction that
matches the information need of the user. The basic idea is to dynamically adapt the
heat map of term recommendations when the user clicks into the map or changes
search terms, i.e. throughout the entire retrieval process.
3</p>
    </sec>
    <sec id="sec-3">
      <title>Conclusion and Future Work</title>
      <p>The paper proposes a use case for using heat map visualizations of term
cooccurrence matrices which can be used as a visual navigation tool through an
information space. The approach is to provide a “big picture” view of the relevant fraction
of the information space that can be adapted dynamically during a search session.
Contrasting areas of different “heat” within a set of co-word relationships highly
associated with the user’s search term enables the user to indicate main issues of the
field in question more easily. We assume that this approach might also help the user
to better locate a particular information need within a larger information space. The
paper may also provide a principle idea of how knowledge maps of information
spaces can be integrated in information seeking processes.</p>
      <p>
        For future work we intend to evaluate the proposed approach with real users on the
basis of an evaluation panel [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ], apply the approach also to result sets, find more
suitable metrics and color codes for calculating and displaying heat maps, and –
finally – develop a generic model for the use of heat map visualizations of term
suggestions in interactive retrieval systems.
      </p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Ruthven</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kelly</surname>
            ,
            <given-names>D</given-names>
          </string-name>
          . (eds.):
          <article-title>Interactive Information Seeking, Behaviour and Retrieval</article-title>
          . London:Facet Publishing (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Belkin</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dumais</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kando</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Sanderson</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>NII Shonan meeting on WholeSession Evaluation of Interactive Information Retrieval Systems</article-title>
          ,
          <year>October 2012</year>
          , http://www.nii.ac.jp/shonan/seminar020 (
          <year>2012</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Hearst</surname>
            ,
            <given-names>M.A.</given-names>
          </string-name>
          :
          <article-title>Search User Interfaces</article-title>
          . Cambridge University Press. New York (
          <year>2009</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4. Wilson,
          <string-name>
            <given-names>M.L.</given-names>
            ,
            <surname>Kules</surname>
          </string-name>
          ,
          <string-name>
            <surname>B.</surname>
          </string-name>
          :
          <article-title>From Keyword Search to Exploration: Designing Future Search Interfaces for the Web</article-title>
          .
          <source>Foundations and Trends in Web Science</source>
          <volume>2</volume>
          (
          <issue>1</issue>
          ),
          <fpage>1</fpage>
          -
          <lpage>97</lpage>
          (
          <year>2010</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Dix</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Introduction to Information Visualization</article-title>
          . In: Agosti,
          <string-name>
            <surname>M.</surname>
          </string-name>
          et al. (eds.):
          <article-title>Information retrieval meets Information Visualization</article-title>
          .
          <source>PROMISE Winter School 2012. LNCS 7757</source>
          , Springer, Berlin-Heidelberg,
          <fpage>1</fpage>
          -
          <lpage>27</lpage>
          (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Marchionini</surname>
          </string-name>
          , G.:
          <article-title>Exploratory Search: from Finding to Understanding</article-title>
          .
          <source>Communications of the ACM</source>
          <volume>49</volume>
          (
          <issue>4</issue>
          ),
          <fpage>41</fpage>
          -
          <lpage>46</lpage>
          (
          <year>2006</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Mutschke</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Mayr</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Schaer</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sure</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          :
          <article-title>Science models as value-added services for scholarly information systems</article-title>
          .
          <source>In: Scientometrics</source>
          ,
          <volume>89</volume>
          (
          <issue>1</issue>
          ),
          <fpage>349</fpage>
          -
          <lpage>364</lpage>
          (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Mayr</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Scharnhorst</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Larsen</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Schaer</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Mutschke</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Bibliometric-enhanced information retrieval</article-title>
          . In: de Rijke,
          <string-name>
            <surname>M.</surname>
          </string-name>
          et al. (eds.):
          <source>Advances in information retrieval: 36th European Conference on IR Research</source>
          , ECIR
          <year>2014</year>
          , Amsterdam, The Netherlands,
          <source>April</source>
          <volume>13</volume>
          -
          <fpage>16</fpage>
          . LNCS vol.
          <volume>8416</volume>
          , Berlin: Springer,
          <fpage>798</fpage>
          -
          <lpage>801</lpage>
          (
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Nicholas</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          et.al.:
          <article-title>Reappraising Information Seeking Behaviour in a Digital Environment: bouncers, checkers, returners, and the like</article-title>
          ,
          <source>Journal of Documentation</source>
          ,
          <volume>60</volume>
          (
          <issue>1</issue>
          ),
          <fpage>24</fpage>
          -
          <lpage>43</lpage>
          (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Westerman</surname>
            , S.
            <given-names>J.</given-names>
            ; Collins, J.
          </string-name>
          ; Cribbin,
          <string-name>
            <surname>T.</surname>
          </string-name>
          :
          <article-title>Browsing a Document Collection Represented in Two-</article-title>
          and
          <string-name>
            <surname>Three-dimensional Virtual</surname>
          </string-name>
          Information Space,
          <source>International Journal of HumanComputer Studies</source>
          ,
          <volume>62</volume>
          (
          <issue>6</issue>
          ),
          <fpage>713</fpage>
          -
          <lpage>736</lpage>
          (
          <year>2005</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Börner</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ;
          <string-name>
            <surname>Chen</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ;
          <string-name>
            <surname>Boyack</surname>
            ,
            <given-names>K. W.</given-names>
          </string-name>
          :
          <article-title>Visualizing knowledge domains</article-title>
          .
          <source>Annual Review of Information Science and Technology</source>
          ,
          <volume>37</volume>
          ,
          <fpage>179</fpage>
          -
          <lpage>255</lpage>
          (
          <year>2003</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Shiffrin</surname>
          </string-name>
          , R.; Börner,
          <article-title>Katy: Mapping Knowledge Domains</article-title>
          .
          <source>Proceedings of the National Academy of Sciences of the United States of America</source>
          , Vol.
          <volume>101</volume>
          (
          <issue>Suppl</issue>
          . 1) (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Börner</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          :
          <article-title>The Atlas of Science</article-title>
          . MIT Press (
          <year>2010</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Klavans</surname>
            , R.; Boyack,
            <given-names>K. W.</given-names>
          </string-name>
          :
          <article-title>Toward an objective, reliable, and accurate method for measuring research leadership</article-title>
          .
          <source>Scientometrics</source>
          ,
          <volume>82</volume>
          (
          <issue>3</issue>
          ),
          <fpage>539</fpage>
          -
          <lpage>553</lpage>
          (
          <year>2010</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Skupin</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ;
          <string-name>
            <surname>Biberstine</surname>
            ,
            <given-names>J.R.</given-names>
          </string-name>
          ; Börner,
          <string-name>
            <surname>K.</surname>
          </string-name>
          :
          <article-title>Visualizing the Topical Structure of the Medical Sciences: A Self-Organizing Map Approach</article-title>
          .
          <source>PLoS ONE</source>
          <volume>8</volume>
          (
          <issue>3</issue>
          ): e58779 (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Sahal</surname>
            ,
            <given-names>A.A.</given-names>
          </string-name>
          ;
          <string-name>
            <surname>Wyatt</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ;
          <string-name>
            <surname>Passi</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ;
          <string-name>
            <surname>Scharnhorst</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <string-name>
            <surname>Mapping</surname>
            <given-names>EINS</given-names>
          </string-name>
          -
          <article-title>An exercise in mapping the Network of Excellence in Internet Science</article-title>
          .
          <source>Conference Proceedings of the First International Conference on Internet Science</source>
          , Brussels,
          <fpage>75</fpage>
          -
          <lpage>78</lpage>
          (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Boyack</surname>
            ,
            <given-names>K. W.</given-names>
          </string-name>
          ; Klavans,
          <string-name>
            <surname>R.</surname>
          </string-name>
          :
          <article-title>Creation of a highly detailed, dynamic, global model and map of science</article-title>
          .
          <source>Journal of the American Society for Information Science and Technology (</source>
          <year>2013</year>
          , forthcoming)
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <given-names>Akdag</given-names>
            <surname>Salah</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.A.</given-names>
            ;
            <surname>Scharnhorst</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.; Ten</given-names>
            <surname>Bosch</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            ;
            <surname>Doorn</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            ;
            <surname>Manovich</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            ;
            <surname>Salah</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.A.</given-names>
            ;
            <surname>Chow</surname>
          </string-name>
          ,
          <string-name>
            <surname>J.</surname>
          </string-name>
          :
          <article-title>Significance of Visual Interfaces in Institutional and User-Generated Databases with Category Structures</article-title>
          . Proceedings of the second international ACM workshop on “
          <article-title>Personalized access to cultural heritage (PATCH)”</article-title>
          . ACM Multimedia Conference, Nara, Japan (
          <year>2012</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Petras</surname>
          </string-name>
          , V.:
          <article-title>Translating Dialects in Search: Mapping between Specialized Languages of Discourse and Documentary Languages</article-title>
          . University of California, Berkley (
          <year>2006</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Kern</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ; Mutschke,
          <string-name>
            <surname>P.</surname>
          </string-name>
          ; Mayr,
          <string-name>
            <surname>P.</surname>
          </string-name>
          :
          <article-title>Establishing an Online Access Panel for Interactive Information Retrieval Research</article-title>
          .
          <source>In: 2014 IEEE/ACM Joint Conference on Digital Libraries</source>
          ,
          <fpage>473</fpage>
          -
          <lpage>474</lpage>
          . London, United Kingdom: IEEE. doi:
          <volume>10</volume>
          .1109/JCDL.
          <year>2014</year>
          .
          <volume>6970231</volume>
          (
          <year>2014</year>
          ).
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>