=Paper=
{{Paper
|id=Vol-2080/paper7
|storemode=property
|title=Users are not Influenced by High Impact and Core Journals while Searching
|pdfUrl=https://ceur-ws.org/Vol-2080/paper7.pdf
|volume=Vol-2080
|authors=Ameni Kacem,Philipp Mayr
|dblpUrl=https://dblp.org/rec/conf/ecir/KacemM18
}}
==Users are not Influenced by High Impact and Core Journals while Searching==
https://ceur-ws.org/Vol-2080/paper7.pdf
BIR 2018 Workshop on Bibliometric-enhanced Information Retrieval
Users are not influenced by high impact and core
journals while searching
Ameni Kacem, Philipp Mayr
GESIS – Leibniz Institute for the Social Sciences, Cologne, Germany
[ameni.sahraoui,philipp.mayr]@gesis.org
Abstract. Much attention has been paid recently in analyzing the inter-
action between the user and the search system. Interactive information
retrieval researchers have become more interested in studying the user
behavior in academic search engines in order to improve ranking and to
better understand the user expectations. In our previous work, we ana-
lyzed the user behavior towards Marcia Bates’ search stratagems such as
’footnote chasing’, ’citation search’, ’keyword search’ and ’author search’.
In this paper, we propose to examine in more detail the user behavior
towards journal paper search known as the ’journal run’ stratagem. We
study the frequency and the stage of ’journal run’ use in a social sciences
academic search engine log. We found that the journal run is mostly used
at the end of retrieval sessions and has an approximate usefulness of 55%
measured by the presence of positive user actions in the log. Moreover,
we include typical journal metrics in our case study and test whether
there is an influence of the journals’ impact factor (IF) and coreness of a
journal on the overall session conduct. We found that the metric IF and
coreness of a journal do not have a measurable impact on the journals
utilization of social sciences users.
Keywords: Log Analysis, Interactive Information Retrieval, Stratagem Search,
Journal Run, Information Behavior, Social Sciences Users.
1 Introduction
Much attention has been paid recently in analyzing the interaction between the
user and the search system. In fact, researchers have moved from considering only
the current query and result set to focus more on the user’s past interactions
and the analysis of whole retrieval sessions. Research approaches in interactive
information retrieval aim to understand the user searching behavior in order to
improve the ranking of results after submitting a query and enhance the user
experience within an IR system.
Digital Libraries (DLs) users behave differently when interacting with the
system as underlined by Marcia Bates who highlighted different concepts such
as moves, tactics, stratagems, and strategies. According to Bates [1] footnote
chasing is defined as checking the cited references and related material of a
work backward in time. Citation searching refers to a forward chaining of works
63
BIR 2018 Workshop on Bibliometric-enhanced Information Retrieval
citing the seed document through a citation index. Keyword searching consists of
looking up the indexing terms representing research topics. The fourth stratagem
author searching is defined as looking for specific author names to investigate
more written material from a concrete author. Journal run is defined as the
subject access to issues and volumes of a specific journal. Bates describes this
stratagem as follows: "Journal run consists of reviewing contents pages of core
journals in an area. (...) [This] technique, by definition, guarantees complete
recall within that journal, and, if the journal is central enough to the searcher’s
interests, this technique also has tolerably good precision." [2]. In fact, a recent
study shows that the journal run stratagem is often utilized as exploratory search
functionality in DLs [3].
Scientific searchers are often over-flooded with a variety of papers and for each
submitted query they are confronted with a list of ranked relevant documents.
They browse documents or reformulate their queries until clicking on one or
more item(s) that interest(s) them. By clicking on this particular document and
if this document is published in a journal, the user can get more details about
the article such as the authors, the keywords describing covered topics and the
source. If the source is a journal and the user browses its content, then we talk
about "journal run" (see Figure 1 as an example). In general, there is a strong
demand and need for user support while navigating through scholarly document
collections (esp. for area scanning, journal run and author search; see the same
argument in [4]). This paper seeks to address the journal run stratagem and to
analyze its usage in an academic search engine in the social sciences, the sowiport
search engine. In this context, we want to understand better the user behavior
toward this specific stratagem performed by real DL users.
Fig. 1. Sample of an interface presenting a journal run
In particular, we address the following research questions:
RQ 1: Which usage patterns can be observed from clicked journal
papers?
We focus on analyzing the usage pattern of "journal run" in real retrieval
sessions in terms of frequency of their use and the stage of appearance. Fur-
thermore, we want to study the success of sessions which cover journal papers
64
BIR 2018 Workshop on Bibliometric-enhanced Information Retrieval
exploration behavior. We examine the interactions of the users in the sowiport
DL in order to measure the usefulness and the precision of sessions containing
journal runs. We determine the session success based on the presence of positive
actions proposed recently by Hienert and Mutschke [5]. To be more specific, we
measure the number of positive actions before and after journal run occurrence
in a retrieval session.
RQ 2: Do journal properties like impact measures and coreness
influence the click behavior in real life retrieval?
We aim at investigating whether the journal papers clicked by users are in-
fluenced by the journal reputation (in terms of impact measures) or the coreness
(in terms of journal publication frequency). Among the most used metrics to
identify the relevance and the reputation of a journal is the impact factor (IF)1 .
Coreness of journals is measured according to Bradford zones [7].
The remainder of this paper is organized as follows. In the next section, we
present an short overview of papers addressing stratagems and journal run in
particular. In Section 3 we analyze the user behavior towards journal run and
how using journal run affects the quality of the whole session search. Then, we
verify whether the journal properties impact metrics and coreness have an influ-
ence on the clicked papers. Finally, we summarize our findings and present some
perspectives relevant for future work.
2 Related Work
Bates [1] has specified different types of user behavior towards a search system,
among them we cite: moves, tactics, stratagems and strategies. A move refers to
a basic action performed by the user. A tactic uses additional moves along with
a search. As for stratagems, they indicate complex and multiple moves/tactics
having knowledge of a particular search domain. A strategy is a combination of
moves, tactics and stratagems as a plan to pursue during the search session.
Some approaches studied the user behavior towards tactics, moves or strata-
gems. For instance, Schneider and Borlund [8] studied the effectiveness of using
stratagems in constructing and maintaining thesauri vocabulary and structure.
Mahoui and Cunningham [9] specified the importance of understanding the infor-
mation of DL users in creating useful and stable search systems. They analyzed
transaction logs to study usage patterns of CiteSeer in terms of query and search
patterns. Xie [10] analyzed the users’ search behaviors and their relationships
with their information needs by specifying a hierarchical level of users’ goals.
Shute and Smith [11] identified 13 knowledge-based tactics arranged into three
categories: broaden topic scope, narrow topic scope and change topic scope.
We notice that many research works have been proposed in the literature that
analyzed the information-seeking process of a search engine usage. A transaction
1
The IF is the result of Garfield’s work with citation indexes and has been adopted by
the Institute for Scientific Information’s (ISI) as the ratio of the number of citations
of articles published in years Y-1 and Y-2 by indexed journals during the year Y [6].
65
BIR 2018 Workshop on Bibliometric-enhanced Information Retrieval
log is defined as a file of the transactions (communications) between a system
and its users. A specific type of transaction logs are the ones describing sessions’
search extracted from academic search engines.
For instance, Carevic and Mayr [12] proposed bibliometric-enhanced search
facilities such as "journal run" or "citation search" and their possible integration
in DLs. In their position paper, they argue that bibliometric-enhanced strata-
gems can facilitate domain specific search activities by applying bibliometric
measures for re-ranking and/or rearranging DL-entities like documents, jour-
nals or authors. They propose different types of stratagem implementations like
"extended journal run", "context-preserving journal run" and extended versions
of citation search. In [3] the authors presented a first approach on gathering a
deeper understanding on the usage of stratagems by conducting an online survey
with 128 respondents from twelve different fields of research. The survey showed
a general need for a contextual ranking in exploratory search which the authors
tested using a journal run scenario in which the respondents were asked to ar-
range the content of a journal run based on two contextual features and four
non-contextual features like date or title. The results of the survey showed that
the respondents assess the ranking features based on contextualisation noticeably
higher than the four non-contextual features2 .
Hemminger et al. [14] studied the information seeking behavior of academic
scientists by conducting a survey at the University of North Carolina at Chapel
Hill. They found that journal search has been considered the primary source of
information with 56.04% of searches at daily basis which indicates a higher usage
compared to book, proceedings and other types of resource. Ortega [15] studied
the relationship between usage metrics and bibliometric indicators for authors
using different academic systems such as Google Scholar and ResearchGate. Ge
examined the information-seeking behavior of scholars in the fields of social
sciences and humanities [16]. According to his study, he found that electronic
journals were rated as the third most important resource, and that users look
for multiple journals related to their keywords rather than just one.
3 Case Study
In this section, we first provide details about the dataset that we used for our
analysis and then we describe the analyses in order to answer the research ques-
tions raised in Section 1.
3.1 Dataset
In order to analyze the utilization of the stratagem journal run, we used the
digital library sowiport which was a publicly available DL dedicated to the Social
Sciences. It contained more than nine million records, full texts and research
2
A recent empirical study shows the effectiveness of simple contextual browsing fea-
tures in a real-live DL [13].
66
BIR 2018 Workshop on Bibliometric-enhanced Information Retrieval
projects included from twenty-two different databases whose content is in English
and German [17].
In Table 1, we give an overview of the dataset by presenting the number of
log entries, total sessions and a subset of sessions with stratagems (incl. Keyword
Search, Citation Search, Footnote Chasing and Journal Run) as well as those
with journal run only. The data we used was collected from September 2016 to
May 20173 . After a filtering process, we mapped the transaction activities of the
3,377,000 log entries to a list of 58 different user actions which cover all types
of activities and pages that can be carried out/visited within the system (e.g.
typing a query, visiting a document, selecting a facet, exporting a document,
etc.). For each action, a session id, the date stamp and additional information
(e.g. query terms, document ids, and result lists) are stored.4
Table 1. Statistics of the filtered dataset
Number
Log Entries 3,377,000
Total Sessions 208,557
Sessions with Stratagems 105,789
Sessions with Journal Run 22,721
Searching the sowiport DL can be performed through an All fields search
box (default search without specification), or through specifying one or more
field(s): title, person, institution, number, keyword or year. The main action
types are divided into two categories: "Query"-related (search keyword, search
person, advanced search, ...) and "Document"-related (see document details,
access source, mark as favorite, ...) actions. Another categorization of actions
was proposed in [5] by specifying search interactions and successive positive
actions that we will use later (see Section 4) to measure the quality of sessions
and the impact of the stratagem use.
3.2 Journals in this study
In this study, the journal run was performed in 22,721 individual sessions. In
total, the number of journals accessed is 2063 unique journals. For these 2063
journals, we extracted their sizes according to the number of papers indexed in
sowiport. Then, we considered two bibliometric journal indicators: the Impact
Factor from 2015 and the SCImago Journal Rank from 2016. The impact factor
(IF) is the measure used in the Journal Citation Reports5 . The SCImago Journal
3
The dataset can be downloaded at https://git.gesis.org/amur/SUSS-16-17
4
An overview of the user actions in the transaction log is given in [18].
5
https://clarivate.com/products/journal-citation-reports/
67
BIR 2018 Workshop on Bibliometric-enhanced Information Retrieval
Rank (SJR) is used by SCImago Journals6 and considers both the number of
citations received by a journal and the importance of the journals where these
citations came from by using a principle similar to PageRank [19]. The measure
SJR is among four measures used by Scopus which is the largest abstract and
citation database of peer-reviewed literature.
3.3 Measurements
To answer the first research question described in Section 1, we analyze the
sessions with the mentioned stratagem "Journal Run" using measures used in
our previous work [20].
For a session S during which a set of interactions {I} is performed by the
user, we define:
– Strat is the stratagem "Journal Run",
– P os is a positive interaction present in our data set among the following set
{P } described in [5]:
goto_fulltext, goto_google_scholar, goto_local_availability, goto_google_books,
view_description, export_cite, export_bib, export_mail, to_favorites, ex-
port_search_mail, save_search, save_search_history, save_to_multiple_favorites.
To answer the second research question, we measure the precision of a stratagem
before (P recision(Strat)b ) and after (P recision(Strat)a ) so we verify if it has
an influence on the conduct of a session. We verify if we can find more posi-
tive actions after using a stratagem comparing to the number of positive actions
before its utilization.
|P os ∈ {P }|
P recision(Strat)b = (1)
|I| b
|P os ∈ {P }|
P recision(Strat)a = (2)
|I| a
To have an overview of a stratagem benefit, we measure the Usefulness as
the percentage of successful sessions in terms of positive actions among all the
sessions including both of the studied stratagems. This measure is inspired from
the Global Usefulness measure proposed by [5]:
s+
Strat
U sef ulness(Strat) = (3)
|sStrat |
where s+ (Strat) indicates session success in terms of positive actions occurrence
after using a specific stratagem, and |sStrat | represents the number of sessions
using a stratagem (footnote chasing or citation search) no matter the type of
user’s interactions (positive or not).
6
http://www.scimagojr.com/
68
BIR 2018 Workshop on Bibliometric-enhanced Information Retrieval
4 Results
In this section, we present the analysis of journal run use on the session search
conduct and the impact of journals’ metrics and coreness on their usage.
4.1 Journal Run Usage
To answer the first research question described in Section 1, we analyzed the
sessions with journal run activity having at least one clicked document.
While searching sowiport, the user typically checks the document content and
metadata such as title, keywords, source. By clicking on the source (a journal in
this case), the user expresses his interest in the source and can then browse other
documents from the same journal. Regarding the Journal Run usage illustrated
in Figure 2, we found that it is the most used stratagem with 39.78% among
other stratagems namely Keyword Search, Author Search, Citation Search and
Footnote Chasing. Journal Run was performed in average 1.85 times per session.
We consider, at this level, the position of the journal run in a session. In
most of the sessions, the users accessed a journal after viewing a document.
Journal run is distributed at different positions in the session and was mostly
performed at the end of a session (see Figure 2). In fact, it was performed as
the last action in a session for 37.61% of the studied sessions. This result is
in line with the findings in [12] where the authors highlighted the need for an
extended approach regarding journal run. In fact, an improvement of the user
experience could be achieved by a journal-based re-ranking of results instead of
a document-based one.
Fig. 2. Position of Journal Run in the search sessions
In order to answer the first research question mentioned in Section 1, we
examined the usage behavior and found that the utilization of journal run is not
69
BIR 2018 Workshop on Bibliometric-enhanced Information Retrieval
similar compared to the other stratagems. Mostly users browse journal papers at
the end of sessions and rarely in the middle. For the other stratagems, Citation
Search and Footnote Chasing are mostly used in the middle and the end of
a search session (compare [20]), while Author Search and Keyword Search are
mostly used in the middle and the beginning.
In order to study the impact of journals on the search behavior based on the
academic search engine sowiport, we considered the measures shown in Section
3.3 using the equations 1, 2 and 3. The results of the measures are summarized in
Table 2. We notice that the users of journal run lead to successful sessions with
a percentage of 55.83%. In the rest of the sessions, the users do not perform
positive actions (e.g. exporting citation, adding to favorite, etc.), they rather
perform browsing actions by viewing the journals and many users access the
external link of these journal. In fact, the percentage of positive (27.42%), non-
positive (17.76%) and neutral (54.82%) impacts are measured by the number
of sessions having more positive actions after performing a journal run, those
having more positive ones before its usage and the sessions having equal number
of positive actions before and after its usage respectively.
Table 2. Evaluation of Journal Run usage effects on the session conduct
Measure Value
P recision(Strat)b 0.043
P recision(Strat)a 0.096
Gain in Precision7 5.32
U sef ulness(Strat) 55.83%
Positive Impact 27.42%
Neutral Impact 54.82%
Non-positive Impact 17.76%
4.2 Impact of Journals Metrics and Coreness
After analyzing the user behavior toward journal run, we here focus on the
second research question mentioned in Section 1. We observe the potential effect
of journal indicators (IF, SJR) and coreness of journals (according to their sizes
based on Bradford Zones [7]) on the users click behavior.
In Figure 3, we present the distribution of journals using the IF, Bradford
Zones and the frequency of their real usage in sowiport. In Figure 3.a we catego-
rize the journals according to their IF values considering high IF with IF ≥ 3,
medium IF for 1 ≤ IF < 3, low IF are the journals with IF < 1 and those
that have no IF. In Figure 3.b we group according to Bradford zones. We notice
that among the 2063 accessed journals in our dataset, based on these zones, 63
7
Gain is computed as the percentage of increase between the precision-after and the
precision-before.
70
BIR 2018 Workshop on Bibliometric-enhanced Information Retrieval
are core journals, 262 journals are in zone 2 and 1738 journals are zone 3 (the
periphery). Similarly, Figure 3.c shows three categories of journals according to
their frequency of usage (number of views) where 80% have a frequency lower
than 20 (f < 20), 10% have an intermediate frequency (20 ≤ f < 50) and 10%
have high frequency (f ≥ 50) with the highest frequency reaching 883.
Fig. 3. Categorization of the journals accessed in our dataset according to impact
factor (a), Bradford Zones (b) and usage frequency (c)
In Table 3 we describe the amount and the reputation of the journals in each
Bradford Zone in terms of presence of indicators (IF and SJR), and the mean
value of each indicator. We notice that 50% of the Core and Zone 2 journals
have indicators (IF and/ or SJR) while only 30.44% of Zone 3 journals have an
IF or SJR.
Table 3. Amount and reputation of journals for each Bradford zone
Zone Presence of IF/SJR Mean IF Mean SJR
Core journals 50.73% 1.635 1.102
Zone 2 journals 50% 1.392 0.717
Zone 3 journals 30.44% 0.949 0.604
On the one hand, we measure the Pearson correlation between IF and SJR
and we found a positive correlation with R = 0, 82. We conclude a strong positive
linear relationship between these two metrics. On the other hand, Figure 4 shows
the relation between the impact factor and the size of journals. We notice that
there is a very weak positive linear relationship (with R = 0, 22) as the journals
with high IF do not necessarily have a large size, and journals with a small
71
BIR 2018 Workshop on Bibliometric-enhanced Information Retrieval
size can have high or low IF as well. For instance, the "American Journal of
Psychiatry" with the highest IF of 5.68 in our journal set was clicked only once
in the filtered log.
Fig. 4. Relation between the impact factor and the size of journals
In Figure 5, we study the impact of size and the reputation of journals on
the frequency of their usage. In order to answer our second research question,
we considered only sessions with positive actions to measure the effect of jour-
nal indicators (IF) and coreness. We found that 27.42% of the journals lead to
successful sessions.
For the successful sessions8 , we controlled the size (Bradford Zones) of the
journals and found 7.66% core journals, 23.92% zone 2 journals and 68.42%
zone 3 journals. In Figure 5, we present the relation between the journals’ usage
frequency and their corresponding IF (5.a) and size (5.b). For a better visu-
alization, we eliminated the top two most clicked journals ("Kölner Zeitschrift
für Soziologie und Sozialpsychologie" and "Zeitschrift für Pädagogik") in Figure
5. From Figure 5.a, we measured the Pearson correlation (R) and found that
users are not influenced by the impact factor of the journals (there is no linear
relationship with R = −0.07). The users are rather interested in journals that
answer their information need and are relevant to queries they submitted. We
notice that 3.82% of the frequently used journals have a high IF, 22.24% have a
medium IF, 33.00% have a low IF and 40.93% are without IF. As for the size of
journals (Figure 5.b), we notice also a non-linear relationship between the size
and the frequency of usage of journals (R = 0.223).
8
Successful sessions are those having a higher number of positive actions (saving,
exporting, downloading) after clicking on a journal than before.
72
BIR 2018 Workshop on Bibliometric-enhanced Information Retrieval
Fig. 5. Relation between the usage frequency of journals and their size (a) and impact
factor (b)
In summary, we studied in this section the distribution of journals in terms
of size (Bradford zones), usage frequency and journal metrics. We measured the
impact of the journal run using the presence of positive actions (e.g. adding a
paper to favorites or exporting a citation) before and after using this stratagem.
We found that more than 50% of sessions contain positive actions and more
than 60% of clicked journals lead to a positive effect. This means: the number of
positive actions after using a journal run is higher than the number of positive
actions before its usage. We conducted this analysis in order to check which
journal metric has an impact on the sessions’ success, and we found that the
impact factor is not a criterion that affects the journal run whereas the size
of the journal has a slight influence. In fact, users tend to click on journals
that answer their information need and are relevant to the submitted queries
regardless of their reputation or indicator (such as IF). This can be also explained
by the fact that users, after finding a relevant journal (with high, medium or
low impact factor), they access its external link (leave the retrieval system) and
do not perform many interactions after the journal run. We also found that
similar indicators from the same source (such as IF and SJR) correlate better
than indicators from different sources (such as IF and size) as discussed before
by Ortega[15].
5 Conclusion
In this paper, we investigated the usage of a specific search stratagem – the
Journal Run – in an academic search engine. In fact, studying the user behavior
towards stratagems can enhance the user-system interactions and lead to more
useful academic search engines [12].
73
BIR 2018 Workshop on Bibliometric-enhanced Information Retrieval
First, we examined the frequency and stage of use of the journal run as well as
its impact on sessions. Then, we examined the impact of journal reputation and
size on the usage of journals. We found that journal run is more often performed
by the users at the end of the sessions, unlike other stratagems that are more
present in the middle of the session [20]. The journal run is in the 3rd position
in terms of sessions’ success metrics (Precision, Usefulness and Positive Impact)
after Citation Search and Footnote Chasing.
In addition, we analyzed the journals’ distribution using their different indi-
cators such as Impact Factor, coreness and usage frequency in the same digital
library. We also measured the correlation between these indicators to study the
influence and impact of the journals’ properties (such as the Impact Factor and
Bradford Zones) on the sessions’ success. We found that the frequency of usage
of the analyzed journals is not related to the Impact Factor within these sessions
and that the size of the journal has an insignificant correlation. Thus, the users
are more likely to click on journals that are relevant to their current information
need and their choice is not strongly influenced by the other journal properties.
Regarding future work, we need to investigate if similar patterns can be ob-
served in other domains, e.g. Natural Sciences, and if the users’ level of expertise
has an impact on their search behaviors (e.g. practitioners, students, researchers,
professors). In addition to log analysis, we need to do user studies in order to
compare user feedback with the findings of this study.
6 Acknowledgement
This work was funded by Deutsche Forschungsgemeinschaft (DFG), grant no.
MA 3964/5-1 (AMUR project). The AMUR project aims at improving the sup-
port of interactive retrieval sessions following two major goals: improving user
guidance and system tuning.
References
1. Bates, M.J.: Where should the person stop and the information search interface
start? Information Processing & Management 26(5) (October 1990) 575–591
2. Bates, M.J.: Speculations on browsing, directed searching, and linking in relation
to the bradford distribution. In: Emerging frameworks and methods: Proceedings
of the Fourth International Conference on Conceptions of Library and Information
Science (CoLIS 4). (2002) 137–150
3. Carevic, Z., Mayr, P.: Survey on high-level search activities based on the stratagem
level in digital libraries. In: Proceedings of TPDL 2016. (2016) 54–66
4. Brajnik, G., Mizzaro, S., Tasso, C., Venuti, F.: Strategic help in user interfaces
for information retrieval. Journal of the American Society for Information Science
and Technology 53(5) (2002) 343–358
5. Hienert, D., Mutschke, P.: A usefulness-based approach for measuring the local and
global effect of iir services. In: Proceedings of CHIIR 2016, ACM (2016) 153–162
6. Garfield, E.: The history and meaning of the journal impact factor. JAMA 295(1)
(2006) 90–93
74
BIR 2018 Workshop on Bibliometric-enhanced Information Retrieval
7. Mayr, P.: Relevance distributions across Bradford Zones: Can Bradfordizing im-
prove search? In Gorraiz, J., Schiebel, E., Gumpenberger, C., Hörlesberger, M.,
Moed, H., eds.: Proceedings of ISSI 2013, Vienna, Austria (2013) 1493–1505
8. Schneider, J.W., Borlund, P.: Introduction to bibliometrics for construction and
maintenance of thesauri: Methodical considerations. Journal of Documentation
60(5) (2004) 524–549
9. Mahoui, M., Cunningham, S.J.: Search behavior in a research-oriented digital
library. In: Proceedings of ECDL 2001, Springer (2001) 13–24
10. Xie, H.I.: Patterns between interactive intentions and information-seeking strate-
gies. Information Processing & Management 38(1) (2002) 55 – 77
11. Shute, S.J., Smith, P.J.: Knowledge-based search tactics. Information Processing
& Management 29(1) (1993) 29 – 45
12. Carevic, Z., Mayr, P.: Extending search facilities via bibliometric-enhanced strata-
gems. In: Proceedings of the BIR workshop at ECIR 2015, Vienna, Austria (2015)
40–46
13. Carevic, Z., Schüller, S., Mayr, P., Fuhr, N.: Contextualised Browsing in a Digital
Library’s Living Lab. In: Proceedings of JCDL 2018. (2018)
14. Hemminger, B.M., Lu, D., Vaughan, K., Adams, S.J.: Information seeking behavior
of academic scientists. JASIST 58(14) (2007) 2205–2225
15. Ortega, J.L.: Relationship between altmetric and bibliometric indicators across
academic social sites: The case of csic’s members. Journal of Informetrics 9(1)
(2015) 39 – 49
16. Ge, X.: Information-seeking behavior in the digital age: A multidisciplinary study
of academic researchers. College & Research Libraries 71(5) (2010) 435–455
17. Hienert, D., Sawitzki, F., Mayr, P.: Digital Library Research in Action – Supporting
Information Retrieval in Sowiport. D-Lib Magazine 21(3/4) (2015)
18. Mayr, P., Kacem, A.: A Complete Year of User Retrieval Sessions in a Social
Sciences Academic Search Engine. In: Proceedings of TPDL 2017. (2017)
19. Chen, P., Xie, H., Maslov, S., Redner, S.: Finding Scientific Gems with Google.
ArXiv Physics e-prints (April 2006)
20. Kacem, A., Mayr, P.: Analysis of Footnote Chasing and Citation Searching in an
Academic Search Engine. In: Proceedings of the 2nd BIRNDL workshop at SIGIR
2017, Tokyo, Japan (2017) 91–100
75