Much attention has been paid recently in analyzing the interaction between the user and the search system. Interactive information retrieval researchers have become more interested in studying the user behavior in academic search engines in order to improve ranking and to better understand the user expectations. In our previous work, we analyzed the user behavior towards Marcia Bates' search stratagems such as 'footnote chasing', 'citation search', 'keyword search' and 'author search'. In this paper, we propose to examine in more detail the user behavior towards journal paper search known as the 'journal run' stratagem. We study the frequency and the stage of 'journal run' use in a social sciences academic search engine log. We found that the journal run is mostly used at the end of retrieval sessions and has an approximate usefulness of 55% measured by the presence of positive user actions in the log. Moreover, we include typical journal metrics in our case study and test whether there is an influence of the journals' impact factor (IF) and coreness of a journal on the overall session conduct. We found that the metric IF and coreness of a journal do not have a measurable impact on the journals utilization of social sciences users.
Much attention has been paid recently in analyzing the interaction between the user and the search system. In fact, researchers have moved from considering only the current query and result set to focus more on the user’s past interactions and the analysis of whole retrieval sessions. Research approaches in interactive information retrieval aim to understand the user searching behavior in order to improve the ranking of results after submitting a query and enhance the user experience within an IR system.
Digital Libraries (DLs) users behave differently when interacting with the
system as underlined by Marcia Bates who highlighted different concepts such
as moves, tactics, stratagems, and strategies. According to Bates [
Scientific searchers are often over-flooded with a variety of papers and for each
submitted query they are confronted with a list of ranked relevant documents.
They browse documents or reformulate their queries until clicking on one or
more item(s) that interest(s) them. By clicking on this particular document and
if this document is published in a journal, the user can get more details about
the article such as the authors, the keywords describing covered topics and the
source. If the source is a journal and the user browses its content, then we talk
about "journal run" (see Figure 1 as an example). In general, there is a strong
demand and need for user support while navigating through scholarly document
collections (esp. for area scanning, journal run and author search; see the same
argument in [
In particular, we address the following research questions:
We focus on analyzing the usage pattern of "journal run" in real retrieval
sessions in terms of frequency of their use and the stage of appearance.
Furthermore, we want to study the success of sessions which cover journal papers
exploration behavior. We examine the interactions of the users in the sowiport
DL in order to measure the usefulness and the precision of sessions containing
journal runs. We determine the session success based on the presence of positive
actions proposed recently by Hienert and Mutschke [
We aim at investigating whether the journal papers clicked by users are
influenced by the journal reputation (in terms of impact measures) or the coreness
(in terms of journal publication frequency). Among the most used metrics to
identify the relevance and the reputation of a journal is the impact factor (IF)1.
Coreness of journals is measured according to Bradford zones [
Bates [
Some approaches studied the user behavior towards tactics, moves or
stratagems. For instance, Schneider and Borlund [
We notice that many research works have been proposed in the literature that
analyzed the information-seeking process of a search engine usage. A transaction
1 The IF is the result of Garfield’s work with citation indexes and has been adopted by
the Institute for Scientific Information’s (ISI) as the ratio of the number of citations
of articles published in years Y-1 and Y-2 by indexed journals during the year Y [
For instance, Carevic and Mayr [
Hemminger et al. [
In this section, we first provide details about the dataset that we used for our analysis and then we describe the analyses in order to answer the research questions raised in Section 1. 3.1
In order to analyze the utilization of the stratagem journal run, we used the
digital library sowiport which was a publicly available DL dedicated to the Social
Sciences. It contained more than nine million records, full texts and research
2 A recent empirical study shows the effectiveness of simple contextual browsing
features in a real-live DL [
In Table 1, we give an overview of the dataset by presenting the number of log entries, total sessions and a subset of sessions with stratagems (incl. Keyword Search, Citation Search, Footnote Chasing and Journal Run) as well as those with journal run only. The data we used was collected from September 2016 to May 20173. After a filtering process, we mapped the transaction activities of the 3,377,000 log entries to a list of 58 different user actions which cover all types of activities and pages that can be carried out/visited within the system (e.g. typing a query, visiting a document, selecting a facet, exporting a document, etc.). For each action, a session id, the date stamp and additional information (e.g. query terms, document ids, and result lists) are stored.4
Searching the sowiport DL can be performed through an All fields search
box (default search without specification), or through specifying one or more
field(s): title, person, institution, number, keyword or year. The main action
types are divided into two categories: "Query"-related (search keyword, search
person, advanced search, ...) and "Document"-related (see document details,
access source, mark as favorite, ...) actions. Another categorization of actions
was proposed in [
In this study, the journal run was performed in 22,721 individual sessions. In
total, the number of journals accessed is 2063 unique journals. For these 2063
journals, we extracted their sizes according to the number of papers indexed in
sowiport. Then, we considered two bibliometric journal indicators: the Impact
Factor from 2015 and the SCImago Journal Rank from 2016. The impact factor
(IF) is the measure used in the Journal Citation Reports5. The SCImago Journal
3 The dataset can be downloaded at https://git.gesis.org/amur/SUSS-16-17
4 An overview of the user actions in the transaction log is given in [
To answer the first research question described in Section 1, we analyze the
sessions with the mentioned stratagem "Journal Run" using measures used in
our previous work [
For a session S during which a set of interactions fIg is performed by the
user, we define:
– Strat is the stratagem "Journal Run",
– P os is a positive interaction present in our data set among the following set
fP g described in [
To answer the second research question, we measure the precision of a stratagem before (P recision(Strat)b) and after (P recision(Strat)a) so we verify if it has an influence on the conduct of a session. We verify if we can find more positive actions after using a stratagem comparing to the number of positive actions before its utilization.
P recision(Strat)b = jP os 2 fP gj jIj
b P recision(Strat)a = jP os 2 fP gj (2) jIj a
To have an overview of a stratagem benefit, we measure the Usefulness as
the percentage of successful sessions in terms of positive actions among all the
sessions including both of the studied stratagems. This measure is inspired from
the Global Usefulness measure proposed by [
U sef ulness(Strat) = + sStrat jsStratj where s+(Strat) indicates session success in terms of positive actions occurrence after using a specific stratagem, and jsStratj represents the number of sessions using a stratagem (footnote chasing or citation search) no matter the type of user’s interactions (positive or not). 6 http://www.scimagojr.com/ (1) (3) 4
In this section, we present the analysis of journal run use on the session search conduct and the impact of journals’ metrics and coreness on their usage. 4.1
To answer the first research question described in Section 1, we analyzed the sessions with journal run activity having at least one clicked document.
While searching sowiport, the user typically checks the document content and metadata such as title, keywords, source. By clicking on the source (a journal in this case), the user expresses his interest in the source and can then browse other documents from the same journal. Regarding the Journal Run usage illustrated in Figure 2, we found that it is the most used stratagem with 39.78% among other stratagems namely Keyword Search, Author Search, Citation Search and Footnote Chasing. Journal Run was performed in average 1.85 times per session.
We consider, at this level, the position of the journal run in a session. In
most of the sessions, the users accessed a journal after viewing a document.
Journal run is distributed at different positions in the session and was mostly
performed at the end of a session (see Figure 2). In fact, it was performed as
the last action in a session for 37.61% of the studied sessions. This result is
in line with the findings in [
In order to answer the first research question mentioned in Section 1, we
examined the usage behavior and found that the utilization of journal run is not
similar compared to the other stratagems. Mostly users browse journal papers at
the end of sessions and rarely in the middle. For the other stratagems, Citation
Search and Footnote Chasing are mostly used in the middle and the end of
a search session (compare [
In order to study the impact of journals on the search behavior based on the
academic search engine sowiport, we considered the measures shown in Section
3.3 using the equations 1, 2 and 3. The results of the measures are summarized in
Table 2. We notice that the users of journal run lead to successful sessions with
a percentage of 55.83%. In the rest of the sessions, the users do not perform
positive actions (e.g. exporting citation, adding to favorite, etc.), they rather
perform browsing actions by viewing the journals and many users access the
external link of these journal. In fact, the percentage of positive (27.42%),
nonpositive (17.76%) and neutral (54.82%) impacts are measured by the number
of sessions having more positive actions after performing a journal run, those
having more positive ones before its usage and the sessions having equal number
of positive actions before and after its usage respectively.
After analyzing the user behavior toward journal run, we here focus on the
second research question mentioned in Section 1. We observe the potential effect
of journal indicators (IF, SJR) and coreness of journals (according to their sizes
based on Bradford Zones [
In Figure 3, we present the distribution of journals using the IF, Bradford Zones and the frequency of their real usage in sowiport. In Figure 3.a we categorize the journals according to their IF values considering high IF with IF 3, medium IF for 1 IF < 3, low IF are the journals with IF < 1 and those that have no IF. In Figure 3.b we group according to Bradford zones. We notice that among the 2063 accessed journals in our dataset, based on these zones, 63 7 Gain is computed as the percentage of increase between the precision-after and the precision-before. are core journals, 262 journals are in zone 2 and 1738 journals are zone 3 (the periphery). Similarly, Figure 3.c shows three categories of journals according to their frequency of usage (number of views) where 80% have a frequency lower than 20 (f < 20), 10% have an intermediate frequency (20 f < 50) and 10% have high frequency (f 50) with the highest frequency reaching 883.
In Table 3 we describe the amount and the reputation of the journals in each Bradford Zone in terms of presence of indicators (IF and SJR), and the mean value of each indicator. We notice that 50% of the Core and Zone 2 journals have indicators (IF and/ or SJR) while only 30.44% of Zone 3 journals have an IF or SJR.
On the one hand, we measure the Pearson correlation between IF and SJR and we found a positive correlation with R = 0; 82. We conclude a strong positive linear relationship between these two metrics. On the other hand, Figure 4 shows the relation between the impact factor and the size of journals. We notice that there is a very weak positive linear relationship (with R = 0; 22) as the journals with high IF do not necessarily have a large size, and journals with a small size can have high or low IF as well. For instance, the "American Journal of Psychiatry" with the highest IF of 5.68 in our journal set was clicked only once in the filtered log.
In Figure 5, we study the impact of size and the reputation of journals on the frequency of their usage. In order to answer our second research question, we considered only sessions with positive actions to measure the effect of journal indicators (IF) and coreness. We found that 27.42% of the journals lead to successful sessions.
For the successful sessions8, we controlled the size (Bradford Zones) of the
journals and found 7.66% core journals, 23.92% zone 2 journals and 68.42%
zone 3 journals. In Figure 5, we present the relation between the journals’ usage
frequency and their corresponding IF (5.a) and size (5.b). For a better
visualization, we eliminated the top two most clicked journals ("Kölner Zeitschrift
für Soziologie und Sozialpsychologie" and "Zeitschrift für Pädagogik") in Figure
5. From Figure 5.a, we measured the Pearson correlation (R) and found that
users are not influenced by the impact factor of the journals (there is no linear
relationship with R = 0:07). The users are rather interested in journals that
answer their information need and are relevant to queries they submitted. We
notice that 3.82% of the frequently used journals have a high IF, 22.24% have a
medium IF, 33.00% have a low IF and 40.93% are without IF. As for the size of
journals (Figure 5.b), we notice also a non-linear relationship between the size
and the frequency of usage of journals (R = 0:223).
8 Successful sessions are those having a higher number of positive actions (saving,
exporting, downloading) after clicking on a journal than before.
In this paper, we investigated the usage of a specific search stratagem – the
Journal Run – in an academic search engine. In fact, studying the user behavior
towards stratagems can enhance the user-system interactions and lead to more
useful academic search engines [
First, we examined the frequency and stage of use of the journal run as well as
its impact on sessions. Then, we examined the impact of journal reputation and
size on the usage of journals. We found that journal run is more often performed
by the users at the end of the sessions, unlike other stratagems that are more
present in the middle of the session [
In addition, we analyzed the journals’ distribution using their different indicators such as Impact Factor, coreness and usage frequency in the same digital library. We also measured the correlation between these indicators to study the influence and impact of the journals’ properties (such as the Impact Factor and Bradford Zones) on the sessions’ success. We found that the frequency of usage of the analyzed journals is not related to the Impact Factor within these sessions and that the size of the journal has an insignificant correlation. Thus, the users are more likely to click on journals that are relevant to their current information need and their choice is not strongly influenced by the other journal properties.
Regarding future work, we need to investigate if similar patterns can be observed in other domains, e.g. Natural Sciences, and if the users’ level of expertise has an impact on their search behaviors (e.g. practitioners, students, researchers, professors). In addition to log analysis, we need to do user studies in order to compare user feedback with the findings of this study. 6
This work was funded by Deutsche Forschungsgemeinschaft (DFG), grant no. MA 3964/5-1 (AMUR project). The AMUR project aims at improving the support of interactive retrieval sessions following two major goals: improving user guidance and system tuning.