Relevance assessment in retrieval test collections and citations/mentions of scienti c documents are two di erent forms of relevance decisions: direct and indirect. To investigate these relations, we combine arXiv data with Web of Science and Altmetrics data. In this new collection, we assess the e ect of relevance ratings on measured perception in the form of citations or mentions, likes, tweets, et cetera. The impact of our work is that we could show a relation between direct relevance assessments and indirect relevance signals.
One of the long-running open questions in Information Science in general and
especially in Information Retrieval (IR) is on what constitutes relevance and
relevance decisions. In this paper, we would like to borrow from the idea of
using IR test collections and their relevance assessments to intersect these
explicit relevance decisions with some implicit or hidden relevance decisions in the
form of citations. We see this in the light of Borlund's discussion of relevance
and its multidimensionality [
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). BIR 2020, 14 April 2020, Lisbon, Portugal.
A test collection that incorporates both direct and indirect relevance
decisions is the iSearch collection introduced by Lykke et al. [
Carevic and Schaer [
To improve on the known limitations of having a small overlap of citations and relevance judgments in iSearch, we expand the iSearch document collection and its internal citation data. We complement iSearch with external citation data from the Web of Science. Additionally, we add di erent Altmetric scores as they might introduce some other promising insights on relevance indicators. These di erent data sources will be used to generate a dataset to investigate whether there is a correlation between intellectually generated direct relevance decisions and indirect relevance decisions incorporated through citations or mentions in Altmetrics.
Our expanded iSearch collection allows us to compare and analyze direct and indirect relevance assessments. The following research questions are to be addressed with the help of this collection and a rst data evaluation: RQ1 Are arXiv documents with relevance ratings published in journals with a higher impact? RQ2 Are arXiv documents with a relevance rating cited more highly or do they receive more mentions in Altmetrics? RQ3 In the literature, a connection between Mendeley readerships and citations is described. Is there evidence of a link between Mendeley readerships and citations in the documents with relevance ratings?
The paper is structured as follows: In Section 2, we describe the related work. Section 3 is about the data set generation and on the intersections between arXiv, Web of Science, and the Altmetrics Explorer. In Section 4, we use this new combined data set to answer the previous research questions. We discuss our rst empirical results in Section 5 and draw some rst conclusions. 2
Borlund [
White [
To the best of our knowledge, Altmetrics has not been a mainstream topic within the BIR workshop series. Based on literature analysis of the Bibliometricenhanced-IR Bibliography4 only two papers explicitly used Altmetrics-related measures to design a study, Bessagnet in 2014 and Jack et al. in 2018. The reason for this low coverage of Altmetrics-related papers in BIR is unclear, as the inherent advantages in comparison to classic bibliometric indicators are apparent. One of the reasons Altmetrics has been approached is the time lag caused by the peer review and publication process of journal publications: It takes two years or more until citation data is available for a publication and thus, something can be said about its perception. The advantage of Altmetrics can, therefore, be a faster availability of data in contrast to bibliometrics.
On the other hand, there is no uniform de nition of Altmetrics, and therefore
no consensus on what exactly is measured by Altmetrics. A semantic analysis of
contributions in social media is lacking for the most part, which is a major
issue making the evaluation of Altmetrics counts so di cult. Mentions are mostly
counted based on identi ers such as the DOI. However, it is not possible to mass
evaluate which mentions should be deemed as positive and which should be
deemed as negative, which means that a \performance paradox" develops. This
problem exists in a similar form in classical bibliometrics and must be considered
as an inherent problem of the use of quantitative metrics [
documents are rated twice or more, we ltered out duplicates before investigating the DOI coverage of the documents with di erent relevance levels. As it can be seen, the percentage of DOI coverage increases with higher relevance for documents being rated as relevant. Besides, the percentage DOI coverage of marginally (71; 0 %) and highly (71; 8 %) relevant documents is slightly higher compared to that of documents that are rated non-relevant (70; 0 %).
In sum, there are 1228 (out of 8670) documents with two or more ratings across di erent topics. In the following, we consider the relevance rating on a binary scale by treating ratings of 2 and 0 as non-relevant and ratings of 1, 2 and 3 as relevant. 136 (11; 1 %) documents are exclusively rated as relevant, 698 (56; 8 %) documents are exclusively rated as non-relevant, and the remaining 394 (32; 1 %) documents are rated both relevant and non-relevant across di erent topics. Therefore we only have a small intersection of contradicting relevance assessments for the same document which is under 5 % of the total count of judged documents (394 out of 8670). Category Nuclear & Particles Physics Fluids & Plasmas General Physics Astronomy & Astrophysics Applied Physics
In the next step, we combine the arXiv data with WoS and the Altmetrics Explorer6 to obtain statements on the impact of these publications both in the scienti c world and beyond in social media. The matching between arXiv and WoS is carried out via DOI. For 4061 out of 10; 263 ratings, WoS data is available. Of the documents with DOI that were matched with the WoS, the publications in arXiv can essentially be assigned to four major categories, as shown in Table 1. The distribution of relevance ratings by category shows a slightly di erent picture, which indicates a small shift and shows that the category with the largest number of articles is not automatically the category with the most relevance assessments.
RQ1 focuses on the relationship between positive relevance assessments and the perception of documents. Or in other words: Whether a publication with a positive relevance rating can achieve a higher scienti c perception or a higher perception in Altmetrics. The question is, therefore, whether there is a connection between high relevance rating and high perception.
If we look at Table 2, we see that the unrated documents, which form the vast majority, have a lower citation rate than the relevant-rated documents, on average, about 41 citations per document. The documents with relevance assessments reach higher citation rates than the group of documents without relevance assessment. The highest citation rate is achieved by the documents that have been grouped into the highest relevance level 3: With a citation rate of 76.2, they achieve a citation rate almost twice as high as documents without a relevance rating. The group of documents with relevance rating is small compared 6 Altmetrics is also topic of the project UseAltMe (funding code 16lFl107): On the way from article-level to aggregated indicators: understanding the mode of action and mechanisms of Altmetrics https://www.th-koeln.de/en/ information-science-and-communication-studies/usealtme_68578.php Relevance Non-relevant Marginal (1) Fair (2) High (3) Not rated to the group without relevance ratings. Nevertheless, from the authors' point of view, the group size is su cient for them to be able to read rough trends from it using bibliometric methods and to nd answers to the research questions. It is, therefore, not so much the small deviations that are important here, but rather the more speci c trends.
Table 1 shows the citation rates for the ve subject categories, which together account for 90 % of the arXiv documents, for the documents with and without relevance assessment. It can be seen that the citation rates for publications with a relevance assessment are signi cantly higher for all categories shown than for publications without a relevance rating in the same category (Pearson = 0.997). The presentation of the citation rates in Table 2 for the individual groups of documents with and without a relevance rating goes in the same direction: One of the trends also lies in the observation that all publications with a relevance rating have a higher citation count than the group whose documents were not selected as relevant.
If there is a correlation, it can be demonstrated at other points: When looking at the results, it is noticeable that the citation rate changes depending on the degree of relevance assessment. In the highest level of relevance assessment, level 3, the citation rate of 76.2 is almost twice as high as for the non-assessed documents. The citation rate increases continuously from the rst assessment level 0 to level 3. Level 2 is an exception, where the citation rate is lower than in level 1, but still higher than the citation rate of the non-evaluated documents.
Are there di erences in the composition of the groups that would explain the di erences in citation rates described above? The Journal Impact Factor (JIF) does not show a di erence for any of the groups. It only di ers by fractions of a decimal point. The documents without relevance rating have an average JIF of 4.5, the documents with relevance rating have an average JIF between 4.4 and 4.7. This shows that there is no signi cant di erence in the composition of the individual groups as to whether they publish more in high- or low-impact journals. The structure of all groups is the same in terms of average journal impact. Thus, RQ1 can be answered to the e ect that the impact of a journal has no in uence on the decision about the relevance of a document. 4.2
If we look at RQ2, we can say that arXiv documents with relevance ratings achieve a higher perception in terms of citation rate than unrated documents. This observation cannot be directly transferred to Altmetrics, where there is no di erence in the average number of tweets or news articles. The only measurable di erence refers to the documents with a relevance rating of 3, here an above-average number of tweets or mentions in patents can be observed. This e ect is the result of a skewed distribution and is presumably favored by two publications: A publication for which 253 tweets exist and another publication for which 50 mentions in patents are recorded. Without these two publications, there are no signi cant di erences between documents with and without relevance rating. Thus, RQ2 can be answered to the e ect that documents with a relevance rating receive a higher number of citations on average, but with the exception of Mendeley, there is no e ect on Altmetrics.
An explanation of why we do not see an e ect for Altmetrics in the arXiv
data may be in the year of publication: The majority of the publications were
published between 2003 and 2009. During this time, social media were already
being used actively in society, but not yet in science. This changed only slowly
towards 2008, with the Altmetric Explorer, for example, being founded by Euan
Adie in 2011. It is not known to what extent publications prior to the founding
year of Altmetric Explorer were retroactively re-indexed and to what extent
this is technically possible at all. This is because, in contrast to scienti c journal
publications, communication in social media is fast and can also be deleted before
it has been indexed.These e ects have to be taken into account when dealing
with altmetrics, as well as the fact that the publications originate from several
publication years, so some had more time to generate attention than others. If
it is necessary, \that older articles are compensated for lower altmetric scores
due to the lower social web use when they were published" [
In the literature, the question of whether there is a correlation between
citations of scienti c publications in Web of Science, Scopus, or Google Scholar
to Mendeley readerships (RQ3) has often been examined. Li & Thelwall [
News
Google Scholar. A correlation between citations from the three mentioned
citation databases to Mendeley was also measurable by Li & Thelwall, but it is much
worse than the correlation between the citation databases. This is not surprising
since Mendeley readerships are not scienti c citations. Bookmarking of
publications takes place for other reasons than citing a scienti c paper. But also, Costas
et al. [
The present study is based on about 435,000 arXiv publications. However, it was not possible to determine citations or Mendeley readerships counts for all of these documents: For 32,081 documents out of the total number, both citation data and data from Altmetric Explorer are available. This quantity contains 1037 documents with a relevance rating. This is where RQ3 comes into account: Is there evidence of a link between Mendeley readerships and citations in the arXiv-documents with relevance ratings? A correlation coe cient, according to Pearson of 0.83 was determined for the total quantity of 32,081 documents, which indicates an existing correlation, even if the value is not perfect. Instead, the result indicates that Mendeley and Web of Science are not entirely overlapping in terms of the perception of the documents.
The result is not signi cantly di erent if one takes the 1037 documents with relevance rating out of this set; Pearson is about at the same level at 0.8 (see Table 4). It should be noted that reasons for bookmarking a publication can be di erent from reasons for a citation: There are publications that are roughly equal in both data sources, for example, 10.1103/PhysRevE.67.026126, which receives 1068 citations and 1192 Mendeley reads. There are also examples of unequal perception: 10.1088/0067-0049/182/2/543 has 3151 citations but is bookmarked \only" 405 times on Mendeley or 10.1088/0954-3899/33/1/001 has 3903 citations but is bookmarked only 24 times, or in the opposite direction 10.1142/S0218127410026721 is cited only eight times but is bookmarked 106 times on Mendeley. So outliers can occur in both directions, which can lead to distortions in the measured correlation.
It becomes interesting if we take individual groups out of the entire group of 1037 documents with relevance assessments: Both for the group of 786 publications, which were singled out as possibly relevant but then rated 0 as nonrelevant. For the group of 35 publications, which were rated 3 and thus placed Relevance in the group of the most relevant documents, we get an even better correlation: For the 786 publications rated 0 we get a Pearson correlation of 0.85 and for the 35 publications rated 3 we get a correlation of 0.89. For the groups rated 1 (Pearson = 0.72) or 2 (Pearson = 0.76), we get a worse correlation value in each case because, in these two groups, the number of outliers is higher. From our point of view, the result is to be understood in such a way that a clear relevance decision lters out the papers that receive roughly the same perception on both sides, citation and Mendeley. Decisions in the middle of the relevance scale, on the other hand, lter out papers where the perception may tend to be more to one side. Thus, two things can be observed concerning RQ3: there is a link between citation data and Mendeley readerships. This becomes all the more visible if the paper is part of a set of documents that have previously been subjected to a corresponding relevance assessment. 5
The results of our small study on the intersection of relevance assessments within
the iSearch collection and corresponding citation counts show that direct
relevance decisions of a single assessor and indirect decisions of many external
authors citing this work are related. What sounds intuitive and like common
sense is not fully backed by the literature as the connection between citations
and relevance is not undisputed. Ingwersen [
Another thing we have to notice about our work is that the di erentiation of
direct and indirect relevance decisions is not an established concept in
information theory. While relevance is often described as multidimensional, layered, et
cetera, the terms direct and indirect are suggestions of the authors of this paper.
A concept that is aligned to the di erent levels and forms of relevance might
be the principle of poly representation [
When looking at JIF, Kacem and Mayr [
The impact of our work is that we could show a relation between direct relevance assessments and indirect relevance signals originating from bibliometric measures like citations. This relation is visible but not fully explainable. There seems to be something inherent in relevant documents that let them gather a higher number of citations. We are sure that it is not the impact of the corresponding journal. Otherwise, there would be no uncited documents within Nature. Popularity alone seems not to explain this e ect. Maybe citations in relation to relevance assessments are a marker for \quality", although we are aware that this term is highly controversial in the bibliometrics community.
It remains a future work to evaluate and investigate the phenomena that are the reason for the relationship we have seen. The principle of poly representation might be an excellent framework to bring together these di erent factors originating from relevance theory, bibliometrics, and Altmetrics. Additionally, it might help to design a retrieval study to follow these open questions further.