In the context of academic publishing and peer review, this study investigates the relationship between postpublication expert evaluations, their agreement levels, and the subsequent scientific and public recognition of the reviewed research. Using expert statements from the Science Media Center Germany as a dataset, we analyze Research in Context reviews to examine the alignment between qualitative post-publication assessments and bibliometric as well as altmetric indicators. We employ a Large Language Model to translate unstructured expert reviews into a structured rating scheme. Furthermore, we correlate these evaluations with citation counts from the Web of Science and alternative impact metrics such as the Altmetric Attention Score, news mentions, and Mendeley readership statistics from the Altmetric Explorer. We investigate the alignment of positive or critical post-publication reviews and high or low citation or altmetric counts.
Academic peer review is a fundamental element of the scientific publication and credibility system [
Peer reviewers are expected to examine the rationale behind research questions and evaluate the
originality, strengths, or weaknesses. Reviewers have the role of a gatekeeper, as only publications
that meet a discipline’s (unoficial and sometimes contradicting and arbitrary) quality standard are
published. Reviewers evaluate what is relevant enough to be published. Relevance is a central aspect of
peer reviewing in this case, as topicality, research field-specific requirements, and research standards
are among the most influential factors to be considered. We argue that the decisions that reviewers [
However, researchers are not the only target group for scientific information. The general public and
science journalists, who act as intermediaries between science and the public, are important actors in
scientific communication. In this role, science journalists must scan the vast scientific output to find
groundbreaking studies that are potentially relevant to the public, a challenge Herbert Simon described
as “a wealth of information creates a poverty of attention” [
While RIC and the other SMC services aim to provide early and recent access to science, the scientific system relies on credibility and recognition systems that mostly use citations but also mentions in social media and the like, known as Altmetrics. Previous research has shown how these two forms of scientific attributions go hand in hand, but very little research was conducted that connects the outcome and content of peer reviews with the later scientific and public attribution, measured by citations and altmetrics, respectively.
With the help of an exclusive data set of RIC post-publication reviews on scientific studies, we would like to work along the following research questions: RQ1 How can a Large Language Model transfer unstructured qualitative post-publication reviews of experts into a quantitative rating scheme? How good is the inter-annotator agreement for diferent configurations of this transfer process? RQ2 Is there an alignment of positive or critical post-publication reviews and high or low citation rates? Do these alignments also correspond with altmetrics?
The SMC Germany is a non-profit organization funded by the Klaus Tschira Trust. Its mission is to support journalists by providing access to exclusive pre-publication versions of scientific studies, so-called fact sheets on complex topics curated by specialists, or specific services around recent research and peer reviewing. Academic publishers like Springer, Nature, etc., have established a system that provides journalists with information about upcoming studies in advance under a press embargo (between 2-5 days). The studies are not yet publicly available, but journalists can get early access after registration and verification. In addition to access to the studies themselves, the SMC curates the “Research in Context” (RIC) service (now called “Statements”1, where experts in the field help to provide insights into cutting-edge research results and to provide journalists with orientation before reports on recent studies are written and published. Experts have first-hand access to the full texts of the studies. They must give an informed review to highlight strengths and weaknesses and provide an overall evaluation of the underlying research. This allows journalists to judge the significance of a new scientific finding by relying on the perspective of independent researchers more quickly and before the embargo period expires. After the embargo period, the RIC reviews will be published to the general public on the SMC website, where the reviewers’ names are revealed.
RIC reviews are a special case of peer reviews on scientific literature. While they are independent of classic pre-publication peer reviews, their reviewers don’t have to weigh the pros and cons to come up with a clear suggestion for or against publication; their reviews are publicly visible and are not blinded. So, RIR reviewers are both more free to judge and put their finger in the wound, but they might also be more restricted due to the public nature of their reviews. However, we see these RIC reviews as an interesting research data set as we gain access to recent, real-world reviews on scientific literature without any domain-specific filtering. The articles cover various topics, such as climate research, energy, digital sciences, and medicine. The disciplines and topics addressed by the Science Media Center (SMC) are primarily those with direct social relevance and of public interest, such as medical topics and diseases like cancer, as well as environmental issues like climate change. 1https://www.sciencemediacenter.de/angebote?story_type=Statements (last accessed: 26 March 2025) Scientific
Journal
Publication
Post Publication Reviews organized by the SMC
Extracted DOI to original paper
Matched with WoS citation data
Matched with Altmetric Explorer data Extracted statements from domain experts
Map statements into a score using gpt4o-mini
Match into one data set for this study via DOI
We crawled the website of SMC Germany in May 2024 and extracted a total of 521 RIC articles. These articles include diferent reviews (called statements) by domain experts on a specific primary article. We extracted the DOI or the reference to the primary article. For all 521 RIC articles, we extracted a total number of 1943 statements (≈ 3.72 review statements per article). We ignored all editorial content around the reviewers’ statements, such as highlighted quotes or subtitles, focusing only on the review statements themselves (see Figure 1).
The extracted review statements are unstructured text without any score-based rating. To map the text-based reviews into a comparable score, we utilized OpenAI’s gpt4o-mini2. The prompt instructed the model to generate a score between 0 and 1 for each review along the following criteria: K0 Examination of the research question (e.g. are the aims and rationale clearly formulated?) K1 Evaluation of originality (contribution, increase in knowledge in the literature or in the subject)
K3 Specific comments on the writing of the manuscript (e.g. spelling, organisation, illustrations, etc.)
K5 Comments on the statistics where appropriate (e.g. whether they are robust and fit for purpose and whether the controls and sampling mechanisms are suficiently and well described) Values close to 0 mean that the RIC reviewers expressed weaknesses or problems in the respective evaluation criteria. Values close to 1 mean that the evaluation criteria have been fulfilled to the reviewers’ satisfaction. If there is insuficient data for the evaluation criteria or a clear decision, the LLM was advised to give a value of NA. The evaluation criteria K0 to K5 were extracted from the oficial Elsevier guide on how to conduct a review3.
Next, we computed the inter-rater agreement between the scores for the diferent reviewers per
paper to measure the agreement level and identify controversial or non-controversial articles. We
utilized Krippendorf’s Alpha to measure the agreement as an alternative to the widely used Fleiss’
Kappa values. The advantages of Alpha over Kappa statistics are that while for Kappa, all assessors
have to rate the same number of subjects and use the same scale, the Alpha coeficient can usually
handle more variations and computes reliabilities that are comparable across any numbers of assessors
and values, diferent metrics, and unequal sample sizes [
With the Web of Science as developed by Eugene Garfield in the 1970s, it was possible to search for literature and see how often other scientists have cited an individual publication. Although the reasons why scientists cite each other are varied, the number of citations that a publication has received indicates how relevant other scientists consider this publication.
Altmetrics complements traditional bibliometrics with citation statistics from social media and other
online media. Thus, altmetrics can be compared to the introduction of the Science Citation Index, which
enabled scientists to track where they have been cited for the first time. The only diference is that
these “citations” are called news items, blog posts, likes, reads, shares, or Mendeley readerships: this
type of altmetrics indicates how many users have saved a specific publication to their personal libraries,
reflecting interest and potential future use. It “calculates impact indicators for authors based on the
number of users who stored their articles in the reference management system Mendeley” [
All kinds of altmetrics make scientific impact visible more quickly than traditional bibliometrics because they evolve more quickly and dynamically.
Together, altmetrics and bibliometrics form the basis for a multidimensional view of scientific impact from diferent angles: Bibliometrics is the part of classical scientific communication that takes much more time to make changes visible, because every citation is a publication that has to go through the complete cycle of a paper, especially a time-consuming peer review process.
The encoding of unstructured reviews onto values between 0 and 1 for six diferent evaluation criteria was conducted using OpenAI’s gpt4o-mini. We ran the experiment three times to see the range of values and to evaluate the inter-rater agreement between the diferent LLM runs. In the set of 521 crawled review articles, the oldest dates back to April 2016, and the newest is from March 2024.
In a pretest based on gpt4o with only the first 50 RIC articles, we ran the annotation process three times in a row to compare the variance produced by the LLM. The inter-rater agreement for the three runs was 0.24, 0.2, and 0.28.
In Table 1 (left), we see the outcome of the diferent Alpha values on 0.2 steps for the results of all 521 RIC articles with gpt4o-mini. 114 articles had contradicting reviewing scores indicated by a negative Alpha value. In contrast, there were 49 articles, and all reviews were perfectly aligned. For 48 articles, we could not compute an inter-rater agreement due to missing data or parsing errors during the web crawls. The agreement averages an Alpha value of 0.27, a comparable value to the pretest that was based on the larger and more expensive gpt4o model4.
In Table 1 (right), we see the results of the LLM encoding on the six criteria K0 to K5. From a total of 1943 possible single reviews, the most missing criterion is K5, with 1390. So, in 71% of the reviews, the LLM could not find any mention of the underlying statistics. While this might be uncommon for a typical pre-publication peer review, the special setting of RIC seems to encourage reviewers to leave out these comments. Maybe because the papers under (post-)review are already accepted, and therefore, comments on these fundamental issues are already sorted out.
We can note two outcomes on RQ1: First, while there are diferences in the LLM annotation outcome, it doesn’t matter much with respect to the achieved inter-rater agreement in the end. Second, the cost diferences between gpt4o-mini and gpt4o don’t shine through on a higher inter-rater agreement. For this kind of task, cheaper and smaller models seem good enough. Nevertheless, we have to note that the general agreement between the virtual raters is not very high. Given the overall controversial nature of the underlying data, that might not be surprising, and we have to consider that we applied a rating
scheme that was not mandatory for the original domain experts when writing their RIC statements.
However, in a large meta-study [
0.8 - 1, is higher than the average value, but even in the lowest class 0 - 0.19 the citation rate is not diferent from the average. Therefore, from our point of view, the picture here is indiferent.
Altmetrics are diferent, as we see in Table 3: While bibliometrics determine the perception of science
within the scientific community, altmetrics focus on the attention paid to science in online media. In
practice, this means that to receive a citation in bibliometrics, a completely new scientific paper that
has successfully completed the long peer review process and is also published in a scientific journal is
required. This process is very time-consuming [
If we look at the results, we can clearly see that both the Altmetrics Attention Score per paper and News Mentions per paper increase the higher the averaged K0 to K5 values get. In other words, the clearer and more positive the RIC reviews by the peers, the stronger the measured perception per paper. This is particularly evident in the two highest classes, but the overall trend across all classes is also very clear here. This confirms both the selection of papers and topics by the SMC and the assessments of the peers surveyed. It is not really surprising that this efect is measurable in the online media in particular, as this is precisely the field of application of the assessments for which the SMC’s assessments are obtained. The described efect does not apply to Mendeley Readerships (bookmarks on a scientific paper). This is because this parameter, which is one of the altmetrics, is more firmly anchored in science in terms of content and also shows a corresponding reaction here.
In summary, based on this small case study, we can say that there is a correlation between the evaluation of a paper by peers, less if you look at bibliometric parameters and more if you look at altmetrics. We argue that the results of this case study might help to understand what diferent criteria might be underlying when deciding on the relevance of scientific articles. A clear limitation of our approach is that the sample of Research in Context reviews is only a proxy for these decisions and that we needed to rely on LLM encodings to analyze the unstructured reviews. Although we can see reasonable LLM-based annotations when checking random results, we could not re-assess the whole document set. Nevertheless, our proposed pipeline can describe what articles are controversial, where reviewers agree or disagree, and how these measured criteria relate to a later reception in academia and social media: Two diferent user groups that obviously give diferent kinds of recognition.
In a broader sense, we think that this work can lead to a better understanding of relevance decisions in
information access, as we presented in our previous work [
In future work, we would like to investigate the underlying dynamics further. What is the specific efect of the six reviewed criteria? What of those caused the most disagreement? Is there a diferent efect for diferent scientific disciplines? How is the influence of newer LLMs, like DeepSeek-V3 or OpenAI’s o3?
The bibliometric data used in this paper are based on the local installation of Web of Science from the Competence Network for Bibliometrics located at DZHW in Berlin. The altmetric data used originate from the Altmetrics Explorer of Altmetric.com. We want to thank Meik Bittkowski for providing access and inside information on RIC and Nils Grote for crawling the SMC websites and conducting the initial LLM-based prestudy.
During the preparation of this work, the authors used ChatGPT and Grammarly to draft content, check grammar and spelling, paraphrase, and reword. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.