The scholarly publishing process relies on peer review to uphold the quality of scientific knowledge. However, challenges such as increasing submission volumes and potential malicious behavior undermine its efectiveness. In this study, we evaluate Readersourcing, an alternative peer review approach that leverages communitydriven judgments. Using simulations with synthetic data based on a probabilistic model and a publicly available implementation, we assess six quantities and examine the impact of each component on the outcomes. Our ifndings show that the co-determination algorithm captures distinct aspects of manuscript judgments compared to simpler aggregation strategies. Key simulation parameters consistently influence the computed quantities across diferent settings. We also publicly release the data, code, and simulation runs.
The primary method for disseminating scientific knowledge is the scholarly publishing process, which
relies on peer review. In this process, a scientific article authored by individuals is assessed and evaluated
by peers with equivalent expertise. Although peer review is a well-established method for ensuring the
quality of scientific publications, it is not without drawbacks [
While recent advancements in Artificial Intelligence (AI) technologies make automating peer review
in the scholarly publishing process an appealing prospect, several issues and concerns warrant further
investigation. For instance, existing tools struggle to understand and interpret manuscripts within
the broader context of scientific literature [
In light of this, we specifically focus on the Readersourcing model (RSM), originally introduced by
Mizzaro [
Specifically, we investigate the following Research Questions (RQs): RQ1 How does the probabilistic approach used in our simulations influence the computed quantities of the model? What improvements could enhance its outcomes? RQ2 How efectively does RSM capture meaningful and distinct aspects of judgments made by readers?
How do the insights generated by RSM compare with those from simpler aggregation strategies? RQ3 What is the impact of each component of the RSM model on the computed quantities? How do the diferent components influence the overall results? How does their interaction contribute to the model’s outcomes?
The data, code, and all supplementary materials related to our study are publicly available to the research community at: https://osf.io/kwv47/.
Our contributions are as follows: (i) We show that the co-determination algorithm in RSM provides meaningful diferentiation compared to traditional, widely adopted aggregation strategies. (ii) We identify key simulation parameters that significantly afect the model’s outputs, highlighting those that play a prominent role in computing critical quantities. (iii) We confirm that these key parameters consistently shape the co-determination process across various simulation settings. (iv) We publicly release our simulation dataset to support further research and analysis.
The remainder of this paper is structured as follows: Section 2 provides an overview of the related literature, Section 3 describes the methodology, Section 4 presents the results, and Section 5 discusses the impact and outlines the limitations of our approach. Finally, Section 6 presents the conclusions and indicates directions for future research.
Scholarly publishing, which relies on peer review, is the primary method for disseminating scientific
knowledge. In this process, scientific articles authored by researchers are evaluated by peers with
comparable expertise and, if deemed of suficient quality, are made available to the broader
community [
To address these limitations, several solutions have been proposed [
More recently, eforts have been made to incorporate automated tools and AI into the peer review
process [
As Large Language Models (LLMs) are tested in the field of generating scientific hypotheses [
All in all, while AI and LLMs can undoubtedly enhance the eficiency of the peer review process,
they also raise ethical concerns, particularly regarding the transparency of the process, disclosure
agreements, and the potential replication and amplification of biases inherent in the data or systems [
Readersourcing (RSM) is a crowdsourcing approach to peer review [
RSM involves three key entities: a set ℳ of manuscripts (also referred to as publications, articles, or papers), a set of authors, and a set ℛ of readers. When a reader ∈ ℛ reads a manuscript ∈ ℳ, they assign a numerical value r,m ∈ [0, 100], referred to as judgment (or rating).
Each entity in the model is associated with a score: • The manuscript score m of a manuscript ∈ ℳ is calculated as an aggregation of its judgments, serving as an indicator of its quality. • The author score a of an author ∈ is derived from the aggregation of the scores of the manuscripts they have published, serving as an indicator of their reputation and skills. • The reader score r of a reader ∈ ℛ is determined by comparing their judgments on manuscripts with those of other readers, serving as an indicator of their reputation and skills.
Scores are dynamic and evolve over time based on user behavior and interactions. Each score is paired with a steadiness value, denoted with , that reflects its stability: m for manuscripts, a for set of authors authors, and r for readers. For example, an older manuscript with many evaluations tends to exhibit a high steadiness value, whereas a newly registered reader will have a low steadiness value. Steadiness influences how scores are updated: lower steadiness leads to faster score adjustments in response to new inputs. As the score stabilizes, its steadiness increases.
One can consider RSM as a tripartite graph whose nodes correspond to three sets: authors, manuscripts, and readers. Authors are connected to the manuscripts they publish, and readers are connected to the manuscripts they read. More formally, an edge exists between an author ∈ and a manuscript ∈ ℳ if publishes (such edges are unweighted). Conversely, there is an edge between a reader ∈ ℛ and a manuscript ∈ ℳ if reads ; the weight of this edge corresponds to the judgment r,m that expresses on .
Note that if a user acts as both an author and a reader, they maintain two distinct scores and steadiness values.
The simulation flow is illustrated in Figure 2, while the parameters used in the simulation are summarized in Table 2. Each simulation starts with a fixed number of authors and readers. In this study, these two quantities are set to be equal, i.e., || = |ℛ| (see Step 1 in Figure 2).
It is well established that scholarly publications follow Power Law distributions [
The probability that a specific reader ∈ ℛ reads a specific manuscript ∈ ℳ is denoted as r,m(, ). Assuming the two events are independent, the joint probability that reader reads manuscript is expressed as r,m(, ) = r() · m(). (see Step 5 in Figure 2). To model judgments of readers on manuscripts, we proceed as follows. For each manuscript ∈ ℳ, we draw a reference score (i.e., the “ideal” score of the manuscript), denoted as m, from a Power Law distribution G() with parameter G (see Step 6 in Figure 2).
The judgment r,m assigned by reader ∈ ℛ to manuscript ∈ ℳ is drawn from a Gaussian
distribution with mean m set to the reference score m and standard deviation m. Judgment values
are bounded between 0 and 100 (see Step 7 in Figure 2). In the original work [
The simulation flow is based on the following assumptions, derived from the foundational work of
Mizzaro [
As discussed in Section 6, these assumptions will be addressed and refined in future work.
Power Law Pm s it p r c s u n a m f o # # of readers
αm Joint probability between Pr and Pm 2 2 3 5 2 1 4 | | = | ℛ | 2/5
Step 3: Manuscripts are read following Pm Step 4: Readers read following Pr
manuscripts follows Pr,m
follows PG Probability the manuscript is read by one of the readers Probability the reader reads one of the manuscripts 2 2/5⋅3/7 3
0 56 μmk = gmk = 56 100
To conduct our simulations, we generate a set of configurations by choosing discrete values for six parameters (see Table 2): the number of authors/readers || = |ℛ|, the scaling exponents for each Power Law ( a, m, r, G), and the standard deviation of the Gaussian distribution for the reference score m. We fix the upper limit on the number of manuscripts, max, to a constant value in all simulations, thereby excluding it as a parameter.
The simulations use a population of 250 authors/readers, which can be scaled up in future work. We select exponents of 1.1, 1.2, and 1.3 to explore diferent power law steepness levels, where higher values lead to more concentrated distributions. Standard deviations of 2.5, 5.0, 7.5, and 10.0 control data variability: smaller values keep data closer to the mean, while larger values foster the presence of outliers. These choices allow us to examine the simulation thoroughly under varied conditions. The total number of configurations is the product of the possible values for each parameter: 1 population size × 81 values × 4 standard deviation values, for a total of 324 configurations. We repeat each configuration 10 times to account for stochasticity, leading to 3,024 executions in total.
We conducted all experiments on a machine equipped with an Intel Core i7-10700 CPU, 64 GB of DDR4 RAM, NVIDIA RTX A6000 and RTX 3090 GPUs, a 1 TB NVMe SSD, and a 4 TB HDD. We ran the simulations using Python 3.9.21 in a Conda-based environment. To ensure compatibility with the Parquet serialization format, we used NumPy 2.0.2 and pyarrow 18.1.0.
We use the Python-based implementation of RSM, which is publicly available on GitHub: https: //github.com/EddyMaddalena/Readersourcing_OO.
3.4.1. Approach Our analysis examines the relationship between simulation parameters and the quantities (score and steadiness) computed by the RSM co-determination algorithm for each entity (authors, manuscripts, and readers), based on individual judgments. We compare these computed values across all simulation runs and repetitions, with each comparison focusing on a specific entity.
To ensure reliable results, we exclude inactive entities from the simulation flow, such as unread manuscripts, readers who do not provide judgments, and authors who publish only unread or excluded manuscripts (see Section 3.2). We also omit entities with a steadiness of 0, as it indicates a lack of active participation in the model. Given their absence from the computation, we refer to them as inert. In contrast, we classify as active those entities that participate in the process, i.e., readers who provide judgments, manuscripts that receive judgments, and authors who publish manuscripts that are actively judged. From now on, we will generally refer to quantities using a single specifier. For instance, we will use m instead of m to refer to the manuscript score. 3.4.2. Aggregation Functions We evaluate the performance and distinctiveness of the co-determination algorithm in RSM by comparing its outputs with aggregations derived from simpler, widely adopted strategies for combining individual judgments.
We examine the impact of three commonly used aggregation functions: the arithmetic mean, geometric mean, and median. These functions are relevant in data aggregation tasks for diferent reasons: the arithmetic mean summarizes central tendencies, the geometric mean is well suited for products and rates, and the median is less afected by outliers. By comparing these results with those produced by the co-determination algorithm, we aim to assess whether RSM provides meaningful diferentiation and additional insights beyond traditional aggregation methods. 3.4.3. Random Forest Regressor (RFR) The Random Forest Regressor (RFR) is an ensemble learning method that constructs multiple decision trees during training and outputs the average prediction from all trees. It is highly valued for its ability to capture complex, non-linear relationships between input features and the dependent variable, delivering robust performance with minimal hyperparameter tuning.
In our study, we use RFR to model the relationship between the simulation parameters and the quantities produced by the co-determination algorithm of RSM. We treat a, m, r, G, and m (see Table 2) as features, and a, m, r, a, m, and r (see Table 1) as outputs. We apply RFR separately to the data for each of the three entities. For example, one RFR run compares a with a and a.
The feature importance rankings generated by the Random Forest help pinpoint which simulation
parameters most significantly afect the output quantities. We leverage these insights to evaluate
the model’s robustness and to confirm the importance of these parameters in subsequent analyses,
such as Analysis of Variance (ANOVA). We use the RandomForestRegressor implementation from
scikit-learn [
We use one-way ANOVA to assess the influence of simulation parameters on the quantities computed by the co-determination algorithm of RSM. We adopt the same definitions of features and outputs as for RFR (see Section 3.4.3). By analyzing how these quantities vary across diferent parameter settings, we aim to identify which parameters significantly afect the results, thus validating the insights from our feature importance analysis. We use the f_oneway implementation from the scipy package [42].
To evaluate the efect of our probabilistic approach in simulations (RQ1), we focus on inert entities. By design, our approach is designed toallow their presence (see Section 3.2 and Section 3.4.1). To guide future experimental designs, we investigate inert entities using a twofold approach.
First, we examine this phenomenon quantitatively by counting the number of active and inert entities in our simulations, ensuring that inert entities do not overwhelmingly dominate. Table 3 shows the number of active authors, manuscripts, and readers. For each entity, we report the mean, minimum, maximum, and the first, second, and third quartiles. Notably, authors tend to remain active longer in the network due to higher connectivity; of the 250 simulated authors, 214 (85%) are active.
Next, we analyze correlations to understand how these values relate to the input parameters, aiming to identify potential interventions for tuning simulation parameters to reduce the occurrence of inert entities. Figure 3 shows the correlation values between the simulation parameters and the number of active entities (first to third columns), the number of inert entities (fourth to sixth columns), and the ratio between active and inert entities (seventh to ninth columns). Positive correlations are shown in blue, while negative correlations appear in light green.
As the figure indicates, some simulation parameters increase the number of active entities, whereas others increase the number of inert ones. Specifically, the G and m parameters do not afect inert entities. This is expected because they govern aspects related to individual judgments and do not influence edge formation in the tripartite graph. In contrast, the three parameters a, m, and r, which govern the Power Law distributions for authors, manuscripts, and readers, do afect inert entities. The correlations show that these parameters decrease the number of active entities, increase the number of inert ones, and ultimately increment the ratio between inert and active entities.
To evaluate how efectively RSM captures distinct aspects of manuscript judgments (RQ2), we compare the model’s outputs with simpler strategies for aggregating individual judgments. Figure 4 presents the resulting Kendall’s correlation coeficients. The first column shows correlations with the reference score, while the second to fourth columns show correlations with judgments aggregated using the arithmetic mean, geometric mean, and median, respectively. The final column shows the manuscript score. Positive correlations are shown in blue, and negative correlations in light green.
The manuscript score m correlates moderately with the reference score but strongly with simpler aggregation strategies, namely the arithmetic mean, geometric mean, and median. This stronger correlation is likely due to the efect of the normal distribution, as the median aligns well with such a distribution. Furthermore, because the dataset is generated from skilled readers with highly consistent 0.3 0.3 1 and precise judgments, it minimizes the influence of rewarding better-performing readers. Consequently, the model emphasizes overall trends, while its weaker correlation with the reference score suggests that it captures a distinct dimension of judgment. In summary, the model places more weight on aggregated judgment patterns than on the reference score. We discuss this further in Section 6.
To evaluate the impact of individual features on the outputs (RQ3), we begin by examining their correlations. Figure 5 shows Kendall’s correlation coeficients with respect to both simulation parameters and computed quantities. In this figure, as well as in subsequent ones, positive correlations are shown in blue, while negative correlations in light green.
In the lower-right section of the heatmap, we observe a strong correlation among the steadiness values computed for each entity. High steadiness in one entity is associated with high steadiness in others, indicating that steadiness is a system-wide property observed uniformly across all entities. In the top-center blue section, the positive correlation among author scores indicates that authors with higher manuscript scores tend to receive higher individual scores. An unexpected result is the negative correlation between author and manuscript scores, on the one hand, and reader scores on the other: when authors and manuscripts have higher scores, readers tend to have lower scores, and vice versa. We discuss this further in Section 5.
Focusing on the feature importance analysis conducted using RFR, Figure 5 shows the computed feature importance scores, revealing each simulation parameter’s contribution. The first three columns pertain to scores, while the fourth through sixth columns represent steadiness values.
As expected, the G parameter has the greatest influence on steadiness, since variations in the manuscript’s reference score cause fluctuations in steadiness. In contrast, the r parameter afects the manuscript score by determining how many readers provide judgments. Meanwhile, the a parameter, which governs the number of manuscripts published, influences the reader score. This suggests a relationship between manuscript amount and the consistency of reader judgments.
Next, we present the results of our one-way ANOVA analysis. Figure 7 shows the F-statistic values, which quantify the strength of the relationships between simulation parameters and computed quantities. All values are statistically significant ( < 0.0001), except for the a column.
As expected, the manuscript score m is strongly influenced by all features, given that it arises from interactions among the parameters. The m parameter, which influences all quantities, especially the reader score r, afects judgment variability. When the standard deviation is low, errors in judgments are smaller, efectively making the reader “skilled”. Examining higher variability could lead to additional insights into the reader score, as noted in Section 6.
Both the RFR and ANOVA analyses consistently indicate that simulation parameters significantly influence the manuscript score m, with G and r exerting particularly strong efects. While RFR highlights the relative importance of each feature, ANOVA confirms the statistical significance of these impacts. The high importance of G revealed by RFR is confirmed by ANOVA, underscoring how variations in the reference score drive changes in steadiness. Similarly, both analyses highlight the influence of the standard deviation parameter on the reader score, highlighting the role of judgment variability in consistency.
Our findings indicate that the probabilistic approach efectively accounts for inert entities within simulations. Correlation analyses reveal that certain simulation parameters afect the number of inert entities by reducing the number of active entities and increasing the number of inert ones. These insights guide parameter tuning to minimize the occurrence of inert entities and refine experimental designs in future studies (RQ1).
Additionally, our results show that RSM captures distinct aspects of manuscript judgments by focusing on aggregated judgment patterns, such as the mean and median, rather than the reference score. Its strong correlation with simpler aggregation methods reflects overall judgment trends, while its weaker correlation with the reference score suggests it captures a diferent dimension of judgments (RQ2).
Further analysis of RSM model components reveals how each feature contributes to the final results. The strong correlation among steadiness values highlights its role as a consistent, system-wide property. Feature importance analysis suggests that a higher manuscript count leads to more stable judgments, while ANOVA highlights the impact of standard deviation on score steadiness. Greater variability in judgments produces less stable outcomes, reflecting the complex interactions of model components (RQ3).
A limitation of our approach is that reference scores follow a Power Law distribution, often resulting in many manuscripts with near-zero scores. This issue is worsened by the bounded scale, where clipping values outside the predefined range can distort judgments [ 43]. Low-quality manuscripts with near-zero scores are considered “easy” to judge, which leads to small judgment errors. This dynamic creates a countervariance efect where skilled readers provide high-quality judgments, while authors of low-quality manuscripts receive lower scores. An unbounded scale, such as Magnitude Estimation [44], could mitigate this issue.
Another aspect requiring further attention is the treatment of inert nodes. While excluding them from the analysis is reasonable (see Section 3.4.1), real-world publishing systems might indeed include unread manuscripts and inactive readers. Exploring alternative approaches to inert nodes could provide further insights, such as applying graph growth models where inert nodes are “drawn” to highly connected nodes [45].
Our simulation flow also employs four bounded power laws, which introduce four additional parameters and increase complexity. A potential solution is proposed by Antipov et al. [46], who reduce the number of parameters by randomly selecting values from a suitably scaled power law distribution at each iteration.
In conclusion, our study shows the efectiveness of the co-determination algorithm in RSM, highlighting key simulation parameters that significantly influence the model’s outputs and confirming their consistent impact across various settings.
Future research should aim to enhance RSM by comparing it with other models and exploring more
complex simulations with diverse scenarios and variables. Mizzaro [
A possible comparison with existing models would involve evaluating RSM reader scores against
those from the co-determination algorithm in TrueReview [
Future simulations could benefit from real-world data, such as the 1.5 million preprints available
on arXiv [47]. As proposed by Mizzaro [
Finally, incorporating LLMs into simulations is becoming increasingly relevant due to their ability
to replicate human-like behavior. For instance, Park et al. [48] describe an agent-based framework
that emulates realistic individual behaviors and attitudes. We believe such a framework could enrich
simulations by modeling reader judgments influenced by demographic, ideological, and personality
traits. However, caution is advised when using LLMs as proxies for human decision-making [
This research is partially supported by the PRIN 2022 Project – “MoT—The Measure of Truth: An Evaluation-Centered Machine-Human Hybrid Framework for Assessing Information Truthfulness” (Code No. 20227F2ZN3, CUP No. G53D23002800006), funded by the European Union – Next Generation EU – PNRR M4 C2 I1.1, and by the Strategic Plan of the University of Udine–Interdepartment Project on Artificial Intelligence (2020-25). [37] N. Ferro, Y. Kim, M. Sanderson, Using Collection Shards to Study Retrieval Performance Efect Sizes,
ACM Trans. Inf. Syst. 37 (2019). URL: https://doi.org/10.1145/3310364. doi:10.1145/3310364. [38] N. Ferro, G. Silvello, A General Linear Mixed Models Approach to Study System Component Efects, in: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’16, Association for Computing Machinery, New York, NY, USA, 2016, p. 25–34. doi:10.1145/2911451.2911530. [39] N. Ferro, G. Silvello, Toward an anatomy of IR system component performances, Journal of the Association for Information Science and Technology 69 (2018) 187–200. doi:doi.org/10.1002/ asi.23910. [40] K. Roitero, B. Carterette, R. Mehrotra, M. Lalmas, Leveraging Behavioral Heterogeneity Across Markets for Cross-Market Training of Recommender Systems, in: Companion Proceedings of the Web Conference 2020, WWW ’20, Association for Computing Machinery, New York, NY, USA, 2020, p. 694–702. doi:10.1145/3366424.3384362. [41] F. Zampieri, K. Roitero, J. S. Culpepper, O. Kurland, S. Mizzaro, On Topic Dificulty in IR Evaluation: The Efect of Systems, Corpora, and System Components, in: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 909–912. doi:10.1145/3331184.3331279. [42] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, et al., SciPy 1.0: fundamental algorithms for scientific computing in Python, Nature Methods 17 (2020) 261–272. doi:10.1038/s41592-019-0686-2. [43] M. Hubert, P. J. Rousseeuw, K. Vanden Branden, Robust Statistics: Theory and Methods, 2nd ed.,
Wiley, 2011. doi:10.1002/0470010940. [44] E. Maddalena, S. Mizzaro, F. Scholer, A. Turpin, On Crowdsourcing Relevance Magnitudes for Information Retrieval Evaluation, ACM Transactions on Information Systems 35 (2017). doi:10.1145/3002172. [45] F. Menczer, S. Fortunato, C. A. Davis, A First Course in Network Science, Cambridge University
Press, 2020. doi:10.1017/9781108653947. [46] D. Antipov, M. Buzdalov, B. Doerr, Lazy Parameter Tuning and Control: Choosing All Parameters
Randomly from a Power-Law Distribution, 2023. doi:10.1007/s00453-023-01098-z. [47] C. B. Clement, M. Bierbaum, K. P. O’Keefe, A. A. Alemi, On the Use of ArXiv as a Dataset, arXiv, 2019. doi:10.48550/arXiv.1905.00075. arXiv:1905.00075. [48] J. S. Park, C. Q. Zou, A. Shaw, B. M. Hill, C. Cai, M. R. Morris, R. Willer, P. Liang, M. S. Bernstein, Generative Agent Simulations of 1,000 People, arXiv, 2024. doi:10.48550/arXiv.2411.10109. arXiv:2411.10109.