Labeling bias, arising from disagreement and inconsistencies among annotators, threatens the reliability of datasets and downstream AI models. This paper introduces a framework based on Subjective Logic (SL) that quantifies dataset trustworthiness by mapping inter-annotator responses into opinions consisting of trust (belief), distrust (disbelief), and uncertainty. The approach provides an interpretable representation of how annotation volume and disagreement influence trust at the instance level. Using CIFAR-10H, we illustrate how trust and uncertainty evolve with diferent numbers of annotations and parameter settings, and how annotator-level reliability can be estimated. These findings highlight the potential of SL to support responsible data curation, such as identifying controversial items for re-labeling and profiling annotator consistency. While our study focuses on image classification, the framework is general and can be extended to probabilistic and textual annotations. Benchmarking against alternative aggregation methods is left as future work.
Artificial intelligence (AI) systems increasingly support high-stakes decisions in healthcare, finance, and
autonomous mobility. While research has advanced the trustworthiness of AI models through fairness,
explainability, and privacy-preserving techniques, less attention has been paid to the trustworthiness
of the training datasets that underpin those models. Empirical studies show that dataset-level issues
such as sampling bias, label noise, and privacy vulnerabilities can erode fairness, robustness, and
interpretability [
While such examples highlight distributional risk (coverage and sampling), labeling bias often appears
together with class imbalance and fairness concerns. Labeling bias refers to systematic deviations in
how data points are annotated, often arising from subjectivity, inconsistent guidelines, or annotator
disagreement. Unlike sampling bias, which afects the dataset at the distribution level, labeling bias is
inherently a local or entry-level phenomenon, since it manifests in the reliability of individual labels.
Inter-Annotator Agreement (IAA) metrics such as Cohen’s Kappa provide a statistical measure of
consistency among annotators, capturing uncertainty coming from disagreement [
These limitations motivate an uncertainty-aware representation that preserves per-item label distributions, quantifies residual doubt from limited or conflicting annotations, and accounts for annotatorspecific reliability when aggregating multiple judgments. Such a representation should map observed annotation evidence into distinct components of support and contradiction for a decided label and behave predictably as annotation volume or disagreement thresholds change.
In this paper, we adopt Subjective Logic (SL) [
More specifically, our work is guided by the following research objectives: • Derive per instance SL opinions from annotation signals and analyze their sensitivity to disagreement thresholds. • Examine and investigate how opinions evolve when the number of annotators changes, and how this afects the balance of belief, disbelief, and uncertainty.
• Explore annotator level reliability as an additional signal for dataset quality management. Contributions. This paper makes three contributions: 1. We formalize a mapping from inter annotator evidence to Subjective Logic opinions that characterize trust, distrust, and uncertainty at the data instance level. 2. We demonstrate the framework on the CIFAR 10H dataset, showing how opinions evolve with annotation. 3. We discuss how our framework can support responsible AI practices, including identifying items for re labeling and profiling annotator reliability.
Our focus in this work is to demonstrate the feasibility and interpretability of applying SL to labeling bias.
Summary and Paper Structure. We show that trust scores derived from SL provide finer-grained insights than majority voting alone. Our approach enables both per-instance trust estimation and annotator-level reliability profiling, ofering actionable signals for dataset refinement and quality assurance. The remainder of this paper is organized as follows: Section 2 reviews related work and the foundations of Subjective Logic. Section 3 presents our methodology. Section 4 outlines the experimental setup. Section 5 presents our results and discusses their implications. Section 6 concludes the paper and identifies future directions.
Code Availability. All code and experiments are available at: https://github.com/Ouatt-Isma/ -Trustworthiness-of-AI-Training-Dataset
Ouattara et al. [
Regarding labeling bias, recent work by Vasilakes et al. [
SL has also been used to evaluate trustworthiness in machine learning metrics and model behavior.
Herd and Burton [
Eforts to systematize dataset evaluation have also emerged through scorecard frameworks such
as METRIC [
Beyond global agreement metrics, a long line of work in crowd annotation modeling has developed
statistical methods to infer latent ground truth. The seminal Dawid and Skene model [
Subsequent methods extended this framework. GLAD (Generative model of Labels, Abilities, and
Dificulties) [
In contrast, our framework based on Subjective Logic explicitly retains belief, disbelief, and uncertainty as first-class quantities . Instead of just merging annotation evidence into a single label, we additionally compute interpretable trust opinions at the instance level, making both consensus and disagreement directly observable. This perspective complements prior work: while Dawid–Skene, GLAD, and MACE focus on inferring hidden ground truth, our approach emphasizes auditable trust quantification , enabling transparent assessment of label reliability and supporting downstream tasks such as dataset curation, targeted re-labeling, and annotator profiling within trustworthy AI pipelines.
Subjective Logic (SL) [
To aggregate evidence from multiple sources or points of view, SL provides discounting and several
fusion operators [
We adopt the four–step trust quantification methodology introduced in previous work [
The methodology unfolds in four steps. We first define the trust proposition , such as assessing whether the dataset is free from bias. Next, we decompose this proposition into multiple atomic propositions, one of which addresses labeling. Focusing on this labeling proposition, we then identify trust sources and collect evidence, with one trust source capturing agreement based on annotator responses. Finally, we quantify subjective opinions by converting annotation evidence into Subjective Logic opinions, and, when multiple trust sources are present, we aggregate these opinions to obtain an overall trust assessment.
To quantify the opinion based on the agreement trust source, we assume the existence of a reference label for each instance (e.g., majority vote or adjudication). For each annotator , we compute a divergence score measuring the distance between the annotator’s label and the reference label. Using a divergence threshold , we partition the evidence into positive evidence (, the number of annotations with ≤ ), negative evidence (, the number of annotations with > ), and uncertainty, modeled either based on the amplitude of the divergence values or based on a prior informative weight .
From this evidence, two quantification schemes can be applied. The Constant–Uncertainty
Quantification [
The Baseline–Prior Quantification [
= , = , = ,
+ + + + + + where > 0 (usually set to 2) is the prior weight that moderates how quickly uncertainty shrinks as more evidence accumulates.
This formulation enables each instance to be assigned a nuanced trust opinion, which can in turn be aggregated across instances to assess dataset–level trustworthiness. The overall reasoning process can be summarized as follows: (1) Trust Proposition → (2) Atomic Propositions → (3) Trust Sources + Evidence Collection → (4) SL Opinion quantification + Aggregated Trust.
Consider an image in the CIFAR–10H dataset annotated by 50 crowdworkers. Suppose 46 annotators agree with the decision label * and 4 disagree. With a prior weight = 2, the opinion is 46 4 2 = 52 ≈ 0.885, = 52 ≈ 0.077, = 52 ≈ 0.038.
This opinion indicates high belief in the correctness of the label, some degree of distrust due to disagreement, and low residual uncertainty. As the number of annotations decreases, uncertainty increases according to = + , where = + .
For clarity, Algorithm 1 summarizes the procedure.
Algorithm 1 Opinion construction from annotations Require: Annotations { ()} or { ()}, decision label * , threshold , prior weight 1: for each instance do 2: ← |{ : divergence( (), * ) ≤ }| 3: ← |{ : divergence( (), * ) > }| 4: ← /( + + ) 5: ← /( + + ) 6: ← /( + + ) 7: Output opinion = (, , , ) 8: end for
We evaluate the proposed framework on the CIFAR–10H dataset [
Each image is treated as a trust proposition. We assume the majority label as the decision label * , and compute divergence for each annotator. With threshold = 0, annotators who agree with * contribute to positive evidence , and those who disagree contribute to negative evidence . Uncertainty is controlled by the prior weight that we set to 2.
To study the efect of annotation volume, we subsample annotations at diferent rates (5, 10, 20, 50 annotators per instance) while preserving the decision label. This allows us to examine how belief, disbelief, and uncertainty evolve as evidence increases.
We report descriptive statistics of opinion masses across the dataset and analyze correlations between trust , distrust and uncertainty against inter–annotator disagreement (measured as 1− majority proportion). This captures whether (, , ) reflects the true dificulty of instances. We also visualize the distribution of opinions using boxplots.
This section presents results from our CIFAR-10H experiments and explores the efects of annotation volume on trust scores. We also discuss how the proposed framework can be extended to richer annotation formats and used for annotator profiling.
Trust Mass Distribution (Figure 1). On the full CIFAR-10H dataset (left), using all available annotations per instance, we observe a high average trust mass, with most instances showing strong annotator agreement. However, this setting may inflate confidence for instances with disproportionately many annotations. After normalizing to 10 annotators per instance via majority-preserving subsampling (right), the relative weight of distrust and uncertainty increases, reducing the dominance of trust mass observed in the full dataset. This shift reflects the trade-of between annotation volume and confidence in label quality, as fewer annotations provide weaker evidence and make disagreement more visible. 00..883030 0.600 Correlation with Disagreement (Table 1). We analyzed how Subjective Logic trust (), distrust (), and uncertainty () masses correlate with inter-annotator disagreement (defined as one minus the majority proportion). As shown in Table 1, trust mass consistently shows a strong negative correlation with disagreement, while distrust mass shows a strong positive correlation, reflecting their complementary roles. Uncertainty mass exhibits a weaker and less stable relationship: it is moderately positive with few annotations (5), but decreases as annotation volume grows and eventually approaches zero. At the same time, average trust increases steadily with more annotations, while both average distrust and uncertainty remain low. These results indicate that SL trust and distrust masses are reliable indicators of annotator disagreement across annotation volumes, whereas uncertainty mass is most informative in low-evidence settings.
Extension to Probabilistic and Natural Language Annotations. Our current analysis assumes categorical annotations. However, many modern datasets include probabilistic labels or free-text annotations. For probabilistic labels, divergences can be computed using KL divergence or crossentropy between annotator distributions and the reference label. For text labels, semantic similarity measures (e.g., cosine similarity between embeddings) can provide divergence values relative to a gold ontology or adjudicated reference. These extensions broaden the applicability of our framework to more complex labeling scenarios.
Calibrating the Divergence Threshold . The divergence threshold determines whether an annotation is counted as positive or negative evidence. It can be calibrated empirically from highconfidence samples, set statistically (e.g., one standard deviation around the mean divergence), or fixed according to domain-specific policy (e.g., = 0 for categorical labels, = 0.1 for soft labels). This lfexibility allows tailoring the framework to diferent annotation contexts.
Inferring Annotator reliability. The same approach as the one detailed in Section 3.2 can be extended to estimate annotator reliability. For annotator , we partition the evidence into positive evidence ( , the number of data with ≤ ) and negative evidence ( , the number of data with > ).
By treating each response as positive or negative evidence, we can construct an opinion about the annotator’s reliability. Such opinions enable the identification of low quality annotators whose responses may warrant down–weighting or further review.
Implications for Responsible AI. The framework contributes to responsible dataset management in three ways. First, high-uncertainty instances can be flagged for re-labeling (or more annotators), supporting active dataset curation. Second, annotator opinions provide auditable signals for quality control in crowdsourcing pipelines. Third, per-instance trust scores can be used to re-weight data during model training, reducing the influence of uncertain or controversial labels. Finally, systematic disagreement patterns can help detect unreliable annotators (e.g., spammers or consistently inconsistent contributors), enabling targeted interventions such as retraining, filtering, or exclusion. These mechanisms complement existing dataset documentation practices and provide actionable insights for trustworthy AI development.
We presented a Subjective Logic-based framework to quantify labeling bias and derive instance-level trust scores from human annotations. By representing inter-annotator disagreement as belief, disbelief, and uncertainty, our method provides a richer characterization of annotation reliability than majority voting or global agreement metrics.
Our experiments on the CIFAR-10H dataset demonstrated how trust varies with annotation volume, highlighting the value of modeling uncertainty explicitly in dataset diagnostics. In particular, reduced annotations led to increased uncertainty and reduced confidence in label correctness, which could have significant implications for downstream learning tasks.
We also outlined extensions of the framework to handle probabilistic and natural language annotations and proposed how trust signals could inform annotator profiling. These capabilities open avenues for more informed data filtering, re-annotation strategies, and training pipeline integration.
Future work includes refining trust threshold calibration, validating annotator trust metrics on larger and more diverse datasets, and exploring how trust scores can be used to dynamically weight training data. Overall, this framework contributes toward principled, transparent tools for assessing dataset reliability within trustworthy AI development pipelines.
This research has received funding from the European Union’s Horizon 2020 EU Research & Innovation program under Grant Agreement No 101069688 (H2020-EU CONNECT). The authors thank the project partners for their collaboration and feedback.
During the preparation of this work, the authors used ChatGPT-4 in order to: Grammar and spelling check.