Linking textual values in tabular data to their corresponding entities in a Knowledge Base is a core task across a variety of data integration and enrichment applications. Although Large Language Models (LLMs) have shown State-of-The-Art performance in Entity Linking (EL) tasks, their deployment in real-world scenarios requires not only accurate predictions but also reliable uncertainty estimates, which require resource-demanding multi-shot inference, posing serious limits to their actual applicability. As a more eficient alternative, we investigate a self-supervised approach for estimating uncertainty from single-shot LLM outputs using token-level features, reducing the need for multiple generations. Evaluation is performed on an EL task on tabular data across multiple LLMs, showing that the resulting uncertainty estimates are highly efective in detecting low-accuracy outputs. This is achieved at a fraction of the computational cost, ultimately supporting a cost-efective integration of uncertainty measures into LLM-based EL workflows. The method ofers a practical way to incorporate uncertainty estimation into EL workflows with limited computational overhead.
of confidence scores enables the identification of potential errors by prioritizing links that humans could review to improve the quality of the results. Understanding and quantifying the uncertainty of the LLM output is essential to ensure the robustness and trustworthiness of the model output, to highlight where the model is fragile or ungrounded, and to direct human intervention in order to maximize its eficacy.
An established way of measuring uncertainty in LLMs is to determine the likelihood of an answer being consistent over multiple independent generations [13, 14, 15]. In the context of LLMs, this introduces significant computational overhead, as the generation time scales linearly with the number of tokens in both the prompt and the output, even with caching techniques [16]. Additionally, if we consider that EL may be applied to large datasets, this approach may be unsustainable. To mitigate the resource demands of large-scale EL scenarios, we propose an eficient approach to estimate uncertainty in LLMs without relying on multiple generations during inference. More precisely, we propose an eficient self-supervised method that learns to estimate the uncertainty observed over multiple generations of an LLM using observables from a single generation. The method leverages token-level features, derived from the probability distribution over the output vocabulary, to train a lightweight regression model targeting the “true” observed uncertainty.
Although in this paper we focus on EL on tabular data, we believe that our method could be applied to diferent closed-form tasks that can benefit from uncertainty quantification.
This paper makes the following contributions: • Formalizes uncertainty-aware EL on tabular data and proposes a self-supervised regressor that learns to approximate multi-shot uncertainty from single-shot token-level features; • Introduces a lightweight, model-agnostic feature set from output-layer probabilities and optional intermediate-layer signals, requiring no task labels; • Evaluates the approach across several instruction-tuned LLMs, showing how uncertaintyawareness can efectively lead to the detection of low-accuracy outputs, which can then be corrected to improve accuracy under a constrained review budget.
Uncertainty in EL for Tabular Data. Associating confidence scores in matching tasks, including EL for tabular data, is a well-established practice [17, 18, 4], where scoring a list of retrieved candidates is a typical intermediate step. This is also true in approaches combining a pretrained language model with classifiers, such as TURL [10] and UNICORN [19]. While scores computed in pre-LLMs methods somehow support confidence estimation, less attention has been dedicated to systematic analyses of uncertainty estimation in this task. One approach based on a deep neural network proposes a supervised method that considers the matching score of the best candidate and its distance from the second-best to quantify confidence [ 20]. The authors show that computed confidence scores help prioritize links to revise with a progressive budget. To the best of our knowledge, recent approaches based on LLMs [7, 8, 9, 3, 5] have not investigated how to exploit uncertainty measures in generative approaches. Uncertainty measures may support the decision whether to link or not a top candidate (e.g., based on a threshold), including the detection of NIL entities (i.e., values associated with entities not in the KB) [4]. Additionally, little attention has been dedicated to the impact of the variability of the links predicted by LLMs under diferent generations within this task. Uncertainty Estimation for LLMs. Confidence Score methods leverage single-shot proxies from output probabilities (e.g., entropy, log-probabilities, perplexity) but can be overconfident when wrong [21, 13, 22, 23, 24]; Semantic Consistency approaches, involving the generation of multiple outputs for the same prompt and measuring their consistency or semantic similarity, are efective but computationally expensive [22, 14]; Supervised methods learn calibrated uncertainty from features of generated text or hidden states and often outperform unsupervised heuristics, but require an annotated dataset [25, 26]; Ensemble- and Bayesian-Inspired approaches estimate approximate Bayesian uncertainty (e.g., deep ensembles, Monte-Carlo dropout) but are generally impractical at LLM scale [27, 28, 29]; Verbalized and Self-Reported Uncertainty methods improve interpretability by prompting models to report confidence, with mixed reliability across tasks [30, 31, 32]. Finally, [33, 34] employ Uncertainty-Aware In-Context Learning to filter/refine or to guide iterative prompting, improving reliability on open-ended tasks. This work.
Our approach intersects multiple-generations and supervised paradigms by leveraging a self-supervised uncertainty regressor that learns from multiple-generations outputs how to estimate uncertainty from a single generation. Although this work shares some conceptual similarities with [26], our method does not require supervised labels and focuses on tabular EL.
Let a tabular EL instance be defined by an input tuple (, , KG), where is a table, is a mention at coordinates (, ), and KG = {1, . . . , } is a retriever-provided set of candidate entities, dependent on the particular Knowledge Graph KG. An LLM conditions on a prompt x built from (, , KG) and generates a textual answer from which a deterministic post-processor extracts a selected candidate ^() ∈ KG. For non-deterministic decoding, repeated generations {() }=1 induce an empirical distribution ^ans(·| x) over answers3, from which an uncertainty (· ) can be estimated. Here, (· ) is the uncertainty score derived from the empirical distribution of generations4. Our goal is to output, for each x, both a selected candidate ^() and an uncertainty score ^() that correlates with (· ) computed a posteriori from multiple generations. We learn this score via an uncertainty regressor ℎ from single-shot token-level features (cf. Section 3.4). At deployment, we use ^() to flag instances for manual review. Algorithm 1 outlines the complete training (warm-up) and inference workflow.
(
=1 I[() = ], where I[] = 1 if the predicate holds, and 0 otherwise. 5Let : → map an answer to its semantic class = () [14]. The empirical distribution over semantic classes is ^sem(|x) = ∑︀ ∈(x) ^ans(|x)I[() = ] = 1 ∑︀
=1 I[(()) = ], where (x) is the set of observed distinct answers.
Our practical objective is to estimate uncertainty and use it to flag uncertain answers for manual review, without relying on an explicit ground truth. The underlying intuition is that higher uncertainty is correlated with lower accuracy, while low uncertainty alone does not imply higher accuracy. Accuracy is calculated as the average answer correctness across generated outputs for each input prompt. Uncertainty–Accuracy Relationship. Figure 1 highlights the efect of setting a threshold on uncertainty to flag items for manual review. Interpreting the flag decision as a binary classifier, the green area contains correctly flagged low-accuracy cases (true positives). Reviewing these comes at the cost of also reviewing cases in the yellow area (false positives), where accuracy would be satisfactory but uncertainty is high. Cases in the red area are consistently wrong but with low variability; these are false negatives and are not recoverable by uncertainty-based thresholding. Intuitively, lowering the uncertainty threshold expands the region of flagged cases. On the other hand, raising the threshold shrinks both green and yellow zones, reducing workload at the risk of missing additional bad cases. Notably, the subset of red points with zero uncertainty, which are consistently wrong across generations, remains invariant to the threshold and unrecoverable by uncertainty thresholding alone. Sweeping the threshold traces the ROC analysis discussed later.
Eficient Uncertainty Estimation via Self -Supervised Regression. A large class of uncertainty estimation methods, including widely used metrics like PE and SE, depend on repeated sampling for their computation, which comes with significant resource overhead. We therefore propose an eficient self-supervised method to approximate such uncertainty measures using information from a single generation, making them more suitable for downstream use. The target variable – either PE or SE, measured on independent runs of an LLM – is regressed from observable features using a Random Forest model implemented via XGBRFRegressor, with 100 estimators [35, 36]. We optimize the Mean Squared Error (MSE) between the predicted score ^() and the normalized multi-shot target (^ans) (Section 3.2). We utilize 10-fold cross-validation grouped by prompt to avoid data leakage and to obtain reliable performance estimates on unseen cases. The regressor is trained in a warm-up phase, during which generations per prompt are collected and the derived target variable is learned. At runtime, the regressor estimates the uncertainty from a single generation in response to the same prompt. We denote the learned uncertainty regressor by ℎ, which maps features (, x) extracted from a target model ’s single generation = (x) to an estimate ^() = ℎ( (, x)). The method is self-supervised as it does not require externally labeled data. Moreover, the regression incurs negligible computational overhead. Further considerations regarding learning convergence are provided in Section 4.5.
We collect the features, as defined in Section 3.4, from independent runs of an LLM. We benchmark
the proposed method considering: (
We rely on features (, x) that are observable or easily computable during the inference process of an LLM . We compute token-wise features for specific portions of the prompt and generated tokens. Instead of aggregating the features at the sequence level [13], we maintain the features at the token level. Let be a vocabulary of tokens and let x be a sequence of tokens from . Given the input prompt x, an LLM computes a logit vector z = (x) ∈ R| | for each token. It then applies a softmax function to derive the stochastic vector p, i.e., the probability of being the next token for every ∈ : exp(/ ) p ∈ R| | s.t. = (z) = ∑︀∈[| |] exp(/ ) where is the temperature parameter, which controls the randomness of the output distribution. We calculate for each token7: • Max probability. The maximum probability observed across the output vocabulary, measured at the output layer of the model. • Entropy. The entropy of the token probability distribution
M(p) = m∈[|ax|] H(p) = − ∑︁ log ∈[| |] over all vocabulary tokens at the output layer . Lower entropy indicates higher confidence, while higher entropy reflects more uncertainty. • LogitLens [37]. The Kullback-Leibler (KL) divergence between the probability distribution at each intermediate layer and the probability distribution at the output layer [37].
KL(p ‖ p) =
∑︁ log
∈[| |]
where p is measured for each layer ∈ [− 1] and p is measured on , resulting in − 1 divergence
measures quantifying the deviation between the distribution at the intermediate layers and the
output layer. At each layer , p = (z) = (︀ LayerNorm[h] )︀ is obtained by applying the last
LayerNorm to the hidden representation h and multiplying it by the unembedding matrix .
6The token indices collected in (
1. For each prompt x, collect independent generations from the LLM to obtain {()}=1. 2. Compute the a posteriori uncertainty target (^ans/sem) either as normalized PE or SE (Equations 1 and 2, respectively). 3. For each generation ∈ [ ], extract per-generation features ()(, x) using token-level observables (Section 3.4) over selected segments (e.g., Postilla, first Generated tokens). 4. Form training pairs (︀ ()(, x), (^ans/sem)︀) for = 1, . . . , and for all prompts x; train the regressor ℎ on the training pairs.
1. For each new prompt x, run a single generation to produce an answer = (x) and extract (, x).
2. Return and predict ^() = ℎ( (, x)) and use it to decide if the item should be reviewed.
At inference time, the proposed approach reduces the number of LLM generations used to estimate uncertainty from to 1 and adds only a lightweight regression pass on features extracted during generation. In typical settings, the regression overhead is negligible compared to a single forward pass of the LLM; thus, single-shot estimated PE/SE achieves most of the benefit of multi-shot uncertainty at substantially lower cost. For a theoretical analysis of Transformer time complexity with and without KV-cache during generation, see Appendix F.
In this work, we address the following research questions:
Q1 Do our single-shot uncertainty estimates identify low-accuracy answers? Q2 How much does uncertainty-guided manual correction improve accuracy under a budget ? Q3 How much warm-up data is needed to learn the uncertainty regressor ℎ efectively? Q4 How does temperature afect the uncertainty/accuracy trade-of? Reproducible code and plotting scripts are available at: https://github.com/carloalbertobono/llm-u ncertainty.
We address the Entity Linking (EL) task on tabular data, formalized in Section 3.1, where each table mention must be linked to its corresponding entity in a Knowledge Base. The dataset used, TableInstructEL-2K, is adapted from the TableInstruct EL test set [5], which is formatted for LLMs and contains 2,000 mentions annotated with Wikidata entities, with exactly one correct entity per mention. The original dataset included ∼ 600 mentions with only one candidate (i.e., the correct one), which limits our ability to evaluate answer variability. To address this issue, the candidates were enriched with those retrieved using LamAPI [38], a full-fledged retriever that returns richer candidate sets. In the resulting dataset, 1,650 mentions (∼ 91%) include at least 45 candidate entities each. Prompts follow the “Entity Linking” template from [5] and include the following segments: an Instruction that provides context and task guidelines; an Input table, Markdown-serialized; a Question asking which of the provided referent candidates corresponds to a specific table mention; a fixed-width Postilla that clarifies the expected answer format. The Generated segment contains the model’s predicted answer. To support understanding, a minimal, self-contained illustrative example of the prompt, candidate list, and expected answer format is provided in Appendix A.
Never correct (w/o Unc.)
Never correct (w/ Unc.)
Sometimes correct
Recoverable
total Experiments are performed using the following models: Gemma-2-2B-Instruct, Gemma-2-9B-Instruct, Llama-3.1-8B-Instruct, TableLlama, Qwen2.5-7B-Instruct. These models were chosen as a representative set of instruction-tuned, open-source language models. Additionally, TableLlama is considered state-of-the-art in the Entity Linking task on tabular data [3]. The number of generations for each prompt is set to = 10, while the temperature is set to 1.0. Features (observables) are extracted over diferent portions of tokens, specifically from the Postilla and from the first 10 tokens of the Generated segment8, considering features from the output layer alone (M(p) and H(p)), from the intermediate layers alone (LogitLens), and combined. These combinations are motivated by the hypothesis that uncertainty may manifest diferently across prompt segments and feature groups, necessitating an empirical assessment of their impact. We compare the entropy values derived with our method against the following baselines: PE, SE, PP, and an oracle with access to the true answers.
In the remainder of the paper, we utilize two diferent concepts of accuracy. Answer-level accuracy refers to the accuracy of the answers to a given prompt, computed over multiple generations. Answer-level accuracy can then be aggregated at the level of a set of prompts; we refer to dataset-level accuracy to indicate the average answer-level accuracy computed over the whole dataset.
Regarding the performance of the diferent LLMs on the task, Table 1 summarizes the proportion of answers with and without uncertainty – Recoverable and Unrecoverable, respectively – considering = 10 runs per item. Answers with no associated uncertainty are grouped into Always correct and Never correct; answers with associated uncertainty are grouped into Never correct and Sometimes correct. The rightmost column (Recoverable total) adds Never correct (w/ Unc.) and Sometimes correct to give the recoverable fraction, i.e., the share of items that uncertainty thresholding can, in principle, surface for manual correction. It should be noted that Never correct items are partitioned into zero-variance items (w/o Unc.) and items with observed uncertainty (w/ Unc.).
Gemma-2-2B-Instruct, the smallest model in terms of parameters, and Llama-3.1-8B-Instruct show the lowest Always correct proportion, suggesting limited reliability. In contrast, TableLlama and Gemma-2-9B-Instruct show the highest Always correct proportion, with Qwen2.5-7B-Instruct falling in between. The Recoverable total highlights the best-case pay-of of uncertainty-guided review. In general, the fraction of unrecoverable-never-correct cases remains modest. 8The performance difers depending on whether features are extracted from the Postilla or Generated tokens. A more detailed analysis can be found in Appendix C.
Gemma-2-2B-it
Gemma-2-9B-it
We assess the reliability of our estimates through a series of targeted experiments. First, we test whether the uncertainty measures can identify mentions with low answer-level accuracy, where “low” means that the answer-level accuracy is below 0.5 over = 10 generations. To this end, we perform a ROC analysis [39] on both the estimated and baseline entropy measures. We report the results in terms of true positive rate (TPR) and false positive rate (FPR), where TPR corresponds to the actual low-accuracy cases that the uncertainty-based method correctly flags. At the same time, FPR is the fraction of high-accuracy cases (accuracy ≥ 0.5) that the method wrongly flags as low-accuracy. Here, the positive class corresponds to the decision “flag for manual review.” In the ROC plot, each curve traces the trade-of between TPR and FPR as a threshold on the uncertainty score—used to decide “flag or not”—is varied. The diagonal line represents random chance; curves that bow upward (high TPR at low FPR) indicate efective uncertainty signals for identifying low-accuracy items.
Moreover, we assess the impact of the uncertainty measures on budget-dependent manual correction. To this end, we investigate the average accuracy obtained by selecting the most uncertain items up to a budget , according to a given uncertainty measure. We assess the aggregate accuracy at the dataset level after correcting a fraction of the items.
Detecting Low-Accuracy Answers We evaluate how well the uncertainty estimate can identify low accuracy items (answer-level accuracy ≤ 0.5). Figure 2 shows the ROC curves for the considered LLMs, comparing the diferent uncertainty estimates and baselines. ROC curves are a well-established metric for assessing classification performance, and illustrate the trade-of between true positive rate and false positive rate across diferent thresholds, in our case, on uncertainty estimates. PE and SE based on generations (dashed lines) show the highest performance. The gap between these baselines and the estimated PE/SE (solid lines) captures the performance loss due to estimating the multiple-generations entropy with the information from a single generation. Notably, when using the best-performing models, TableLlama and Gemma-2-9B-Instruct, PE and SE computed on the multiple generations closely approximate the estimated PE and SE (cf. green and purple solid and dashed lines). However, across other models, the PE and SE baselines show a better ability to distinguish high- and low-accuracy cases. Moreover, PP is consistently outperformed by other methods. In general, the performance of our method falls between PE/SE and PP. We recall that the estimates of our method are obtained at a fraction of the computational cost, by computing = 1 generations instead of = 10.
Regarding the use of features derived from diferent stages of the generation process, estimates based on the features observed in the Postilla segment appear to be generally less informative compared to the Generated segment. However, focusing on the utilization of the LogitLens features, we highlight that their contribution is substantial when using the Postilla features, while it becomes negligible when using the Generated features.
Uncertainty-Guided Correction under Budget We evaluate how much accuracy improves when a human annotator—assumed to be always correct—uses a limited budget to manually revise the most uncertain cases. Items are ranked by the various uncertainty signals, and the top fraction is corrected; the resulting dataset-level accuracy after these ideal corrections is shown in Figure 3. Each curve corresponds to a diferent ranking method: the proposed PE estimate (e.g., PE(Generated, MaxP+H)), the a posteriori PE/SE baseline, perplexity, random selection, and an oracle that ranks strictly by true low-accuracy severity. The gray oracle curve defines the upper bound, while the yellow random curve gives a reference for uninformed correction. Across the evaluated models, uncertainty-guided correction substantially outperforms random selection, with the largest marginal gains at small budgets. The single-shot regressors closely track the multi-shot PE/SE baselines, recovering most of their improvement while reducing LLM calls from N to 1, whereas the Perplexity baseline produces consistently lower curves. TableLlama and Gemma-2-9B exhibit steeper initial slopes, indicating a high concentration of correctable errors among the most-uncertain items, while other models obtain comparable gains only at larger budget fractions . TableLlama Gemma2-9b-it Qwen2.5-7b-it LLaMA-3.1-8b-instruct
Gemma2-2b-it 4.5. Q3: Learnability of the uncertainty regressor ℎ Since our self-supervised regressor ℎ requires a warm-up phase for learning, we conducted two supplementary experiments to assess its practical applicability. First, we estimated the number of examples that are necessary to reach the stability of the regressors’ performance. As a complementary analysis, we assess if a proxy target measure can be derived, trading some accuracy for a reduced computational cost. Regressor Convergence as a Function of Training Size We train the regressor with an increasing number of cases and assess its Spearman correlation with the target. We perform a 10-fold cross-validation, where in each fold we keep the validation set fixed and train the regressor by expanding the training set one-by-one. Figure 4 reports the average performance over all the folds for each model. We regress the PE target only, using the M(p) and H(p) features on the first 10 Generated tokens. Stable performance is reached with a limited number of items, e.g., 10% − 20% of the dataset, depending on the model. This highlights how, even with a limited number of examples, the approach can successfully learn to estimate uncertainty. This, in turn, impacts the overall eficiency and sustainability of the method, since the number of multiple-generations cases necessary to train the model appears to be limited. We recall that, since the method is self-supervised, it does not rely on external annotations.
TableLlama Gemma-2-9B-it Avg. Accuracy
Gemma-2-2B-it Avg. Accuracy TableLlama Avg. Accuracy
Approximated Ground Truth We finally consider whether reducing the resources dedicated to the multiple generations can retain enough information to approximate the full uncertainty – that is, the uncertainty observed with the original generations when considering all the generated tokens. For this experiment, performed for illustration purposes, we focus on the best-performing model, TableLlama, and examine the PE target only. We cap the number of generations and the number of generated tokens, both of which impact the computational burden linearly. Figure 5 shows the Spearman correlation between the approximated PE and the full PE. In this setup, even under computational constraints, the derived approximate PE is reasonably correlated with the full PE, i.e., computed on the complete set of generations and tokens. This highlights how an approximate target signal could be derived by trading accuracy for eficiency.
Our approach proposes a measure of uncertainty related to output variability. However, the temperature setting afects the trade-of between output variability – needed to estimate uncertainty – and task accuracy. In this section, we evaluate the impact of the temperature on the task performance. We summarize the performance as the area under the curve (AUC) that describes dataset-level accuracy as a function of correction budget (cf. Figure 3 for comparison). We chose this metric as it summarizes the practical downstream usability of the proposed method.
We systematically sweep the temperature value in the range 0.0 ≤ ≤ 2.0 with steps of 0.1 to assess its efect on the AUC. Due to computational budget constraints, we utilize a subset of 200 elements from the original dataset and focus on PE. Figure 6 reports the AUC over the temperature and the average overall accuracy obtained with selected models. It can be observed that the most convenient performance is obtained for balanced values, for which dataset-level accuracy does not change drastically, implying that temperature variations up to a certain level do not compromise the task performance. Setting smaller values degrades the performance, since lower output variability directly reduces observable uncertainty. Similarly, setting a higher value leads to higher observed uncertainties, which are related to noisy outputs and do not reflect the actual answer uncertainty. Based on these observations, we set = 1.0 for all the experiments previously discussed in this paper. Nonetheless, this experiment illustrates how temperature can be adjusted depending on the specific scenario.
In this work, we propose a lightweight, self-supervised approach for obtaining uncertainty estimates. An approximate entropy measure is regressed by leveraging features that are observable at inference time, based on final- and intermediate-layer token probability distributions. This approximation is calculated, after an initial warm-up phase, without relying on multiple, costly generation runs, making it suitable for practical use in real-world EL workflows.
We validate the proposed method on the task of Entity Linking for tabular data, showing a strong correlation with uncertainty observed over multiple generations estimates, with significantly reduced computational overhead. Empirical evaluations, performed across several instruction-tuned LLMs, demonstrate that the method is highly efective at identifying low-accuracy outputs, in particular when using features derived from generated tokens. We also quantitatively assess the number of cases needed for learning to predict uncertainty, showing that a limited warm-up phase is suficient. Finally, we measure how the LLM temperature influences the trade-of between uncertainty and the overall accuracy of the method.
The proposed method is general, and it may benefit other closed-form tasks beyond the considered context. Future experiments, aimed at testing its generalizability across tasks and domains – particularly in open-ended EL settings, where diferent uncertainty measures could have diferent behaviors and eficiency trade-ofs – could extend the results provided in this study. A warm-up phase, involving multiple LLM generations, is required for training, which introduces a computational cost. However, this cost is mitigated by the rapid convergence of the learning phase, maintaining the eficiency of the overall process. Additionally, evaluating the transferability of learned regressors across datasets related to the same task may provide further insights into the robustness of the method.
While our method is model-agnostic and applies unchanged to any model exposing token logprobabilities (and optionally hidden states), we leave the profiling and evaluation of larger-capacity models to future work. When only output-layer probabilities are available (e.g., closed models), our output-layer-only variant remains applicable.
We also show that some of the features related to the intermediate layers during the generation process are partially able to fill the performance gap observed between prompt-related and generation-related tokens. The use of internal LLM state features to anticipate properties of the generated output represents a promising research direction. Overall, the results reinforce the practical applicability of uncertainty-aware methods in LLM-based EL workflows, particularly in contexts where quality, eficiency, and scalability are critical. Beyond mention-level scores, a natural extension is to target the specific decision tokens (e.g., the span that selects a candidate, or yes/no tokens in classification prompts) to obtain position-aware confidence; for multi-answer outputs, per-span uncertainties could further support selective review and partial acceptance.
The work has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreements No. 101189771 (DataPACT) and No. 101070284 (enRichMyData), and the Italian PRIN project Discount Quality for Responsible Data Science (202248FWFS).
During the preparation of this work, the authors used ChatGPT (GPT-4-turbo) in order to perform grammar and spelling checks. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. model, ACM SIGMOD Record 53 (2024) 44–53. [20] R. Avogadro, M. Ciavotta, F. De Paoli, M. Palmonari, D. Roman, Estimating link confidence for human-in-the-loop table annotation, in: 2023 IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), IEEE, 2023, pp. 142–149. [21] Y. Huang, J. Song, Z. Wang, S. Zhao, H. Chen, F. Juefei-Xu, L. Ma, Look before you leap: An exploratory study of uncertainty analysis for large language models, IEEE Transactions on Software Engineering 51 (2025) 413–429. doi:10.1109/TSE.2024.3519464. [22] Z. Lin, S. Trivedi, J. Sun, Generating with confidence: Uncertainty quantification for black-box large language models, Transactions on Machine Learning Research (2023). [23] H. Ma, J. Chen, G. Wang, C. Zhang, Estimating LLM uncertainty with logits, 2025. URL: https://arxiv.org/abs/2502.00290. arXiv:2502.00290. [24] B. Plaut, N. X. Khanh, T. Trinh, Probabilities of chat LLMs are miscalibrated but still predict correctness on multiple-choice Q&A, 2025. URL: https://arxiv.org/abs/2402.13213. arXiv:2402.13213. [25] A. Vazhentsev, L. Rvanova, I. Lazichny, A. Panchenko, M. Panov, T. Baldwin, A. Shelmanov, Token-level density-based uncertainty quantification methods for eliciting truthfulness of large language models, in: Proceedings of NAACL 2025, 2025, pp. 2246–2262. [26] L. Liu, Y. Pan, X. Li, G. Chen, Uncertainty estimation and quantification for LLMs: A simple supervised approach, 2024. URL: https://openreview.net/forum?id=g3aGMMFHW0. [27] E. Fadeeva, R. Vashurin, A. Tsvigun, A. Vazhentsev, S. Petrakov, K. Fedyanin, D. Vasilev, E. Goncharova, A. Panchenko, M. Panov, et al., LM-Polygraph: Uncertainty estimation for language models, arXiv preprint arXiv:2311.07383 (2023). [28] B. Lakshminarayanan, A. Pritzel, C. Blundell, Simple and scalable predictive uncertainty estimation using deep ensembles, Advances in neural information processing systems 30 (2017). [29] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting, The Journal of Machine Learning Research 15 (2014) 1929–1958. [30] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. E. Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. Clark, N. Joseph, B. Mann, S. McCandlish, C. Olah, J. Kaplan, Language models (mostly) know what they know, CoRR (2022). [31] S. Lin, J. Hilton, O. Evans, Teaching models to express their uncertainty in words, arXiv preprint arXiv:2205.14334 (2022). [32] M. Xiong, Z. Hu, X. Lu, Y. Li, J. Fu, J. He, B. Hooi, Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs, arXiv preprint arXiv:2306.13063 (2023). [33] Y. Yang, H. Li, Y. Wang, Y. Wang, Improving the reliability of large language models by leveraging uncertainty-aware in-context learning, arXiv preprint arXiv:2310.04782 (2023). arXiv:2310.04782. [34] Y. A. Yadkori, I. Kuzborskij, A. György, C. Szepesvári, To believe or not to believe your LLM, arXiv preprint arXiv:2406.02543 (2024). [35] L. Breiman, Random forests, Machine Learning 45 (2001) 5–32. [36] T. Chen, C. Guestrin, XGBoost: A scalable tree boosting system, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD’16, Association for Computing Machinery, New York, NY, USA, 2016, p. 785–794. URL: https://doi.org/10.1145/2939672.2939785. doi:10.1145/2939672.2939785. [37] nostalgebraist, Interpreting GPT: The logit lens, https://www.lesswrong.com/posts/AcKRB8wD pdaN6v6ru/interpreting-gpt-the-logit-lens, 2020. LessWrong, 2020. [38] R. Avogadro, M. Cremaschi, F. D’Adda, F. De Paoli, M. Palmonari, et al., Lamapi: a comprehensive tool for string-based entity retrieval with type-base filters., in: OM@ ISWC, 2022, pp. 25–36. [39] K. Hajian-Tilaki, Receiver operating characteristic (roc) curve analysis for medical diagnostic test evaluation, Caspian journal of internal medicine 4 (2013) 627. [40] E. S. Keeping, Introduction to statistical inference, Courier Corporation, 1995.
The example below illustrates the approach to resolve a mention in a table to its corresponding entity, using an LLM. A high-level instruction, describing the entity linking task, is provided. A question contains the mention to be linked, together with a list of entity candidates extracted from a retriever, e.g., Wikidata Lookup Service or LamAPI. The candidates are provided in the following format: <label [DESC] description [TYPES] type1, type2, ..., typeK>. A snapshot of the input table is provided, i.e., N rows above and below the row containing the mention, in Markdown format. The LLM selects a candidate from the predefined list and outputs it verbatim as an answer.
Instruction: This is an entity linking task. The goal for this task is to link the selected entity mention in the table cells to the entity in the knowledge base. You will be given a list of referent entities, with each one composed of an entity name, its description and its type. Please choose the correct one from the referent entity candidates. Input: [TLE] List of high schools in South Dakota. col: |school|type|city|county|mascot| row 0: |Aberdeen High School|Private|Aberdeen|Brown|Knights| row 1: |Agar High School|Public|Agar|Sully| [...] row 35: |Crazy Horse High School|Public|Wanblee|Jackson|Chiefs| row 36: |Crow Creek High School|Public|Stephan|Hyde|Chieftains| row 37: |Custer High School|Public|Custer|Custer|Wildcats| [...] Question: The selected entity mention is: ‘Hyde’. The column name for ‘Hyde’ is ‘county’. The referent candidates are: <Dr. Jekyll and Mr. Hyde [DESC] fictional characters [TYPE] group of fictional characters> <Hyde v Hyde [DESC] landmark case of the English Court of Probate and Divorce [TYPE] legal case> <Hyde County [DESC] county in South Dakota, United States [TYPE] county of South Dakota> <Douglas Hyde [DESC] first President of Ireland (1860-1949) [TYPE] linguist> <Hyde County [DESC] county in North Carolina, United States [TYPE] county of North Carolina> <Hyde [DESC] civil parish in Bedfordshire, UK [TYPE] civil parish> <Hyde Park [DESC] town in Dutchess County, New York, United States [TYPE] town of New York> <Strange Case of Dr Jekyll and Mr Hyde [DESC] novel by R. L. Stevenson [TYPE] literary work> <Hyde Park [DESC] neighborhood in Chicago, Illinois [TYPE] neighborhood> [...] What is the correct referent entity for ‘Hyde’? Postilla: Answer with just a candidate, selected from the provided referent entity candidates list, and nothing else. The selected candidate must be reported verbatim from the list provided as input. Each candidate in the list is enclosed between < and > and reports [DESC] and [TYPE] information.
Answer: <Hyde County [DESC] county in South Dakota, United States [TYPE] county of South Dakota>
We provide representative examples of the strengths and shortcomings of our method, reporting for each example: input table, mention, candidate list, generated answers with their frequencies, baseline, and estimated uncertainties. Recoverable error 1 reports the standard case: a model’s output is uncertain, with borderline accuracy (0.5), and our one-shot prediction successfully highlights such uncertainty. Recoverable error 2 reports a more complex case, in which the same mention is referred to over multiple seasons. As a consequence, the model struggles to fix on one single entity. Again, our method is able to recover this case by predicting an uncertain outcome. We also report an unrecoverable error case, in which there is no answer variability. Our method correctly predicts low answer variability and, consequently, the case is not marked for correction. However, the answer is wrong, and the error cannot be recovered. Notably, the regressor accurately captures uncertainty across all the examples.
Recoverable error 1: high answer variability → low accuracy
Candidate entities with description and type (right and wrong answers)
Observed answers (count):
<Madrid [DESC] capital city of Spain [TYPE] municipality of Spain>
<Madrid [DESC] Spanish Congress Electoral District
[TYPE] electoral district of the Spanish Congress>
<Madrid [DESC] None [TYPE] electoral district>
Predictive Entropy:
Semantic Entropy:
0.649 Predicted using our method (avg): 0.588
0.478 Predicted using our method (avg): 0.421
(
name Agaw Amhara Beja Bilen Gurage Oromo Saho family Cushitic Semitic Cushitic Cushitic Semitic Cushitic Cushitic language region
country Agaw Horn of Africa Amharic Horn of Africa Beja Horn of Africa Bilen Horn of Africa Gurage Horn of Africa Afan Horn of Africa Oromo Saho Horn of Africa ...
Ethiopia. Eritrea Ethiopia Sudan. Eritrea Eritrea Ethiopia Ethiopia. Somalia.
Sudan. Kenya Eritrea. Ethiopia
Candidate entities with description and type (right and wrong answers)
Observed answers (count):
<Bilen people [DESC] ethnic group [TYPE] ethnic group>
(
Predicted using our method (avg): 0.044
Predicted using our method (avg): 0.041
Figure 2 reveals that performance difers depending on whether features are extracted from the Postilla
or Generated tokens. To investigate this further, we designed an experiment that progressively expands a
sliding window over the concatenated Postilla and Generated segments, measuring how information
accumulates as more tokens are included. We evaluate three feature configurations: (
E. Answer Variability as a Function of Temperature D. Spearman Correlation Between Estimated and True PE/SE
To evaluate the variability of answers and recoverable cases, we systematically sweep temperature
values in the range 0.0 ≤ ≤ 2.0 with steps of 0.1 to assess the temperature efect on (
To further assess the practicality and usability of our approach, in this section we derive the time complexity of a Transformer-based architecture, such as the ones used by the models considered in this work. We consider a Transformer with layers, and hidden size. The context length is set to , while the number of generated tokens is . The time complexity of a single forward pass over a prompt ∈ R× can be decomposed into the following components: • Self-attention: Given the , , ∈ R× matrices, the time complexity of the self-attention mechanism is ( 2 · ), where is the context length and is the hidden size. The attention scores are computed as , and the output is computed as ( ) . The time complexity of this operation is ( 2 · ). • Feed-forward: During the feed-forward step, the time complexity is (8 · 2) = ( · 2) overall, where is the hidden size. In this we can include also the projection of the input to the space and the final projection to the output space.
In total one has a time complexity of ( 2 · + · 2) for a single Transformer layer, which becomes ([ 2 · + · 2]), where is the number of layers.
If we now suppose to generate tokens without the use of a KV-cache, the time complexity of the -th generation step is ([( + )2 · + ( + ) · 2]), for every ∈ []. If we then sum over all the generations, we have:
⎛ ⎞ ⎝∑︁ [︀ ( + )2 + ( + )2]︀ ⎠ =
=1 ⎛ ⎞ ⎝ ∑︁ [︀ ( + )2 + ( + )2]︀ ⎠ =
=1 ⎛ ⎞ ⎝ ∑︁ [︀ ( 2 + 2 + 2) + ( + )2]︀ ⎠ =
=1 (︀ [︀ ( 2 + 2 2 + 3) + ( + 2)2]︀)
=
(︀ [︀ ( + )2 + ( + )2]︀) =
(︀ [︀ ( + )2 + ( + )2]︀)
(
When a KV-cache is used, while the time for processing the prompt is the same, a major computational saving is obtained during the generation phase. In this case, the time complexity of the -th generation step can be decomposed into the following components: • Self-attention: During the self-attention, reduces to a single vector ∈ R1× , while , becomes , ∈ R(+− 1)× . Overall, the time complexity of this operation is (( + ) · ).
If we then sum over all the generations ∈ [], we have:
=1 which shows that the time complexity of generating tokens is linear in the number of overall tokens + , when ≪ , otherwise it would become quadratic in the number of generated ones.