This paper presents a contribution to better interpretation of the results we get from GenAI models, more specifically, better interpretation of the mistakes that they make. We have conducted an analysis of 644 (from GPT-4o) + 4858 (from ARIA) mistakes made by two models on a key-value extraction task, and found that they may be categorised into three mutually exclusive groups. These groups are; i problems identifying the requested information, p problems presenting the correct information, and s skewed training data. These categories could be used to indicate which action a user could take to reduce the number of mistakes. Further, we have found a strong correlation between the suggested categories and the Ratclif/Obershelp pattern recognition score between the generated result and the expected result; all faulty results containing minor mistakes are more than 60% similar to the expected result. Only mistakes based on lack of identifying what was requested had less than 60% similarity to the expected result.
CEUR
ceur-ws.org A
Categorisation of LLM
While there are many papers detailing the accuracy of large language models’ (LLMs’) knowledge
of particular topics, or forms of reasoning, we are looking closer at the diferent ways that LLMs
are making mistakes, often called “hallucination”. While hallucinations are mentioned much
in media and AI research, the focus is mainly on avoiding them. One dimensional accuracy
measurements are giving some indication of how well models are presenting results at particular
tasks. The lack of discussion on what form the mistakes are taking in most of those papers are
leaving us with little insight into how one might reach higher accuracy results. For example,
shall we do better prompting, use better models, change the line of questioning, how one might
avoid the wrong results [
This experiment had the aim of identifying patterns in the mistakes made by GenAI models with vision capabilities, to better interpret if a model is the right fit for a certain task. This is relevant to the currently emerging paradigm, where there are GenAI models of varying formats https://lnu.se/personal/nemi.pelgrom/ (N. Pelgrom); https://grahn.cse.bth.se/ (H. Grahn)
© 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). and aims, and the main question is no longer what model structure is the best one, but rather “What model is the best for my particular task”.
We have conducted previous experiments on transcription tasks which had indicated
possibilities to systematically categorise the mistakes made [
The experiment is using the two multimodal generative models GPT-4o and ARIA to complete a transcription task on a dataset of 3000 images. This provides us with an environment where it is immediately clear if the model is understanding the requested task, and where it is easy to automatically separate the correct answers from the incorrect ones, as opposed to pure text questions which often require some qualitative interpretation. Further, we are requesting transcriptions of both numbers and of natural language strings separately, allowing us to identify possible diferences in how they are interpreted by the models.
The development of vision augmented LLM models has gone fast. The first ones were made
easily available just a year ago [
Vision models generally have a three-part structure; one for interpreting the image, one for
interpreting text, and one that combines the end result in a fitting way [
There are also other ways to add additional training-data to a pre-trained model. These
include Retrieval Augmented Generation (RAG) [
A recent addition to the set of GenAI architecture is mixture-of-experts solutions. These aim
at minimising the efort used for any particular task, by allowing for separating the models
into several parts, where some parts may be ignored when they are deemed irrelevant for the
intended task [
With these great progresses, there are still some shortcomings [
This section contains all the details of the conducted experiment, the results are presented in the next section.
The dataset consists of 3000 images of real receipts collected from a wide range of purchases in Sweden. The images are scans or photos containing both full view of receipts, and in most cases some additional background such as hands, tables, and knees. Most of the receipts have wrinkles and are not fully flat, which makes them harder to read, and keeps them representative of receipt scans that are uploaded to receipt-reading services.
These images provide a complex task of identifying the correct information requested in the
prompt, extracting it correctly, and then presenting it in the way specified by the prompt. If
any one of these steps go wrong, the end result will be faulty. This makes it very impressive
that some of these models are able to reach high accuracy results [
So from this raw dataset, we created the data used in our experiment; we ran each image
through each of the two models, and created separate JSON files containing the response of
each reading, for each of the model. The same prompt was used for both models. GPT-4o was
accessed through an API call, and ARIA was run locally on a Nvidia A100 GPU. These models
were chosen for being the best available, and respectively the best available to run locally, in
regards to OCR tasks [
This gave us three datasets for our comparisons: 3000 JSON files with GPT-4o transcriptions, 3000 JSON files with ARIA transcriptions, and 3000 JSON files containing the key for each image.
We made python code that compared a selection of keys from each file: date, total amount to pay, VAT amount, company name, and organisation number. We chose these keys to include in this experiment because they are present in most of the images, so they are more representative of a standard receipt than for example ”tips” which are only present on a small minority of the images. When we originally included more of these keys in the comparison, it meant we had to do much more data cleaning. We decided a smaller dataset is preferable to a more complex cleaning step, to ensure the replicability of our results.
The prompt that was used was developed by Fortnox AB, with support from Microsoft. It is extracting most of the information that could be of book-keeping interest from each receipt. While this means that a large amount of keys have been extracted from each image, we choose to do our analysis on only 5 keys, so that we only looked at what is available on the majority of the images. Including keys that exist on fewer of the images would require more time spent on the data cleaning, without increasing the amount of useful mistakes in a proportional way.
We included TotalAmount as a representation of a free-format number. We included date, and
VAT as a representations of a number to be extracted in a specific format. OrganisationNumber
as a representation of longer number strings (which are known to be dificult for GenAI models
[
Your final output must satisfy the following typescript schema: Valid VAT rates type VatRate = "0%" | "12%" | "25%" | "6%"; Valid payment methods type PaymentMethod = "" | "card" | "gift card" | "mobile" | "swish"; Valid currency types type Currency = "SEK" | "DKK" | "NOK" | "EUR" | "USD" | "GBP" | "JPY" | "AUD" | "CHF" ↪ | "CAD" | "CNY" | "SGD" | "KRW" | "PLN" | "INR" | "HUF" | "other"; Return the response in a JSON-format that satisfies the above typescript type for ↪ Receipt.
Only use the types from above nothing else.
Only output pure json.
Do your best, think step by step.
We have two datasets of 15000 comparisons (the amount of files multiplied with the amount of keys we chose for the comparison) in the format of excel sheets.
For each of them we conduct the following steps:
An additional 488 rows were removed during the annotation process. These were comparisons that were not identifiable earlier in the process as irrelevant to our study, but which could have skewed our results if they have been left in the final dataset. All of these rows were ones where we deemed it inaccurate to count the transcription as wrong, despite it not matching the key. Here are some examples of these instances: • The key stating ”Company Name”, and the model’s result is stating ”Company Name AB”.
Both answers are present on the receipt and may therefore be seen as correct. • The key stating ”179.90” and the transcription stating ”180”, and the image stating that the total amount is 180, but that the card was charged 179.90. Both answers are therefore present on the receipt as a total amount, and may be seen as correct.
Once the set of transcriptions was ready, we annotated each row with one of the following: • i: the mistake is not correctly identifying what information is requested • s: the mistake is skewed data, presenting a word or number that is clearly not a misreading of what is present on the image, but instead a word or number from the training data that the model interprets as equivalent • p: when the correct information is identified, and mostly correctly presented, we have annotated with p for presentation, where the mistake is a presentation problem. Wrong spellings are included here. Reading i as 1, and 8 as 3, are included here.
Category s could be understood as relating to the category ”confabulations” as introduced
in [
Once all rows were annotated, we had noticed a pattern of mistakes in the p category being
close to the correct replies with only a diference of one or two symbols (e.g. misspellings or
reading a 7 as a 1). This is partly per definition, but indicated a possibility of automatically
identifying which category a particular mistake belongs to. Based on this, we ran the document
through Python code that added a Ratclif/Obershelp pattern calculation [
Levenshtein distance, however, did not show any useful correlation with the categories. While it correctly identifies long strings as being close to the correct answer when there are only spelling mistakes, it does not account for length of the compared strings, which makes fully wrong replies score equally good as slightly wrong ones, when the compared strings are short.
Jaccard similarity had some predictive value; lower similarity correlates with category i, and higher with category p, however there is no cut-of point between them, and the distribution is broad. Further, Jaccard similarity is based on only the characters that are present, which means that it calculates two strings as equal, if they contain the same characters, even if the characters are not in the same order. This means that it equates 660 with 600, which is not ideal.
Ratclif/Obershelp pattern matching, also called Gestalt pattern matching, is accounting
for the characters present, the lengths of the strings, and the order of the characters in the
strings. This makes it a useful algorithm for identifying what kind of mistake a generative
model has made, by simply calculating the pattern matching score. And such an estimate may
be performed automatically, making it possible to identify the character of the mistakes a model
makes for a task without any further manual or otherwise qualitative interpretation of faulty
responses from models. When we sorted the rows according to their similarity score, we found
a very strong cut-of point, where there are no instances of i mistakes above it, and no instances
of p mistakes below it. This correlation will be shown in detail in the results section. While this
is here shown to be true in the context of our experiment, we have not yet tested the possibilities
of generalising this to other application areas. It is possible that this correlation holds true
for other tasks where it is possible to systematically identify one unique correct answer in
relation to a prompt. The category s (hallucinations according to the most common usage of it
in literature on Chatbots; ”fictional or erroneous information” [
We present two primary findings from this experiment. First, we identified a consistent and practical categorization scheme for all transcription mistakes made by GenAI models with vision-to-text capabilities, in scenarios where a structured key is available for comparison. These categories—i (identification), p (presentation), and s (skewed training data)—are broadly applicable across all types of errors observed in our experiments, which involved thousands of receipt images.
As shown in Table 1, both ARIA and GPT-4o models produced mistakes that fell across
all three categories, and the distributions of these mistakes are not uniform. ARIA made
significantly more total mistakes (4,858 vs. 644), and the majority of its mistakes (4,124) were
classified as identification mistakes. GPT-4o, by contrast, exhibited a more balanced distribution,
with 403 presentation, and 138 identification mistakes. This uneven distribution suggests that
these errors are not random and may reflect inherent tendencies in how each model handles
uncertainty or incomplete information. While it is well known that GPT-4o avoid giving blank
answers to the degree of producing factual contradictions [
This disparity highlights that these errors are not random; rather, they reflect consistent patterns tied to model behaviour. Table 2 quantifies the diference, using GPT-4o as a baseline: ARIA made 2,888% more identification mistakes, 50% more presentation mistakes, and 25% more skewed data mistakes. The striking increase in i mistakes from ARIA suggests a conservative extraction approach—opting to leave fields blank when uncertain.
Field-level analysis provides more insight. Table 3 breaks down the number and type of
mistakes by data key. ARIA’s highest concentration of identification errors occurred with the
Merchant Name and Organisation Number fields—1,619 and 1,847 respectively. For these, the
model frequently failed to return any value, even when the information was clearly visibly
present in the image. Table 4 supports this: ARIA’s empty response rate was 91% for Merchant
Name and 98% for Organisation Number, versus 40% and 9% for GPT-4o, respectively. These
high omission rates indicate ARIA’s tendency to skip uncertain fields entirely, possibly due to a
narrower confidence threshold or the model being overwhelmed by the amount of information
that was requested of it [
The significant diference in identifying Organisation Numbers may also be explained by the structural properties of receipts and the training-data of the models. For example, Swedish Organisation Numbers are often present but not explicitly labelled as such in the receipts in our dataset. Models not specifically trained to recognize these patterns, such as ARIA, are more prone to fail on identification, especially if they rely on keyword cues. GPT-4o, by contrast, appears more likely to attempt a response even when uncertain, which explains its higher relative rate of presentation mistakes, where the correct value is approximated but not matched exactly to the key.
Additionally, the Merchant Name field revealed another nuance. According to Table 5, this ifeld was the most afected by cleaning in both model outputs. GPT-4o had a 21% retention rate post-cleaning, while ARIA retained 86%. This diference is partly explained by ARIAs tendency to give blank answers. But it brings our attention to another issue with automatic accuracy estimations; instances where there are multiple correct answers. All of the results that were removed in this data-cleaning had the issue of having multiple accurate results. They could have been included in our experiment if the keys were of a format that allowed several answers to be understood as correct.
The second major finding is the strong correlation between Ratclif/Obershelp similarity scores and error category. Table 6 shows that all identification mistakes i had similarity scores below 60%, while presentation mistakes p had scores of 60% or higher. Skewed data mistakes s showed a mixed pattern: most were above 60%, but a small percentage (10% for GPT-4o, 5% for ARIA) fell between 35–60%. This suggests that Ratclif/Obershelp similarity can be used as a heuristic for error classification, high similarity likely indicates p or s errors, while low similarity indicates i errors. The exact threshold may vary depending on text length and structure of the information that is extracted, but the trend holds across both models and all keys that we tested.
Further analysis of skewed data errors reveals model-specific behaviour. ARIA exhibits a high rate of s errors in the Date field (28%), suggesting a tendency to hallucinate plausible but incorrect values when uncertain. In contrast, GPT-4o shows more skewed mistakes in the VAT field (5%), but overall keeps skewed data rates low across most keys (Table 4). These patterns reinforce the idea that ARIA is conservative—risking omissions—while GPT-4o aims for completeness, sometimes at the expense of accuracy.
By combining our categorization with Ratclif/Obershelp similarity metrics, we provide a replicable framework for analysing transcription performance.
We made an experiment focused on identifying categories of mistakes made by GenAI models, with the intention of finding patterns that could help clarify what models may or may not be useful for particular tasks, further than a simple accuracy estimate does. We found two useful patterns; that there are three systematically identifiable categories of mistakes that reoccur across diferent models, and that there is a simple way to automatically sort mistakes into at least two of these categories. This paper reports the details of how the experiment was conducted, and we suggest that further research should be done on the possibilities of generalising these ifndings, and on identifying the reasons for why these mistakes occur.
We want to thank Fortnox AB for providing the dataset, prompt, and financial support for this project, and we also thank the reviewers for their helpful contributions.
During the preparation of this work, the author(s) used GPT-4o for drafting the structure of the paper. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.