Entity Matching (EM) is a fundamental task in data management, involving the identification and linking of records that refer to the same real-world entity across diferent datasets. While Large Language Models (LLMs) have shown promise in addressing complex natural language processing tasks, their substantial computational requirements often limit their practical applicability. In this work, we investigate the use of 7B parameter LLMs with 4-bit quantization for EM tasks executable on commodity hardware. We explore various prompting strategies, including zero-shot, few-shot, and general matching definition prompts, to evaluate their efectiveness in improving EM accuracy. Experiments are conducted on two benchmark datasets with products, which present varying levels of complexity and challenge in product descriptions. Our findings demonstrate that 7B parameter LLMs can efectively perform EM, with the Orca2 model consistently outperforming others across diferent prompting strategies and datasets. The study highlights that few-shot prompting significantly enhances performance over zero-shot approaches, emphasizing the importance of task-specific examples and careful prompt design. We also examine the impact of example order in few-shot prompts and find that it has a substantial efect on model performance. Finally, we examine hardware limitations, demonstrating that efective EM can be achieved with resource-constrained models.
Entity Resolution (ER) constitutes a vital task in data
management that involves identifying and linking records from
diferent datasets that refer to the same real-world entity
[
As an example, consider the product descriptions in
Figure 1. Despite corresponding to the same object (Sony
headphones), there are significant variations in product names,
attributes, and dimensions. These discrepancies illustrate
the challenges in reconciling variations across datasets,
particularly when dealing with unstructured text and linguistic
diferences. Accurate ER in scenarios like this is crucial for
product catalog integration, price comparison, and
recommendation systems [
Due to its quadratic time complexity, ER solutions
typically implement the Filtering-Verification framework [
Traditional EM solutions typically rely on rule-based
approaches, string similarity metrics, or machine learning
algorithms [
Recent advancements in NLP, particularly in Large
Language Models (LLMs), ofer new possibilities for addressing
EM challenges [
In this work, we evaluate the performance of 7B
parameter LLMs in entity matching tasks. While larger LLMs
with hundreds of billions of parameters have shown
impressive results [
To this end, we perform an extensive experimental evaluation that considers the models’ ability to handle diferent types of EM scenarios. We explore novel zero-shot, fewshot, and general matching definition prompting strategies to assess their efectiveness in improving matching accuracy. Our goal is to bridge the gap between the advanced capabilities of LLMs and the practical constraints of real-world EM applications, potentially paving the way for more eficient and accurate ER techniques in diverse domains.
There is a plethora of recent LLM-based EM methods, because LLMs ofer several advantages over traditional EM solutions: (i) contextual understanding, as they understand the context and semantics of entity descriptions better than traditional string matching techniques. (ii) robustness, since LLMs are typically more capable of addressing variations in how entity information is expressed. (iii) zero-shot and few-shot learning, i.e., LLMs can accomplish EM tasks with no or minimal examples of matching decisions. These characteristics render LLMs ideal for most EM tasks, especially those with complex, unstructured product descriptions.
The seminal work on LLM-based EM [
A detailed study was conducted in [
In another line of research, three distinct prompting
strategies were explored in [
BatchER [
These studies collectively demonstrate the potential of LLMs in entity matching tasks, highlighting the importance of prompt engineering, the competitiveness of open-source models, and the efectiveness of batching strategies for improved eficiency. This work builds upon and extends the existing ones by focusing specifically on 7B parameter LLMs with 4-bit quantization. Unlike previous studies that primarily use larger, more resource-intensive models, our work explores the potential of smaller and more accessible LLMs for EM tasks. In this context, we perform a comprehensive evaluation of various novel prompting strategies, including zero-shot, few-shot, and general matching definition approaches, across multiple models and datasets. This approach ofers insights into the practical applicability of LLMs in resource-constrained environments, bridging the gap between advanced language models and real-world EM challenges.
Applied after Filtering, Entity Matching is typically
formulated as a binary classification problem [
In LLM-based settings, EM is framed as a natural language inference task. The LLM is provided with descriptions of two records and asked to determine if they refer to the same entity, returning “True” for a match and “False” otherwise.
In all cases, EM performance is measured with respect to:
• Precision, i.e., the proportion of correctly identified
matches out of all predicted matches.
• Recall, i.e., the proportion of correctly identiefid matches
out of all actual matches.
• F-measure, i.e., the harmonic mean of precision and recall,
providing a balanced measure of performance.
• Run-time, i.e., the time taken to complete the ER process.
The first three measures are defined in [
We now present the EM prompts that are examined in our work. The basic prompt is presented in Figure 2(a). It consists of an instruction that describes the input and the desired output. It lacks any examples, thus constitutes a zeroshot EM prompt, which tests the model’s ability to generalize to new tasks or domains it has not been trained on.
A concise few-shot EM prompt extends the zero-shot one with the examples in Figure 2(b). To provide a balanced context, there are two examples that include a pair of matching entities and a pair of non-matching ones. These examples serve as a form of weak supervision, allowing the LLM to learn from the provided instances and generalize to similar cases. Note that the examples in Figure 2(b) have been carefully selected from dataset 1 (see Table 1) so that they capture typical variations in product descriptions that are encountered in the full dataset.
Note that LLM responses to few-shot prompts sufer from
position bias [
To increase the robustness of LLMs to few-shot EM prompts, we consider two matching approaches for each candidate pair, query with both TF and FT prompts: 1. The union approach labels a candidate pair as True if either the TF or FT prompt results in a True response. 2. The intersection approach labels a candidate pair as True only if both the TF and FT prompts yield a True response.
The above prompts are generic enough to apply to any domain. In our experimental analysis, we also consider domain-specific ones, which are crafted for the product matching task. More specifically, we devise a zero-shot prompt that involves general matching definitions, providing the LLM with explicit guidance on how to determine if two records refer to the same product.
The core assumption of this approach is that the records are described by a clean, aligned schema. This is necessary for building a schema-aware generic definition of duplicate records. In the product matching task, we use four key product attributes: (i) product name, (ii) features, (iii) manufacturer, and (iv) model number. We use them in two diferent configurations: 1. The composite domain-specific EM prompt concatenates all four criteria in the above sequence, as in Figure 3. The goal is to facilitate more nuanced matching decisions. 2. The atomic domain-specific EM prompt uses only the model number as the matching criterion. We selected this attribute because it provides the cleanest and most distinctive values.
These two configurations were chosen after preliminary tests that suggested that they yield the best performance among all other combinations of these four attributes.
Experimental Settings. All experiments were implemented in Python v3.12.0 and Ollama1 v0.1.22. All experiments were carried out on a server running Ubuntu 22.04.1 LTS, equipped with Intel Core i7-9700K 8 core @ 3.6 GHz, 32GB RAM and NVIDIA GeForce GTX 1080 Ti 11GB.
Due to the limited size of the available VRAM, our study focuses on 7-billion-parameter LLMs with optimizations such as quantization, which in our case replaces the 32bit floating-point model weights with 4-bit integers. This reduces the model size, while maintaining reasonable performance levels. In other words, quantization lowers efectiveness, due to the fewer parameters and the lower precision of the model’s weights, but significantly reduces run-times and memory consumption. Therefore, our experimental results are useful for resource-constrained applications, which run LLMs on commodity hardware.
LLMs. There is a plethora of open-source LLMs, with
newer models introduced on a rather frequent basis. During
our study, two models were quite popular: Llama 2 [
In their place, we considered the following open-source
models, which demonstrated high efectiveness in our
preliminary experiments:
1. Orca2 [
In all cases, we use the default latest model with 4-bit quantization and 7B parameters.
Datasets. We used two real-world datasets with products that are widely used in the ER literature: (i) 1 is the AbtBuy dataset, which comprises product listings from two online retailers, Abt Electronics and Buy.com. (ii) 2 is the Walmart-Amazon dataset, which contains product listings from two other online retailers, Walmart and Amazon. 1 primarily focuses on electronic products, while 2 covers a broader range of product categories, matching diverse entity types. Both datasets present important challenges, such variations in product names and descriptions across retailers, inconsistent use of model numbers and other identifiers, diferences in the level of detail provided for each product, variations in formatting and units (e.g., dimensions, weights) as well as missing or null values in certain fields.
Their technical characteristics are summarized in Table
1. Note that each dataset comprises two individually clean
data sources, whose sizes are reported in column
#Entities. Note also that we apply the prompts to the candidate
pairs generated by a state-of-the-art blocking implemented
3https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca
4https://huggingface.co/stabilityai/StableBeluga2
by PyJedAI [
We now examine the relative performance of the selected LLMs over 1 and 2, when coupled with the basic zeroshot EM prompt of Figure 2(a).
We observe that Orca2, OpenHermes, and Zephyr consistently rank as the top three models with respect to FMeasure in both datasets. The last two models switch their ranking positions in the two datasets, whereas Orca2 maintains the lead. The superior performance of Orca2, which demonstrates Orca2’s robustness under diverse EM settings, can be attributed to its fine-tuning on synthetic data designed for reasoning tasks. This enhances its capability to understand and compare complex product descriptions. OpenHermes is fine-tuned on fully open datasets with strong multi-turn chat skills, leveraging advanced language understanding to perform well. Zephyr’s competitive performance probably results from its training on a distilled dataset that improves grammar and chat results, aiding in better interpretation of entity attributes. The lower performance of Mistral-OpenOrca, Stable-Beluga, and Llama-Pro is probably due to the less specialized training data or the smaller model capacities for the specific nuances of EM.
Note that all models exhibit much higher recall than precision in both datasets. This means that they are prone to label a candidate pair as matching, at the cost of introducing numerous false positives. Orca2 consistently exhibits the highest precision, thus yielding the highest F-Measure, too.
Note also that all models exhibit markedly lower efectiveness in 2 compared to 1. This suggests that 2 presents greater EM challenges, potentially due to more diverse or complex product descriptions. While 1 is restricted to electronics, 2 covers a broader range of products and includes more variation in descriptions, attributes, and data quality, rendering EM more dificult. Furthermore, 1 has a 1:1 matching between its two data sources, whereas 2 has a much lower ratio of matches, adding another layer of complexity to the task. The substantial performance gap between 1 and 2 underscores the significant impact of data characteristics on model efectiveness.
We now examine the performance of the aforementioned few-shot prompts over 1 and 2. We disregard MistralOpenOrca, Stable-Beluga, and Llama-Pro, because they exhibited significantly lower efectiveness and less consistent performance in the zero-shot experiments – preliminary experiments verified their poor performance in few-shot settings, too. For brevity, we focus on the top three performing models, namely Orca2, OpenHermes, and Zephyr.
The results are reported in Figure 5. Based on preliminary experiments, we randomly select the examples included in the few-shot prompts from the candidate pairs of the same dataset. The same examples are used in all prompts issued on a particular dataset.
In both datasets, we observe the same patterns as regards the relative performance of TF and FT few-shot prompts: For Orca2, there is a substantial improvement when using the latter; OpenHermes is more robust to position bias, as there is no significant diference between the two prompt strategies; Zephyr works best when coupled with the TF few-shot prompts. These patterns highlight that the impact of position bias on each model is consistent across the two datasets. Note also that with the exception of Orca2 with TF prompts, all models achieve higher recall than precision, remaining more prone to label a candidate pair as matching.
It is also interesting to compare the union approach with the intersection one. For OpenHermes and Zephyr, the latter yields significantly higher F-Measure: by considering as duplicates only the candidate pairs that are marked as matching by both TF and FT few-shot prompts, the reduction in recall is much lower than the increase in precision (as a result, recall remains much higher than precision for both models). This means that considering only the common matches of TF and FT prompts leads to more accurate performance. Note that these patterns are consistent for both models over both datasets.
This is not the case with Orca2, whose performance varies significantly across the two datasets. In 1, the same F1 score is achieved for both approaches, because the intersection raises recall by 12%, while reducing precision to the same degree. In 2, though, the intersection reduces recall by 23% and increases precision by 16%, thus yielding a much lower F-Measure. Note that in both datasets, the recall of the model gets lower than its precision in combination with the intersection approach, unlike the union one.
Overall, we can conclude that Orca2 works best when coupled with FT few-shot prompts, while OpenHermes and Zephyr maximize their efectiveness when intersecting the matches of TF and FT prompts. Among them, the top performers over 1 and 2 are Orca2 (F1=0.799) and Zephyr (F1=0.531), respectively.
In this section, we compare the atomic domain-specific prompt with the composite one. As in Section 5.2, we exclusively consider the three top performing models with respect to the zero-shot prompts: Orca2, OpenHermes, and Zephyr. Their performance is reported in Figure 6.
We observe that in all cases, the atomic prompt outperforms the composite one to a significant extent – the only exception corresponds to Zephyr in 1, where the composite prompt increases F-Measure almost by 15%. This pattern should be attributed to the short, distinctive and clean values provided by the model number. This way, it reduces the noise from other product attributes like product name, which are typically associated with long and diverse texts.
Similar to the above strategies, all LLMs exhibit much higher recall than precision. This means that they remain prone to mark a candidate pair as a match at the cost of introducing false positives – a behavior that permeates all prompt strategies we have examined.
Among the three models, Orca2 is consistently better, albeit to a minor extent in 2. This consistent performance underscores Orca2’s efectiveness in EM tasks under quite diferent prompt designs.
We can conclude that domain-specific zero-shot prompts ofer an efective and reliable alternative in datasets with a clean schema of known characteristics.
Prompt Strategy Zero-shot FT Few-shot Atomic Domain-specific Zero-shot Intersection Few-shot Atomic Domain-specific Zero-shot Intersection Few-shot Composite Domain-specific Precision
We now compare the three top-performing models (Orca2, OpenHermes, and Zephyr) with respect to efectiveness and time eficiency across the three strategies of EM prompts discussed in Section 4. Note that among the few-shot and domain-specific variants, for each LLM we only consider the one with the highest F-Measure in both datasets. Their performance is reported in Table 2.
For Orca2, we observe that the FT few-shot prompts are the top performers in 1. The atomic domain-specific ones follow in very close distance in terms of F-Measure, while exhibiting a much lower run-time. This means that the domain-specific prompts ofer a significantly better balance between efectiveness and time eficiency. In 2, this strategy scores the highest F-Measure for a slightly higher runtime than the second best approach (zero-shot prompts). For these reasons, Orca2 works best in combination with the atomic domain-specific prompts.
Regarding OpenHermes, the diferences between the three types of prompts are minor in terms of F-Measure. As expected, the fastest approach in both datasets corresponds to the zero-shot prompts. This configuration also achieves the highest F-Measure in 1, while in 2, it ranks second, within a negligible distance from the top (<0.5%). Therefore, we can conclude that the zero-shot prompts are the best choice for OpenHermes.
For Zephyr, there is a clear winner in the case of 1: the intersection of few-shot prompts. It exhibits, though, the highest run-time by a large extent. This is expected, as it queries the LLM twice per candidate pair. In the case of 2, the same strategy takes a minor lead over the composite Method ZeroER
Magellan DeepMatcher 1
To put the performance of the selected 7B LLMs into
perspective, we compare it with three state-of-the-art EM
approaches from the literature:
1. ZeroER [
For each method, we consider its best performance as reported in the literature. The results are reported in Table 1.
We observe mixed patterns. In 1, all LLM configurations in Table 2, even the zero-shot prompts, outperform all three baseline methods to a significant extent ( > 21%). This is remarkable, because the simplest prompt strategy requires neither domain expertise nor the labeling candidate pairs, unlike Magellan and DeepMatcher, whose performance is derived from large training and validation sets, which amount to 60% and 20% of all candidate pairs, resp.
The situation is reversed in 2, where all baseline methods achieve a much better performance. In fact, the highest F-measure of Orca2 is lower by 16.5% than the worst baseline (ZeroER). This should be attributed to the more challenging settings of 2, which have already been discussed in Section 5.1. Note also that the records in 2 are noisier, with a much higher portion of missing values. Its records are also longer, an aspect that is crucial for the 7B LLMs we are considering in this study, due to their limited attention window. These settings favor the learning-based functionality of the baseline methods, which take a clear lead over the learning-free functionality of 7B LLMs. Another reason for the poor performance of the latter is that they emphasize recall at the expense of precision, significantly decreasing their F-Measure in 2, due the very low portion of matches in comparison to the total number of entities from each data source. Therefore, more advanced strategies are required for boosting the performance of 7B LLMs in datasets with characteristics similar to that of 2.
Focusing on 7B open-source LLMs, we examined the performance of three main prompt strategies: (i) the basic, domain-agnostic zero-shot prompt, (ii) the few-shot prompt with one example per type of matches, and (iii) the domainspecific zero-shot prompt. We considered several variants for the last two strategies and applied all of them on two established benchmark datasets for product matching. Testing six popular LLMs, we reached the following conclusions: • Few-shot and domain-specific prompting significantly improve the performance of the zero-shot approaches, highlighting the value of task-specific prompts. • In few-shot prompts, the response of LLMs is generally sensitive to order of examples. This suggests that a careful prompt engineering is crucial for optimal performance in real-world ER applications. • This sensitivity can be addressed by the intersection approach to few-shot prompting, which consistently achieves much better results, increasing precision at a higher rate than it reduces recall. • Orca2 consistently outperformed the other LLMs across most prompting strategies and datasets, demonstrating high robustness and efectiveness. In fact, the relative performance of the best models (Orca2 > OpenHermes > Zephyr) remained largely consistent across prompt strategies and datasets, suggesting inherent strengths in their base architectures. • The use of 4-bit quantization and 7B parameter models demonstrated the potential for efective EM with limited computational resources. The efectiveness of the considered models is competitive with established, learningbased EM approaches, especially in datasets with low portion of missing values and short entity descriptions.
In the future, we plan to explore LLMs’ capability in matching entities across diferent languages and to enhance the interpretability and explainability of LLM decisions. Acknowledgments. This work was partially funded by the EU project STELAR (Horizon Europe – Grant No. 101070122).