Entity resolution (ER) is of great importance in many real-world applications, including information retrieval, web search, natural language processing, web information integration, multi-source data analysis, etc. In this paper, we present an extensible block scheme-based method for entity resolution. Specifically, at first we preprocess data records from multiple datasets, so as to remove ambiguity and maintain data attributes that are helpful for entity resolution. Then, the processed records are assigned to diferent blocks, after which similarity between records within a same block is computed. Experiment results on challenge datasets from DI2KG show that our proposed method is promising.
Entity Resolution (ER) is to match data records from two or more
data sources, by analyzing their contents that describe identical
entities in real-world [
In this paper, we focus on the problem of matching products
recorded in electronic commerce website. Performing the entity
matching task on an e-commerce database is challenging, because
of 1) product data is highly heterogeneous; 2) structure of the data
is loosely defined; 3) there are multiple data sources (for example,
in the DI2KG challenge we need to process diferent data sources
from diferent stores). One of the critical missions of ER on large
data sets is how to reduce the computational complexity to
improve eficiency. Currently, there are three prominent solutions, i.e.,
Blocking [
The paper is organized as follows. In Section 1 we give an introduction of entity resolution and the challenge task in DI2KG. We present our framework for entity resolution in Section 2, including the three main components of our framework. In Section 3, we conduct experiments on the challenge datasets to evaluate the efectiveness of our method. Finally, in Section 4 we summarize this paper and give our future work on entity resolution. 2
Our method is based on block scheme and scheme aggregation. As shown in Figure 1, our extensible entity resolution system framework mainly includes three submodules, i.e., Preprocess, Block Scheme and Scheme Aggregation (BS-SA), Similarity and Clustering. In this section, we described three sub-modules in detail.
In data preprocessing step, we remove some less important terms from page titles, and less informative properties from the product description text. Specifically, from table 1 we can see that there are diferences between product profiles. For example, the first product possesses some property feature while the second product contains neither this property nor the corresponding value. We also observe that for the same products, their model names are the same. So, we try to find model names for entity matching. Before finding the model name, there are two problems to deal with, as given below: • Which property contains the model name • How to deal with the situation that the extracted model words cannot be extracted from the predetermined key value or the extracted model names
To solve the above problems, at first some preprocessing steps are necessary, such as lowercasing strings and removing some special symbols. Generally, the model names have at least one number and one non-numeric character. Hence, we use regular expression to match the model name. We find that the model name will be obtained precisely in the properties, like model, model name and so on. And we retain these properties as new descriptions of product. When one is unable to find these properties, we consider property ’<page title>’, because there is part of information which need to be removed, such as resolution and screen size in the ’<page title>’. On the other hand, we consider the situation that the model name is separated by spaces, so we will remove the space between nonnumeric characters and numeric characters to create a model word (i.e. the model name is something like ’P 230’, we will change it to ’p230’). 2.2
After data preprocessing step, the processed data will be passed to
submodule Block Scheme and Scheme Aggregation, through which
each product is assigned to a block. Referring to Figure 1, this
submodule consists of two parts: first, the products are classified
based on the brand list that obtained from the Internet [
In this challenge, the model word is defined as a word that contains at least one number and one non-numeric character, and the entire word is completely extracted as one model word. We adopt two block schemes, i.e., extracting a single model word for title and description, and then aggregating the two schemes by the union aggregator. Eventually, all documents of each brand will be mapped to multiple product entities and each document will only be mapped to one product entity. 2.3
The method we proposed in this paper for computing entity similarity is based on model words and set judgment. The definition of model word is as described in Section 2.2, and we compare the model words set of product and the model words set of product , which measures the similarity of two products. Our proposed similarity function consists of two parts.
The first part evaluates whether the models of product and are the same, by using function (, , ). Here, is a list we constructed from product descriptions, which consists of attributes where the model may appear. We count the model words extracted from the attributes in the list , and select the model word that appears the most frequently as the product model. Moreover, if there is no any model word, then we take the first model word from ’<page title>’ as the model. The rationale is that the product model in the title usually appears first.
The second part determines whether synonymous attributes of product and product are similar, by calling the function (, , , ) multiple times. Here, parameter = [1, , ], where ∈ contains synonymous attributes, and = [1, , ], where operator ∈ is either ′ ′ or ′ ′, which is applied to . Each time (, , , ) is invoked, for the model word set extracted from all the attributes of the product in list , we determine the relationship between two sets, according to the content of operator . Specifically, if is ′ ′, then the two sets must be the same, otherwise one set must be the subset of the other set. For example, a parameter configuration is shown below: (, , [′< >′,′ ′,′ ′],′ ′) Algorithm 1 shows a high-level overview of how to compute product similarity. Specifically, if the model names of the two products are the same, the algorithm will compare whether certain attributes of the product are similar according to the parameter and .
Input: Product profile and ; List = [1, , ], where ∈ contains some synonyms attributes; List = [1, , ], where operator ∈ is either ′ ′ or ′ ′, which corresponds to ; Attributes list ; Furthermore, the following functions are used:
- (, ) is true if models of product and are diferent
- (, , , ) the model words similarity between the products and using the key list and operator . Output: True if product and product are the same; otherwise,
False. 1: function ProdSim(, , , , ) 2: if (, , ) then return False 3: end if 4: for all , in , do 5: if not (, , , ) then return False 6: end if 7: end for 8: return True 9: end function
Algorithm 2 judges whether the two products have the same model. It regards the most frequently occurring model word or the ifrst model word in the title as the model name, and compares the model names of the two products.
Function (, , , ) in Algorithm 3 is used to compute the similarity between product description of and , according to some special product attributes. The number of times this function is called depends on the length of the parameter and . In this algorithm, the model words from some attributes of product and are compared. If is ‘ ′, the model words must be the same, otherwise, one model word set must be the subset of the other set.
Due to transitivity property between products, if A is similar to B, and B is similar to C, then we regard it that A is similar to C. We perform clustering while calculating the similarity, instead of clustering after calculating the similarity between all products. This may sacrifice accuracy a little bit, but can significantly improve eficiency. 3
To evaluate performance of our proposed method, we conduct experiments on the challenge datasets provided by DI2KG. We summarize the best experiment results of our proposed method in Table 3, i.e., on training Dataset Y the Precision, Recall and F-Measure are 1.000, 0.964 and 0.982, respectively, whereas on Dataset X Precision, Recall and F-Measure are 0.915, 0.967 and 0.940, respectively.
In addition, we tested our algorithm on the camera dataset and
the Precision, Recall and F-Measure are 0.98, 0.97 and 0.98 [
- (, ) extracts the model words from the key-value pair of product a if the key appears in list .
- () find the model word appear most frequently in the model word list .
- () true if is empty.
Output: True if the models of product and product are diferent; otherwise, False. 1: function diffModel(, , ) 2: = (, ) 3: = (, ) 4: = (, [′<page title>′]) 5: = (, [′<page title>′]) 6: if ( ) then 7: if [0] in then return False 8: end if 9: = [0] 10: else 11: = ( ) 12: end if 13: if ( ) then 14: if [0] in then return False 15: end if 16: = [0] 17: else 18: = ( ) 19: end if 20: return not ( == ) 21: end function
Input: Product profile and ; Synonyms attributes list ; Operator is either ′ ′ or ′ ′, which corresponds to ; ( ) extracts the model words for a given set of words .
Output: True if the model words of product and extracted from are the same under operator ; otherwise, False. 1: function mwSim(, , , ) 2: , = (, ), (, ) 3: if ==′ ′ then 4: if . ( ) and . () then 5: return True 6: end if 7: end if 8: if ==′ ′ then 9: if . ( ) or . () then 10: return True 11: end if 12: end if 13: return False 14: end function on both data sets, that is, it has good extensivity. In order to be suitable for diferent datasets, we can set the parameters such as brand list, product descriptions list and synchronous attributes list, which can be obtained based on rules or machine learning. It is worth mentioning that our algorithm is mainly based on model words, thus, it is better for data sets that mainly rely on product models for matching, such as cameras, displays and other electronic products.
Our observation is that the key to data preprocessing is to remove some redundant information. Since in Algorithm 2, we try to use the first model word in ’<page title>’ for comparison, it is highly recommended to make the first model word in the ’<page title>’ to be the model number of the product.
On the other hand, in the block scheme the acquisition of brand information requires some manual intervention. Although it can be obtained through machine learning or rules-based methods, we found that there are some special circumstances that need to deal with. For instance, Alienware was acquired by Dell, so Alienware and Dell belong to the same brand. Since brand classification is the first step in the BS-SA, manual intervention for such special situations can improve the recall rate at lower cost. 4
In this paper, we proposed an extensible block-scheme based method for entity resolution. Specifically, our method consists of three components, i.e., data preprocessing, block-scheme and scheme aggregation, and similarity and clustering. Experiments on DI2KG challenge datasets show that our method can achieve 0.982 and 0.94 F-measure values on Dataset X and Dataset Y, respectively.
In our future work, we intend to combine machine learning
techniques and consider weighting key properties, so as to find as
many as possible the best <key, value> pairs of products. Because
by using those strategies, we can make full use of training datasets,
by mapping product similarity to the range [
The authors would like to thank the reviewers for their comments. This paper is partially supported by NSFC under grant Number 61972177.