The amount of available data has been growing significantly over the last decades. Thus, linking entries across heterogeneous data sources such as databases or knowledge bases, becomes an increasingly difficult problem, in particular w.r.t. the runtime of these tasks. Consequently, it is of utmost importance to provide time-efficient approaches for similarity joins in the Web of Data. While a number of scalable approaches have been developed for various measures, the Most Frequent k Characters (MFKC) measure has not been tackled in previous works. We hence present a sequence of filters that allow discarding comparisons when executing bounded similarity computations without losing recall. Therewith, we can reduce the runtime of bounded similarity computations by approximately 70%. Our experiments with a single-threaded, a parallel and a GPU implementation of our filters suggest that our approach scales well even when dealing with millions of potential comparisons.
The problem of managing heterogeneity at both the semantic and syntactic levels among various information resources [12,10] is one of the most difficult problems on the information age. This is substantiated by most of the database research self-assessment reports, which acknowledge that the hard question of semantic heterogeneity, that is of handling variations in meaning or ambiguity in entity interpretation, remains open [10]. In knowledge bases, Ontology Matching (OM) solutions address the semantic heterogeneity problem in two steps: (1) matching entities to determine an alignment, i.e., a set of correspondences, and (2) interpreting an alignment according to application needs, such as data translation or query answering. Record Linkage (RL) and, more recently, Link Discovery 1 (LD) solutions on the other hand aim to determine pairs of entries that abide by a given relation R. In both cases, string similarities are used to compute The rest of the paper is structured as follows: In Section 2 related work is presented, where we focus on approaches that aim to improve the time-efficiency of the link discovery task. In Section 3, we present our nested filters, followed by the Section 4 with the Correctness and Completeness. Section 5 with the evaluation. In Section 6, we conclude.
Our approach can be considered an extension of the state-of-the-art algorithm introduced in [8], which describes a string-based distance function (SDF) based on string hashing [9,7]. The naive approach of MFKC [8] is a metric for string comparison built on a hash function, which gets a string and outputs the most frequent two characters with their frequencies. This algorithm was used for text mining operations. The approach can be divided into two parts: (1) The hashing function is applied to both input strings, where the output is a string that contains the two most frequent characters; the first and third elements keep the characters and second and fourth elements keep the frequency of these characters.
(2) The hashes are compared, where will return a real number between 0 and lim. By default lim = 10, since the probability of having ten occurrences of the two most frequent characters in common between two strings is low. If the output of the function is 10, this case indicates that there is no common character and any value below 10 means there are some common characters shared among the strings.
Our work is similar to the one presented in [13], which features a parallel processing framework for string similarity using filters to avoid unnecessary comparisons. Among the several types of string similarity, emerging works have been done for measures such as Levenshtein-distance [6], which is a string distance function that calculates the minimum number of edit operations (i.e., delete, insert or update) to transform the first into the second string. The Jaccard Index [5], also called Jaccard coefficient, works on the bitwise operators, where the strings are treated at bit level. REEDED [11] was the first approach for the time-efficient execution of weighted edit distances.
Let us call N aiveM F KC the function which computes the MFKC algorithm as described in [8]. Such function works with three parameters, i.e. two strings s and t and an integer lim and returns the sum of frequencies, where f(c i , s) is a function that returns the frequency of the character c i in the string s and s ⊇ {c 1 , ..., c n }, i.e f(a, "andrea") = 2, because the character a has been found twice and the hash functions h(s) and h(t) containing the characters and their frequencies. The output of function is always positive, as shown in Equation (1).
Our work aims to reduce the runtime of computation of the MFKC similarity function. Here, we use a sequence of filters, which allow discarding similarity computations and imply in a reduction of runtime. As input, the algorithm receives datasets D s and D t , an integer number representing the k most frequent characters and a threshold θ ∈ [0, 1]. The similarity score of the pair of strings from the Cartesian product from D s and D t must have a score greater or equal the threshold θ to be considered a good pair, i.e. for a given threshold θ, if the similarity function has a pair of strings with similarity score less than the threshold, σ(s, t) < θ, we can discard the computation of the MFKC score for this pair. Our final result is a set which contains the pairs having similarity score greater than or equal to the threshold, i.e. σ(s, t) ≥ θ.
Our work studies the following problem: Given a threshold θ ∈ [0, 1] and two sets of strings D s and D t , compute the set M = {(s, t, σ(s, t)) ∈ D s × D t × R + : σ(s, t) ≥ θ}. Two categories of approaches can be considered to improve the runtime of measures: Lossy approaches return a subset M of M which can be calculated efficiently but for which there are no guarantees that M = M . Lossless approaches, on the other hand, ensure that their result set M is exactly the same as M . In this paper, we present a lossless approach that targets the MFKC algorithm. Equation (2) shows our definition for the string similarity function σ for the MFKC.
where s and t are strings, such that s, t ∈ * , f(c i , s) is a function that returns the frequency of the character c i in the string s, where s ⊇ {c 1 , ..., c n }, k represents the limitation of the elements that belongs to the hashes; set h(s, k)∩h(t, k) means the intersection between the keys of hashes h(s, k) and h(t, k) (i.e., the most frequent K characters). We expect two steps to obtain the similarity score:
1. Firstly, we transform the strings s and t in two hashes using Most Frequent
Character Hashing [8], according to the following example with k = 3:
We calculate the sum of the character frequencies of matching characters on the hashes h(s, k) and h(t, k), then, we normalize dividing by the sum of the length of |s| and |t| resulting in a similarity score from 0 to 1 according to the Equation ( 3) and the resulting score should be greater or equals the threshold θ.
In this section, the runtime of MFKC defined in Equation ( 2) is improved using filters where N is the output of first frequency filter, L is the output of hash intersection filter and A represents the output of the k similarity filter.
First Frequency Filter As specified in the definition of MFKC [8] this filter assumes that the hashes are already sorted in an descending way according to the frequencies of characters, therefore the first element of each hash has the highest frequency.
Theorem 1. Showing that:
implies that σ(s, t) < θ.
Proof (Theorem 1). Let the intersection between hashes h(t, k) and h(s, k) be a set of characters from c 1 to c n , such that Equation (5):
According to the definition of the frequencies f(c i , t) we have Equation ( 6):
where each c i appears f(c i , t) times, therefore:
Also, as n ≤ k, because t ⊇ {c 1 , ..., c n }, and f(c i , s) ≤ h 1 (s, k) ∀ i=1,...,n , then:
Therefore, from Equation (7) and Equation ( 8), we obtain the Equation ( 9):
Consequently, the rule which the filter relies on is the following.
Hash Intersection Filter In this filter, we check if the intersection between two hashes is an empty set, then the MFKC, represented by σ, will return a similarity score of 0 and we can avoid the computation of similarity in this case. Consequently, the rule which the filter relies on is the following.
we also can say that the Equation ( 12) represents a valid implication.
The Equation (12) means that if the intersection between h(s, k) and h(t, k) is a empty set, this implies that the similarity score will be 0. That means there is no character matching, then there is no need to compute the similarity for this pair of strings.
K Similarity filter For all the pairs left, the similarity score among them is calculated. After that, the third filter selects the pairs whose similarity score is greater or equal than a threshold θ.
This filter provides a validation and we show that the score of previous k similarity is always lower than the next k, according to the Equation ( 14) and in some cases when the similarity score is been reached before compute all elements ∈ h(s, k) ∩ h(t, k), thus saving computation in these cases.
Here k is also used as a index of similarity function σ k (s, t) in order to get the similarity of all cases of k, from 1 to k, also to show the monotonicity.
Therefore we can say that the computation of similarity score occurs until σ k (s, t) ≥ θ.
We will demonstrate that the similarity score of previous k similarity is always lower than the next k similarity, for all k ∈ Z * : k ≤ |s ∩ t|.
We rewrote the equation for the first iteration, according to Equation (15)
Let k ∈ Z + be given and suppose the equation Equation ( 14) is true for k +1.
Therefore, we can notice that the sum of frequencies will be always greater or equal 0, according to f(c k+1 , s) + f(c k+1 , t) ≥ 0. Thus, Equation (14) holds true.
Filter sequence The sequence of the filters occurs basically in 4 steps, (1) We starting to make the Cartesian product with the pairs of strings from the datasets D s and D t , (2) Discarding pairs using the F irst F requency F ilter(N ), (3) Discarding pairs where there is no matching characters with the Hash Intersection f ilter(L) and ( 4) With the Most Frequent Character Filter (A) we will process only similarities greater or equal the threshold θ, if the similarity function σ(s, t) ≥ θ in the first k characters we can stop the computation of the similarity of this pair, saving computation and add to our dataset with resulting pairs Dr, also shown in the Algorithm 1. for all s ∈ Ds do (in parallel) In this section, we prove formally that our MFKC is both correct and complete.
-We say that an approach is correct if the output O it returns is such that O ⊆ R(D s , D t , σ, θ). -Approaches are said to be complete if their output O is a superset of R(D s , D t , σ, θ), i.e., O ⊇ R(D s , D t , σ, θ).
Our MFKC consists of three nested filters, each of which creates a subset of pairs, i.e. A ⊆ L ⊆ N ⊆ D s × D t . For the purpose of clearness, we name each filtering rule:
Each subset of our MFKC can be redefined as
We then introduce A * as the set of pairs whose similarity score is more or equal than the threshold θ.
Theorem 2. Our MFKC filtering algorithm is correct and complete.
Proof (Theorem 2). Proving Theorem 2 is equivalent to showing that A = A * . Let us consider all the pairs in A. While our MFKC's correctness follows directly from the definition of A, it is complete iff none of pairs discarded by the filters actually belongs to A * . Assuming that the hashes are sorted in a descending way according the frequencies of the characters, therefore the first element of each hash has the highest frequency. Therefore, once we have
the pair of strings s and t can be discarded without calculating the entire similarity. When rule R 3 applies, we have σ(s, t) < θ, which leads to R 3 ⇒ R 1 . Thus, set A can be rewritten as:
We are given two strings s and t and the respective hashes h(s, k) and h(t, k), the intersection between the characters of these two hashes is a empty set. Therefore, there is no character matching, which implies that s and t cannot be considered to have a similarity score greater than or equal to threshold θ:
When the rule R 3 applies, we have σ(s, t) < θ, which leads to R 3 ⇒ R 2 . Thus, set A can be rewritten as:
which is the definition of A * in Equation ( 17). Therefore, A = A * .
In order to calculate the time complexity of our MFKC, firstly we considered the most frequent K characters from a string. The first step is to sort the string lexically. Then, we can reach a linear complexity after this sort, because the input with highest occurrences can be achieved with a linear time complexity. The first string can be sorted in O(nlogn) and second string in O(mlogm) times, as some classical sorting algorithms such as merge sort [3] and quick sort [4] that work in O(nlogn) complexity. Thus, the total complexity is O(nlogn) + O(mlogm), resulting in O(nlogn) as upper bound in the worst case.
The aim of our evaluation is to show that our work outperforms the naive approach and the parallel implementation has a performance gain in large datasets with size greater than 10 5 pairs. A considerable number of pairs reach the threshold θ before reaching the last k most frequent character, that also is a demonstration about how much computation was avoided. Instead of all k's we just need k − n where n is the n th most frequent character necessary to reach the threshold θ. An example to show the efficiency of each filter can be found at Figures 1 and 1(a), where 10,273,950 comparisons from DBpedia+LinkedGeoData were performed and Performance Gain (P G) = Recall(N ) + Recall(L). The recall2 can be seen in Figure 2(c). This evaluation has the intention to show results of experiments on data from DBpedia3 and LinkedGeoData4 . We considered pairs of labels in order to do the evaluation. We have two motivations to chose these datasets: (1) they have been widely used in experiments pertaining to Link Discovery (2) the distributions of string sizes between these datasets are significantly different [1]. All runtime and scalability experiments were performed on a Intel Core i7 machine with 8GB RAM, a video card NVIDIA NVS4200 and running Ms Windows 10.
Our algorithm contains parallel code snippets with which we perform a load and balance of the data among CPU/GPU cores when available. To illustrate this part of our idea, we can state: Given a two datasets S, T , that contains all the strings to be compared. Thus, make a Cartesian product of the strings S × T , where each pair is the processed separately in threads that are spread among CPU/GPU cores. Thus, we process the each comparison in parallel. The parallel implementation works better in large datasets with size more than 10 5 , that was more than one time faster than the approach without parallelism and two times faster than the naive approach as shown in Figures 3(a) to 3(c).
The evaluation in Figures 3(a) and 3(b) shows that all filter setup outperform the naive approach, and the parallel approach does not suffer significant changes related to the runtime according to the size of the dataset, as show Figure 3(c).
The experiments related to the variance of k, also were considered, as show in Figure 3
In the experiments (see Figures 3(c), 3(e) and 3(f)), we looked at the growth of the runtime of our approach on datasets of growing sizes. The results show that the combination of filters (N +L+A) is the best option for datasets of large sizes. This result holds on both DBpedia and LinkedGeoData, so our approach can be used on large datasets and achieves acceptable run-times. We also can realize the quantity of avoided pairs in each combination of filters in Figure 1, that consequently brings a performance gain. We looked at experiments with runtime behavior on a large dataset with more than 10 6 labels as shown in Figure 3 increment. Thus, one more point showing that our approach is useful on large datasets, where can be used with high threshold values for link discovery area. About our parallel implementation, Figure 3(c) shows that our GPU parallel implementation works better on large datasets with size greater than 105 .
Our work overcomes the naive approach [8], thus, in order to show some important points we compare our work not only with the state of the art, but with popular algorithms such as Jaccard Index [5]. As shown in Figures 3(c), 3(e) and 3(f), our approach outperforms not only the naive approach, but also Jaccard Index. We show that the threshold θ and k have a significant influence related to the runtime. The naive approach present some points to consider, among them, even if the naive approach states that they did experiments with k = 7, the naive algorithm was designed for only k = 2, there are some cases where k = 2 is not enough to get the similarity level expected, i.e. s = mystring1 and t = mystring2 limiting k = 2, we will have σ 2 (s, t) = 0.2, showing that the similarity is very low, but when k = 8, the similarity is σ 8 (s, t) = 0.8 showing that sometimes we can lose a good similarity case limiting k = 2. Our work fix all these problems and also has a better runtime, as show Figure 3(a), Figure 3(b) and Figure 3(c). An experiment with labels from DBpedia and Yago 5 shows that the f-score indicates a significant potential to be used with success as a string similarity comparing with Jaccard Index and Jaro Winkler, as Figures 2(a) to 2(c) shows.
To summarize the key features that makes our approach outperform the naive approach are the following: We use more than two K most frequent characters in our evaluation, our run-time for more than 10 discard comparisons avoiding extra processing, and a parallel implementation, making our approach scalable. Jaccard does not show significant changes varying the threshold, MFKC and Jaro Winkler present a very similar increase of the f-score varying the threshold.
We presented an approach to reduce the computation runtime of similarity joins using the Most Frequent k Characters algorithm with a sequence of filters that allow discarding pairs before computing their actual similarity, thus reducing the runtime of computation. We proved that our approach is both correct and complete. The evaluation shows that all filter setup outperform the naive approach. Our parallel implementation works better in larger datasets with size greater than 10 5 pairs. It is also the key to developing systems for Record Linkage and Link Discovery in knowledge bases. As future work, we plan to integrate it in link discovery applications for the validation of equivalence links. The source code is free and available online 6 . Acknowledgments available on footnotes7 .