=Paper=
{{Paper
|id=Vol-2032/om2017_Tpaper1
|storemode=property
|title=A high-performance approach to string similarity using most frequent K characters
|pdfUrl=https://ceur-ws.org/Vol-2032/om2017_Tpaper1.pdf
|volume=Vol-2032
|authors=Andre Valdestilhas,Tommaso Soru,Axel-Cyrille Ngonga Ngomo
|dblpUrl=https://dblp.org/rec/conf/semweb/ValdestilhasSN17
}}
==A high-performance approach to string similarity using most frequent K characters==
https://ceur-ws.org/Vol-2032/om2017_Tpaper1.pdf
A High-Performance Approach to String
Similarity using Most Frequent K Characters
Andre Valdestilhas, Tommaso Soru, and Axel-Cyrille Ngonga Ngomo
AKSW/DICE, University of Leipzig, Germany
{valdestilhas,tsoru,ngonga}@informatik.uni-leipzig.de
Abstract. The amount of available data has been growing significantly
over the last decades. Thus, linking entries across heterogeneous data
sources such as databases or knowledge bases, becomes an increasingly
difficult problem, in particular w.r.t. the runtime of these tasks. Conse-
quently, it is of utmost importance to provide time-efficient approaches
for similarity joins in the Web of Data. While a number of scalable ap-
proaches have been developed for various measures, the Most Frequent
k Characters (MFKC) measure has not been tackled in previous works.
We hence present a sequence of filters that allow discarding comparisons
when executing bounded similarity computations without losing recall.
Therewith, we can reduce the runtime of bounded similarity computa-
tions by approximately 70%. Our experiments with a single-threaded,
a parallel and a GPU implementation of our filters suggest that our
approach scales well even when dealing with millions of potential com-
parisons.
Keywords: Similarity Search; Blocking; String Matching
1 Introduction
The problem of managing heterogeneity at both the semantic and syntactic levels
among various information resources [12,10] is one of the most difficult problems
on the information age. This is substantiated by most of the database research
self-assessment reports, which acknowledge that the hard question of semantic
heterogeneity, that is of handling variations in meaning or ambiguity in entity
interpretation, remains open [10]. In knowledge bases, Ontology Matching (OM)
solutions address the semantic heterogeneity problem in two steps: (1) matching
entities to determine an alignment, i.e., a set of correspondences, and (2) inter-
preting an alignment according to application needs, such as data translation
or query answering. Record Linkage (RL) and, more recently, Link Discovery1
(LD) solutions on the other hand aim to determine pairs of entries that abide
by a given relation R. In both cases, string similarities are used to compute
1
The expression "link discovery" in this paper means the discovery of typed relations
that link instances from knowledge bases on the Web of Data. We never use it in
the sense of graph theory.
candidates for alignments. In addition to being central to RL and LD, these
similarities also play a key role in several other tasks such as data translation,
ontology merging and navigation on the Web of Data [10,2].
One of the core tasks when developing time-efficient RL and LD solutions
hence lies in the development of time-efficient string similarities. In this paper, we
study the MFKC similarity function [8] and present an approach for improving
the performance of similarity joins. To this end, we develop a series of filters
which guarantee that particular pairs of resources do not abide by their respective
similarity threshold by virtue of their properties.
The contributions of this paper are as follows:
1. We present two nested filters, (1) First Frequency Filter and (2) Hash Inter-
section filter, that allow to discard candidates before calculating the actual
similarity value, thus giving a considerable performance gain.
2. We present the k similarity filter that allows detecting whether two strings
s and t are similar in a fewer number of steps.
3. We evaluate our approach with respect to its runtime and its scalability with
several threshold settings and dataset sizes.
4. We present several parallel implementations of our approach and show that
they work well on problems where |Ds × Dt | ≥ 105 pairs.
The rest of the paper is structured as follows: In Section 2 related work is
presented, where we focus on approaches that aim to improve the time-efficiency
of the link discovery task. In Section 3, we present our nested filters, followed
by the Section 4 with the Correctness and Completeness. Section 5 with the
evaluation. In Section 6, we conclude.
2 State of the Art and related work
Our approach can be considered an extension of the state-of-the-art algorithm
introduced in [8], which describes a string-based distance function (SDF) based
on string hashing [9,7]. The naive approach of MFKC [8] is a metric for string
comparison built on a hash function, which gets a string and outputs the most
frequent two characters with their frequencies. This algorithm was used for text
mining operations. The approach can be divided into two parts: (1) The hashing
function is applied to both input strings, where the output is a string that
contains the two most frequent characters; the first and third elements keep the
characters and second and fourth elements keep the frequency of these characters.
(2) The hashes are compared, where will return a real number between 0 and
lim. By default lim = 10, since the probability of having ten occurrences of the
two most frequent characters in common between two strings is low. If the output
of the function is 10, this case indicates that there is no common character and
any value below 10 means there are some common characters shared among the
strings.
Our work is similar to the one presented in [13], which features a parallel
processing framework for string similarity using filters to avoid unnecessary com-
parisons. Among the several types of string similarity, emerging works have been
done for measures such as Levenshtein-distance [6], which is a string distance
function that calculates the minimum number of edit operations (i.e., delete,
insert or update) to transform the first into the second string. The Jaccard In-
dex [5], also called Jaccard coefficient, works on the bitwise operators, where
the strings are treated at bit level. REEDED [11] was the first approach for the
time-efficient execution of weighted edit distances.
3 Approach
Let us call N aiveM F KC the function which computes the MFKC algorithm
as described in [8]. Such function works with three parameters, i.e. two strings
s and t and an integer lim and returns the sum of frequencies, where f(ci , s)
is a function that returns the frequency of the character ci in the string s and
s ⊇ {c1 , ..., cn }, i.e f(a, ”andrea”) = 2, because the character a has been found
twice and the hash functions h(s) and h(t) containing the characters and their
frequencies. The output of function is always positive, as shown in Equation (1).
2
X
N aiveM F KC(s, t, lim) = lim − f (ci , s) + f (ci , t) (1)
ci ∈h(s)∩h(t)
Our work aims to reduce the runtime of computation of the MFKC similarity
function. Here, we use a sequence of filters, which allow discarding similarity
computations and imply in a reduction of runtime. As input, the algorithm
receives datasets Ds and Dt , an integer number representing the k most frequent
characters and a threshold θ ∈ [0, 1]. The similarity score of the pair of strings
from the Cartesian product from Ds and Dt must have a score greater or equal
the threshold θ to be considered a good pair, i.e. for a given threshold θ, if
the similarity function has a pair of strings with similarity score less than the
threshold, σ(s, t) < θ, we can discard the computation of the MFKC score for
this pair. Our final result is a set which contains the pairs having similarity score
greater than or equal to the threshold, i.e. σ(s, t) ≥ θ.
Our work studies the following problem: Given a threshold θ ∈ [0, 1] and two
sets of strings Ds and Dt , compute the set M 0 = {(s, t, σ(s, t)) ∈ Ds × Dt × R+ :
σ(s, t) ≥ θ}. Two categories of approaches can be considered to improve the
runtime of measures: Lossy approaches return a subset M 00 of M 0 which can
be calculated efficiently but for which there are no guarantees that M 00 = M 0 .
Lossless approaches, on the other hand, ensure that their result set M 00 is exactly
the same as M 0 . In this paper, we present a lossless approach that targets the
MFKC algorithm. Equation (2) shows our definition for the string similarity
function σ for the MFKC.
P
ci ∈h(s,k)∩h(t,k) f(ci , s) + f(ci , t)
σ(s, t) = (2)
|s| + |t|
P∗
where s and t are strings, such that s, t ∈ , f(ci , s) is a function that returns
the frequency of the character ci in the string s, where s ⊇ {c1 , ..., cn }, k repre-
sents the limitation of the elements that belongs to the hashes; set h(s, k)∩h(t, k)
means the intersection between the keys of hashes h(s, k) and h(t, k) (i.e., the
most frequent K characters). We expect two steps to obtain the similarity score:
1. Firstly, we transform the strings s and t in two hashes using Most Frequent
Character Hashing [8], according to the following example with k = 3:
s = aabbbcc → h(s, k) = {b = 3, a = 2, c = 2}
t = bbccddee → h(t, k) = {b = 2, c = 2, d = 2}
2. We calculate the sum of the character frequencies of matching characters
on the hashes h(s, k) and h(t, k), then, we normalize dividing by the sum of
the length of |s| and |t| resulting in a similarity score from 0 to 1 according
to the Equation (3) and the resulting score should be greater or equals the
threshold θ.
P
ci ∈h(s,k)∩h(t,k) f(ci , s) + f(ci , t)
σ(s, t, k, θ) = ≥θ (3)
|s| + |t|
3.1 Improving the Runtime
In this section, the runtime of MFKC defined in Equation (2) is improved using
filters where N is the output of first frequency filter, L is the output of hash
intersection filter and A represents the output of the k similarity filter.
First Frequency Filter As specified in the definition of MFKC [8] this filter
assumes that the hashes are already sorted in an descending way according to
the frequencies of characters, therefore the first element of each hash has the
highest frequency.
Theorem 1. Showing that:
P
ci ∈h(s,k)∩h(t,k) f(ci , s) + f(ci , t) h1 (s, k)k + |t|
σ(s, t) = ≤ (4)
|s| + |t| |s| + |t|
implies that σ(s, t) < θ.
Proof (Theorem 1). Let the intersection between hashes h(t, k) and h(s, k) be a
set of characters from c1 to cn , such that Equation (5):
h(t, k) ∩ h(s, k) = {c1 , ..., cn } (5)
According to the definition of the frequencies f(ci , t) we have Equation (6):
t ⊇ {c1 , ..., c1 , ..., cn , ...cn } (6)
where each ci appears f(ci , t) times, therefore:
f(c1 , t) + ... + f(cn , t) ≤ |t| (7)
Also, as n ≤ k, because t ⊇ {c1 , ..., cn }, and f(ci , s) ≤ h1 (s, k) ∀i=1,...,n , then:
f(c1 , s) + ... + f(cn , s) ≤ h1 (s, k) + ... + h1 (s, k) = n(k) ≤ k(h1 (s, k)) (8)
Therefore, from Equation (7) and Equation (8), we obtain the Equation (9):
P Pn Pn
ci ∈h(s,k)∩h(t,k) f(ci , s) + f(ci , t) i=1 f(ci , t) + i=1 f(ci , s) h1 (s, k)k + |t|
= ≤
|s| + |t| |s| + |t| |s| + |t|
(9)
Consequently, the rule which the filter relies on is the following.
h1 (s, k)k + |t|
hs, ti ∈
/ N ⇒ hs, ti ∈
/ Ds × Dt ∧ ≤θ (10)
|s| + |t|
Hash Intersection Filter In this filter, we check if the intersection between
two hashes is an empty set, then the MFKC, represented by σ, will return a
similarity score of 0 and we can avoid the computation of similarity in this case.
Consequently, the rule which the filter relies on is the following.
hs, ti ∈ L ⇒ hs, ti ∈ Ds × Dt ∧ |h(s) ∩ h(t)| > 0 (11)
we also can say that the Equation (12) represents a valid implication.
h(s) ∩ h(t) = ∅ ⇒ σ(s, t) = 0 (12)
The Equation (12) means that if the intersection between h(s, k) and h(t, k)
is a empty set, this implies that the similarity score will be 0. That means there
is no character matching, then there is no need to compute the similarity for
this pair of strings.
K Similarity filter For all the pairs left, the similarity score among them is
calculated. After that, the third filter selects the pairs whose similarity score is
greater or equal than a threshold θ.
hs, ti ∈ A ⇔ hs, ti ∈ N ∧ σ(s, t) ≥ θ (13)
This filter provides a validation and we show that the score of previous k
similarity is always lower than the next k, according to the Equation (14) and in
some cases when the similarity score is been reached before compute all elements
∈ h(s, k) ∩ h(t, k), thus saving computation in these cases.
Here k is also used as a index of similarity function σk (s, t) in order to get
the similarity of all cases of k, from 1 to k, also to show the monotonicity.
Therefore we can say that the computation of similarity score occurs until
σk (s, t) ≥ θ.
We will demonstrate that the similarity score of previous k similarity is always
lower than the next k similarity, for all k ∈ Z∗ : k ≤ |s ∩ t|.
σk+1 (s, t) ≥ σk (s, t) (14)
We rewrote the equation for the first iteration, according to Equation (15)
Pk
ci ∈h(s,k)∩h(t,k),i=1 f(ci , s) + f(ci , t)
σk (s, t) = (15)
|s| + |t|
Let k ∈ Z+ be given and suppose the equation Equation (14) is true for k +1.
k
X k
X
f(ci , s) + f(ci , t) + f(ck+1 , s) + f(ck+1 , t) ≥ f(ci , s) + f(ci , t)
ci ∈h(s,k)∩h(t,k) ci ∈h(s,k)∩h(t,k)
(16)
Therefore, we can notice that the sum of frequencies will be always greater
or equal 0, according to f(ck+1 , s) + f(ck+1 , t) ≥ 0. Thus, Equation (14) holds
true.
Filter sequence The sequence of the filters occurs basically in 4 steps, (1) We
starting to make the Cartesian product with the pairs of strings from the datasets
Ds and Dt , (2) Discarding pairs using the F irst F requency F ilter(N ), (3) Dis-
carding pairs where there is no matching characters with the Hash Intersection f ilter(L)
and (4) With the Most Frequent Character Filter (A) we will process only sim-
ilarities greater or equal the threshold θ, if the similarity function σ(s, t) ≥ θ in
the first k characters we can stop the computation of the similarity of this pair,
saving computation and add to our dataset with resulting pairs Dr, also shown
in the Algorithm 1.
Algorithm 1 MFKC Similarity Joins
1: GoodP
P∗ airs = {s1 ; t1 , ..., sn ; tn }|hs, t ∈ 16: for all cf req ∈ hs ∩ ht do
i 17: if i ≥ k then
2: hs = {e1 , e2 , ..., en }|ei = hc, f (c, s)i 18: N ext t ∈ Dt
3: ht = {e1 , e2 , ..., en }|ei = hc, f (c, t)i 19: end if
4: i, f req ∈ N∗ 20: f req = f req + cf req
f req
5: procedure MFKC(Ds , Dt , θ, k) 21: σi = |s|+|t|
6: for all s ∈ Ds do (in parallel) 22: if σi ≥ θ then
7: hs = h(s, k) 23: GoodP airs.add(s, t)
8: for all t ∈ Dt do (in parallel) 24: N ext t ∈ Dt
9: ht = h(t, k) 25: end if
10: if hs|s|+|t|
1 (k)+|t|
< θ then 26: i=i+1
11: N ext t ∈ Dt 27: end for
12: end if 28: end for
13: if |hs ∩ ht| = 0 then 29: end for
14: N ext t ∈ Dt 30: return GoodP airs
15: end if 31: end procedure
4 Correctness and Completeness
In this section, we prove formally that our MFKC is both correct and complete.
– We say that an approach is correct if the output O it returns is such that
O ⊆ R(Ds , Dt , σ, θ).
– Approaches are said to be complete if their output O is a superset of R(Ds , Dt , σ, θ),
i.e., O ⊇ R(Ds , Dt , σ, θ).
Our MFKC consists of three nested filters, each of which creates a subset of
pairs, i.e. A ⊆ L ⊆ N ⊆ Ds × Dt . For the purpose of clearness, we name each
filtering rule:
h1 (s, k)k + |t|
R1 , <θ
|s| + |t|
R2 , |h(s, k) ∩ h(t, k)| =
6 0
R3 , σ(s, t) ≥ θ
Each subset of our MFKC can be redefined as N = {hs, ti ∈ / Ds × Dt : R1 ,
L = {hs, ti ∈ Ds × Dt : R1 ∧ R2 , and A = {hs, ti ∈ Ds × Dt : R1 ∧ R2 ∧ R3 .
We then introduce A∗ as the set of pairs whose similarity score is more or equal
than the threshold θ.
A∗ = {hs, ti ∈ Ds × Dt : σ(s, t) ≥ θ} = {hs, ti ∈ Ds × Dt : R3 } (17)
Theorem 2. Our MFKC filtering algorithm is correct and complete.
Proof (Theorem 2). Proving Theorem 2 is equivalent to showing that A = A∗ .
Let us consider all the pairs in A. While our MFKC’s correctness follows directly
from the definition of A, it is complete iff none of pairs discarded by the filters
actually belongs to A∗ . Assuming that the hashes are sorted in a descending way
according the frequencies of the characters, therefore the first element of each
hash has the highest frequency. Therefore, once we have
h1 (s, k)k + |t|
< θ,
|s| + |t|
the pair of strings s and t can be discarded without calculating the entire simi-
larity. When rule R3 applies, we have σ(s, t) < θ, which leads to R3 ⇒ R1 . Thus,
set A can be rewritten as:
A = {hs, ti ∈ Ds × Dt : R2 ∧ R3 } (18)
We are given two strings s and t and the respective hashes h(s, k) and h(t, k), the
intersection between the characters of these two hashes is a empty set. Therefore,
there is no character matching, which implies that s and t cannot be considered
to have a similarity score greater than or equal to threshold θ:
h(s, k) ∩ h(t, k) = ∅ ⇒ σ(s, t) = 0
When the rule R3 applies, we have σ(s, t) < θ, which leads to R3 ⇒ R2 . Thus,
set A can be rewritten as:
A = {hs, ti ∈ Ds × Dt : R3 } (19)
which is the definition of A∗ in Equation (17). Therefore, A = A∗ .
Time complexity In order to calculate the time complexity of our MFKC,
firstly we considered the most frequent K characters from a string. The first step
is to sort the string lexically. Then, we can reach a linear complexity after this
sort, because the input with highest occurrences can be achieved with a linear
time complexity. The first string can be sorted in O(nlogn) and second string in
O(mlogm) times, as some classical sorting algorithms such as merge sort [3] and
quick sort [4] that work in O(nlogn) complexity. Thus, the total complexity is
O(nlogn) + O(mlogm), resulting in O(nlogn) as upper bound in the worst case.
5 Evaluation
The aim of our evaluation is to show that our work outperforms the naive ap-
proach and the parallel implementation has a performance gain in large datasets
with size greater than 105 pairs. A considerable number of pairs reach the thresh-
old θ before reaching the last k most frequent character, that also is a demonstra-
tion about how much computation was avoided. Instead of all k’s we just need
k − n where n is the nth most frequent character necessary to reach the thresh-
old θ. An example to show the efficiency of each filter can be found at Figures 1
and 1(a), where 10,273,950 comparisons from DBpedia+LinkedGeoData were
performed and Performance Gain (P G) = Recall(N ) + Recall(L). The recall2
can be seen in Figure 2(c). This evaluation has the intention to show results
of experiments on data from DBpedia3 and LinkedGeoData4 . We considered
pairs of labels in order to do the evaluation. We have two motivations to chose
these datasets: (1) they have been widely used in experiments pertaining to Link
Discovery (2) the distributions of string sizes between these datasets are signif-
icantly different [1]. All runtime and scalability experiments were performed on
a Intel Core i7 machine with 8GB RAM, a video card NVIDIA NVS4200 and
running Ms Windows 10.
5.1 Parallel implementation
Our algorithm contains parallel code snippets with which we perform a load and
balance of the data among CPU/GPU cores when available. To illustrate this
2
Depicting DBPedia-Yago results. The YAGO was added to bring a reinforcement to
our evaluations, due to the fact of this dataset have been widely used in experiments
pertaining to Link Discovery. We also considered evaluations on recall, precision,
and f-score.
3
http://wiki.dbpedia.org/
4
http://linkedgeodata.org/
Avoided pairs (%)
N 0.3 Recall (N )
60 L Recall (L)
Recall
A 0.2 PG
40
20 0.1
0.8 0.9 1 0.8 0.9 1
Similarity threshold Similarity threshold
(a) Avoided pairs(10,273,950). (b) Recall.
Fig. 1. Avoided pairs and recall.
part of our idea, we can state: Given a two datasets S, T , that contains all the
strings to be compared. Thus, make a Cartesian product of the strings S × T ,
where each pair is the processed separately in threads that are spread among
CPU/GPU cores. Thus, we process the each comparison in parallel. The parallel
implementation works better in large datasets with size more than 105 , that was
more than one time faster than the approach without parallelism and two times
faster than the naive approach as shown in Figures 3(a) to 3(c).
5.2 Runtime Evaluation
The evaluation in Figures 3(a) and 3(b) shows that all filter setup outperform
the naive approach, and the parallel approach does not suffer significant changes
related to the runtime according to the size of the dataset, as show Figure 3(c).
The experiments related to the variance of k, also were considered, as show in
Figure 3(d), the runtime varies according the size of k, indicating the influence
of k with values from 1 to 120 with 1, 001, 642 comparisons. The performance
(run-time) was improved as shown in Figures 3(a) and 3(b) and according the
recall with a performance gain of 26.07% as shown in Figure 1(a). The time
complexity is based on two sort process O(n log n) + O(m log m) resulting in
O(n log n) as a upper bound in the worst case.
5.3 Scalability Evaluation
In the experiments (see Figures 3(c), 3(e) and 3(f)), we looked at the growth of
the runtime of our approach on datasets of growing sizes. The results show that
the combination of filters (N +L+A) is the best option for datasets of large sizes.
This result holds on both DBpedia and LinkedGeoData, so our approach can be
used on large datasets and achieves acceptable run-times. We also can realize
the quantity of avoided pairs in each combination of filters in Figure 1, that
consequently brings a performance gain. We looked at experiments with runtime
behavior on a large dataset with more than 106 labels as shown in Figure 3(b).
The results suggest that the runtime decreases according to the threshold θ
0.8 1
0.6 0.9
0.4 0.5 0.8
0.2 0.7
0 0 0.6
0.6 0.8 1 0.6 0.8 1 0.6 0.8 1
Similarity threshold Similarity threshold Similarity threshold
(a) y axis = F-score. (b) y axis = Precision. (c) y axis = Recall.
M F KC Jaccard JaroW inkler
Fig. 2. Precision, Recall and F-Measure.
increment. Thus, one more point showing that our approach is useful on large
datasets, where can be used with high threshold values for link discovery area.
About our parallel implementation, Figure 3(c) shows that our GPU parallel
implementation works better on large datasets with size greater than 105 .
5.4 Comparison with existing approaches
Our work overcomes the naive approach [8], thus, in order to show some im-
portant points we compare our work not only with the state of the art, but
with popular algorithms such as Jaccard Index [5]. As shown in Figures 3(c),
3(e) and 3(f), our approach outperforms not only the naive approach, but also
Jaccard Index. We show that the threshold θ and k have a significant influence
related to the runtime. The naive approach present some points to consider,
among them, even if the naive approach states that they did experiments with
k = 7, the naive algorithm was designed for only k = 2, there are some cases
where k = 2 is not enough to get the similarity level expected, i.e. s = mystring1
and t = mystring2 limiting k = 2, we will have σ2 (s, t) = 0.2, showing that the
similarity is very low, but when k = 8, the similarity is σ8 (s, t) = 0.8 showing
that sometimes we can lose a good similarity case limiting k = 2. Our work fix all
these problems and also has a better runtime, as show Figure 3(a), Figure 3(b)
and Figure 3(c).
An experiment with labels from DBpedia and Yago5 shows that the f-score
indicates a significant potential to be used with success as a string similarity
comparing with Jaccard Index and Jaro Winkler, as Figures 2(a) to 2(c) shows.
To summarize the key features that makes our approach outperform the naive
approach are the following: We use more than two K most frequent characters
in our evaluation, our run-time for more than 107 comparisons is shorter (27,594
against 16,916) milliseconds, we do have a similarity threshold, allowing us to
5
http://www.mpi-inf.mpg.de/departments/databases-and-information-systems/
research/yago-naga/yago/
·104
3
Time in milliseconds
Time in milliseconds
naive
3,000 N +L+A
L+A
2 N +A
2,000 A
parallel(N + L + A)
1 Jaccard
1,000
0.8 0.9 1 0.8 0.9 1
Similarity threshold Similarity threshold
(a) Runtime 1,001,642 comparisons. (b) Runtime 10,273,950 comparisons.
·104
Time in milliseconds
Time in milliseconds
4 naive
2,000 N +L+A
L+A
N +A
2 A
1,000
parallel(N + L + A)
Jaccard
0 0
0 0.5 1 0 50 100
7
# comparisons ·10 K
(c) The parallel approach improves (d) Runtime k most frequent characters,
the performance for more with values of k from 1 to 120,
than 105 comparisons. over 1, 001, 642 comparisons, θ = 0.95.
·104
Time in milliseconds
Time in milliseconds
naive
N +L+A
4,000 4 L+A
N +A
A
parallel(N + L + A)
2,000 2
Jaccard
2 4 6 8 2 4 6 8
Number of CPUs Number of CPUs
(e) CPU Speedup of algorithm (f) CPU Speedup of algorithm(10, 273, 950 comparisons).
(1, 001, 642 comparisons).
Fig. 3. Run-time experiments results.
discard comparisons avoiding extra processing, and a parallel implementation,
making our approach scalable. Jaccard does not show significant changes varying
the threshold, MFKC and Jaro Winkler present a very similar increase of the
f-score varying the threshold.
6 Conclusion and Future work
We presented an approach to reduce the computation runtime of similarity joins
using the Most Frequent k Characters algorithm with a sequence of filters that
allow discarding pairs before computing their actual similarity, thus reducing the
runtime of computation. We proved that our approach is both correct and com-
plete. The evaluation shows that all filter setup outperform the naive approach.
Our parallel implementation works better in larger datasets with size greater
than 105 pairs. It is also the key to developing systems for Record Linkage and
Link Discovery in knowledge bases. As future work, we plan to integrate it in
link discovery applications for the validation of equivalence links. The source
code is free and available online6 . Acknowledgments available on footnotes7 .
References
1. K. Dreßler and A.-C. N. Ngomo. On the efficient execution of bounded jaro-winkler
distances. 2014.
2. J. Euzenat, P. Shvaiko, et al. Ontology matching, volume 333. Springer, 2007.
3. H. H. Goldstine and J. von Neumann. Planning and coding of problems for an
electronic computing instrument. Institute for Advanced Study, 1948.
4. C. A. Hoare. Quicksort. The Computer Journal, 5(1):10–16, 1962.
5. P. Jaccard. The distribution of the flora in the alpine zone. New phytologist,
11(2):37–50, 1912.
6. V. I. Levenshtein. Binary codes capable of correcting deletions, insertions, and
reversals. In Soviet physics doklady, volume 10, pages 707–710, 1966.
7. R. Rivest. The MD5 message-digest algorithm. 1992.
8. S. E. Seker, O. Altun, U. Ayan, and C. Mert. A novel string distance function
based on most frequent k characters. IJMLC, 4(2):177–182, 2014.
9. S. E. Seker and C. Mert. A novel feature hashing for text mining. Journal of
Technical Science And Technologies, 2(1):37–40, 2013.
10. P. Shvaiko and J. Euzenat. Ontology matching: State of the art and future chal-
lenges. TKDE, 25(1):158–176, 2013.
11. T. Soru and A.-C. Ngonga Ngomo. Rapid execution of weighted edit distances. In
Proceedings of the Ontology Matching Workshop, 2013.
12. A. Valdestilhas, N. Arndt, and D. Kontokostas. DBpediaSameAs: An approach to
tackle heterogeneity in dbpedia identifiers.
13. C. Yan, X. Zhao, Q. Zhang, and Y. Huang. Efficient string similarity join in
multi-core and distributed systems. PLOS ONE, 12(3):1–16, 03 2017.
6
https://github.com/firmao/StringSimilarity/
7
Acknowledgments to CNPq Brazil under grants No. 201536/2014-5 and H2020
projects SLIPO (GA no. 731581) and HOBBIT (GA no. 688227) as well as the
DFG project LinkingLOD (project no. NG 105/3-2), the BMWI Projects SAKE
(project no. 01MD15006E) and GEISER (project no. 01MD16014).