GeoRoBERTa: A Transformer-based Approach for Semantic Address Matching

Abstract

In this paper, we describe a solution for a specific Entity Matching problem, where entities contain (postal) address information. The matching process is very challenging as addresses are often prone to (data) quality issues such as typos, missing or redundant information. Besides, they do not always comply with a standardized (address) schema and may contain polysemous elements. Recent address matching approaches combine static word embedding models with machine learning algorithms. While the solutions provided in this setting partially solve data quality issues, neither they handle polysemy, nor they leverage of geolocation information. In this paper, we propose GeoRoBERTa, a semantic address matching approach based on RoBERTa, a Transformer-based model, enhanced by geographical knowledge. We validate the approach in conducting experiments on two diferent real datasets and demonstrate its efectiveness in comparison to baseline methods.

1. Introduction

Dakar (NoMatch label). The second address pair has a PartialMatch label as there is a similarity between at least Entity Matching (EM) is the problem of identifying data one of its elements (Road: Avenue Lamine Gueye), apart instances that refer to the same real-world entities [ 1, 2 ]. from the similarity between the City Dakar. The last row In this paper, we address a specific EM problem where represents an example of a Match between addresses as entities consist of postal addresses. More precisely, Given all their elements are similar (except the missing PoBox two postal addresses and , do those addresses refer in address A). to the same real world (address) entity ? We coin this Former address matching approaches [ 6, 7 ] are based problem as Address Matching although that terminology on similarity measures and matching rules. However, may also refer to either work based on Geocoding [ 3 ], or these methods perform a structural comparison of adto software tools such as PlaceKey1. Address matching dresses and are unable to identify some relationship beis a crucial task for various location-based businesses tween two addresses when they have few literal overlaps as one may lose clients or prospects in case of delivery [ 3 ]. In such cases, semantic address matching is required failure. It is a challenging one, especially in absence of a for identifying/exhibiting semantic similarities between standard address model. addresses that have the same location with diferent rep

Formally, the address matching task may be consid- resentations [ 8 ]. ered as a binary classification problem [ 4, 3, 5, 6, 7 ] where Recently, semantic address matching solutions have the predicted class is either Match or No Match. However, been proposed [ 4, 3, 5 ], based mainly on word embedgiven two companies with the same name, it is impor- ding models combined with classical Machine Learning tant to identify addresses that are partially similar, such (ML) or Deep Learning (DL). Nevertheless, these soluas those having the same city and the same road but tions may be impacted by the presence of polysemous difer in the house number or in the case where both words since they are based on static word embedding addresses are correct but one of them corresponds to a models. Polysemy cases may occur in an address when former address company, in order to complete addresses it contains a place name that refers to diferent places in with up-to-date information. As a result, we consider a country or worldwide as illustrated in Table 2. Identithe problem as a multiclass classification one in adding fying and resolving polysemic situations is mandatory a ℎ class. Table 1 shows examples of ad- to avoid matching distortion. This has led to the advent dress matching. Given two Senegalese addresses and of transformer-based solutions [ 9 ] which have shown , the first pair illustrates the case where there is no promising results on general Entity Matching [ 10, 11 ] similarity between address elements apart from the City thanks to their highly contextualized embedding.

This motivated us to explore the efectiveness of Transformers in address matching by proposing an approach based on RoBERTa [ 12 ], a pre-trained Transformer language model, for address matching in the context of French-speaking countries. Nevertheless, since these models produce address embedding mainly from linguistic contexts, they may miss some (domain) knowledge, Published in the Workshop Proceedings of the EDBT/ICDT 2023 Joint Conference (March 28-March 31, 2023, Ioannina, Greece) $ [e-mail] (Y. Guermazi); [e-mail] (S. Sellami); [e-mail] (O. Boucelma)

© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License 1 hCPWrEooUrctkReshtdoinpgpssIhStpN:/c1e:6u1r3-w/-0s.o7r3g/wACwttErwibUu.tpRiolnaW4c.0eoInkrteekrynsa.htiiooon/pal (PCCroBYce4.0e).dings (CEUR-WS.org)

Address B Description 12 Boulevard Djily Immeuble Azur and 12 Boulevard Djily Mbaye Mbaye Dakar Senegal refer to the same geographic location, in Dakar

There is no match between A and B although they contain Les Garennes 78130 the same place name "Garennes" which is a polysemous Les Mureaux France word as it refers to two diferent places: a Road and an industrial zone (in two diferent cities). which is dificult to learn from raw texts. Therefore, we have gained momentum for solving the semantic address propose to enhance the contextual address embedding matching problem. They are integrated in address matchof RoBERTa by two types of geographical knowledge, ing pipeline. For example, several studies [ 13, 14, 3, 5 ] obtained from address tag embedding and address geo- adopted the same pipeline with diferent used techniques graphic coordinates. in the three steps: CRF model, Trie syntax tree algo

The contributions of this paper can be summarized as rithm, jieba library2 or rule based method [ 14 ] as address follows: parser, Word2vec [15] or fastText [16] as word embedding models and several ML (e.g. SVM, XGBoost) and • We defined GeoRoBERTa, a semantic address DL models (e.g. enhanced sequential inference model, matching approach, which relies on RoBERTa, Bi-LSTM, CNN) as classifiers. These works have shown a transformer-based model. the efectiveness of their proposed approaches compared • We injected two types of geographical knowledge to baseline methods (non word embedding-based methinto RoBERTa: address tag embedding and geo- ods) thanks to their capacity to detect semantic similarity graphic coordinates encoding. This enables better between address attributes. handling of polysemy and better identification of However, these approaches may present two weaksemantic similarity between addresses. nesses. The first one is related to the management of • We conducted an extensive experimental study polysemous cases. In fact, these approaches are based on where GeoRoBERTa is compared to baseline meth- static word embedding models, which cannot handle polods. Real (unstructured and structured) data, con- ysemy as they generate static vector representations of sisting of French postal addresses, has been used. words. Contrariwise, contextual word embedding models, among which the transformer-based ones, resolve

The rest of the paper is organized as follows: Related this problem thanks to their highly contextualized embedwork on address matching is reviewed in Section 2. Sec- ding as demonstrated in entity matching works [ 11, 10 ]. tion 3 presents a formalization of the problem. We de- The second weakness is related to the leveraging of gescribe our solution in Section 4, and present experimental ographic information. Indeed, these approaches are deresults in Section 5. Finally, Section 6 concludes the pa- signed without geographic location information, which per. ignores the geographic features when performing address matching. And yet, addresses that belong to the same 2. Related Work geographic area should have intuitively similar geospatial characteristics. However, these assumptions may fail as existing methods rely only on address text which can contain vernacular content or place synonyms and does not follow a standard structure making them inherently ambiguous. Thus, modelling the problem from linguistic perspective alone is not enough.

In this context, former approaches have specifically Address matching pipeline [ 13, 14 ] is generally composed by three steps: (1) address parsing, i.e., decomposition of an address into its diferent components (e.g. street name, zip code), (2) generation of an embedded address vector by means of word embedding models and (3) application of a ML or a DL model resulting in a binary address classiifcation (Match, No Match). Word embedding techniques 2https://github.com/fxsjy/jieba used geocoding in the address standardization process to obtain the geolocation followed by a reverse geocoding, which generates a complete and proper address before performing the matching. This strategy has been applied for example in [17]. Recently, some works [18, 19] focused on the enrichment of Point Of Interest (POI) embedding using geographic information. The most popular form of this information is the encoding string of the geographic coordinates, obtained by the Geohash geocode system 3. In [19], authors proposed a POITransformers framework to generate POI embeddings in order to perform POI Matching. A POI is defined as an entity composed by four attributes: name, category, address and geographic coordinates. The proposed matching approach consists firstly in generating an embedding vector for each POI by fusing the text embedding of the ifrst three attributes using BERT [ 20], a Transformerbased model, and the geographic location embedding of the last attribute. Then, the similarity between each pair of POI’s embedding is computed using two techniques: cosine similarity and SentEval toolkit [21]. The proposed approach achieves results comparable in terms of performance with those of existing DL-based methods (e.g. DeepER [22], DeepMatcher [23]) on general Entity Matching benchmark datasets but it outperforms them on POI Entity Matching datasets.

In summary, Transformer-based models have proven their efectiveness in general entity matching but they are less explored in the address matching task. Their application on these domain-specific data should also take into account geographic information in addition to the linguistic context. From this perspective, some works start introducing geographical knowledge (geohash encoding) in Transformer-based model to perform, especially, POI matching, but they may miss additional domain information to efectively deal with polysemy. Therefore, in this work, we propose GeoRoBERTa a semantic address matching approach based on a pre-trained transformerbased language model (RoBERTa) which incorporates two types of geographical knowledge: address tag embedding and geohash encoding in order to better deal with polysemous cases and to improve the identification of semantic similarity between addresses.

3. Problem Statement

As discussed in Section 1, due to the heterogeneity in address representations, we need to extract some « intuitive/hidden » semantic relationships between addresses. Prior to that, we first present the address model that we adopted in this paper. Then, we provide a definition of semantic address matching, along the lines of the one provided in Xu et al [ 8 ].

3http://geohash.org/

Definition 1 (Address Schema). Given a set of (entity) attributes {1, .., }, an address = {1, .., } where is the i-th address token (word) and is the address length, with ≤ , and () is a "list" constructor.

More formally, to cope with diferent address representations (e.g., France and Senegal in this paper), we distinguish between two types of addresses: 1. A Simple Address is a sequence of attributes (Table 3) which are defined by the address model proposed in [ 24 ]. 2. A Complex Address is a composition of (at least) two simple addresses by means of a spatial operator. Table 4 below illustrates the proximity operator and the intersection operator , while Table 5 shows two complex Senegalese addresses.

3.2. Semantic Address Matching

Definition 2 (Semantic Address Matching). Given two address datasets: 1 = {1, .., } and 2 = {1, .., ′ }, where and ′ are the size of 1 and 2 (respectively), the Semantic Address Matching aims to find each address pair ( , ), satisfying = or ≈ , where and are simple or complex addresses such as ∈ 1 and ∈ 2, = and ≈ represent the equality and the approximation operator, respectively. The addresses on either side of the equality operator refer to the same real-world object with the same geographic location (coincide with relationship). Whereas, the addresses on either side of the approximation operator are semantically related: there is a specific relationship located in between their attributes (i.e. an address is located in an address or vice versa). In this work, the address pairs labels are defined as follows: • Match: it is attributed to an address pair, between which there is the relationship coincide with • PartialMatch: it is attributed in two scenarios: (1) there is a relationship located in between an address pair or (2) there is a relationship coincide with between a partial part of an address pair.

• NoMatch: otherwise

4. Proposed Approach

4.2.1. Geographic Coordinates Encoding We augment each address by a geographical knowledge derived from the encoding of geographical location represented as a latitude (lat) and a longitude (long) pair.

First, we used Google Geocoding API 6 to convert each address into geographic coordinates (lat and long). Then, we translate the two-dimensional location into geographically meaningful embeddings using Geohash [ 25 ] which is a geocoding system that encodes the geographic location of a place into a short string of letters and digits. An important property of geohash is that two places with a long common geohash prefix are close to each other [ 26 ].

We append address texts with geohashes to provide the geospatial context to the RoBERTa model. Figure 2 shows an example of geographic coordinates encoding of a French address.

In this section we describe GeoRoBERTa (Figure 1), a RoBERTa-based approach and model for semantic ad- 4.2.2. Generation of Address Tag Embedding dress matching.

GeoRoBERTa consists of three main tasks: (1) Data Preprocessing in order to clean data, (2) Geographical Knowledge Generation and (3) Address Matching which is based on a pre-trained RoBERTa model enhanced by the geographical knowledge in order to classify each address pair as either Match, PartialMatch or NoMatch.

4.1. Data Preprocessing

The purpose of this step is to normalize and clean addresses with removing special characters and expanding abbreviations. For that, we adopt a dictionary-based approach which provides the keywords that may be used to define the components of addresses as well as common abbreviations of these words. As we are interested in addresses belonging to French-speaking countries, we extract French keywords from oficial sources, in France, such as the Post Ofice, the INSEE 4 service and unoficial sources which generally have common abbreviations, such as the list of abbreviations recognized by the OpenStreetMap 5 query tools. In addition, all addresses are normalized with expanding abbreviations to their corresponding words in the created dictionary which contains a set of keywords that are likely to be used to define address’s components (avenue, road, building, etc.) and their abbreviations.

4https://www.sirene.fr/sirene/public/variable/typvoie 5https://wiki.openstreetmap.org/wiki/Name_finder:Abbreviations

It consists of two steps: address parsing and address tag embedding.

(1) Address Parsing: The parsing of an address = {1, .., } aims to assign a label to each word of among the corresponding list of address tags = {, , , , , , , , , , , , , }

These tags (Table 3 and 4) are defined following the address model described in section 3.1.

We applied the address parsing method (Figure 3) proposed in [ 24 ], thanks to its efectiveness compared to several baseline methods, especially in identifying polysemous address elements. The parsing is based on the use of a RoBERTa model, which generates firstly a contextual representation of an input address , following these two sub-steps: • RoBERTa calculates the input representations of by summing over the token, position, and segment embedding. • Input address representation goes through 12 transformer encoders which capture the contextual information for each token by self-attention and produces a sequence of contextual embeddings.

The resulted representation is then provided to a tagging layer (a Fully Connected Layer) to obtain address tags, using the IOB (Inside–outside–beginning) tagging scheme [ 27 ], where a token is labeled as B-tag if it is at the beginning of the address element, or I-tag if inside the address element but not first, otherwise O-tag.

6https://developers.google.com/maps/documentation/geocoding

The tagging layer takes as input the last hidden state of the obtained sequence of contextual embeddings and provides as result the prediction of the tags.

(2) Address Tag Embedding: The output of the parsing step of the address (respectively address ) is tags (respectively tags). Since these tags are at the word level, their length is equal to the length of (respectively the length of ). We augment these tags by another tag (B-GC) which represents the corresponding geohash of each address. Then, we use a look-up table to map these tags to identifiers and feed a linear layer to obtain the representations of the tags of the address pair.

4.3. Address Matching

It consists of two steps (Figure 1): (1) generating a fusion of two vector representations which are the contextual vector representation of the address pair and the vector representation of the address pair tags, and (2) a classification of each pair according to resulted vectors. 4.3.1. Vectors Fusion

We fuse two embedding vectors as follows:

1. Contextual embedding of address pair: The byte

pair encoding (BPE) tokenizer 7 of RoBERTa was used to encode the input addresses into tokens. These tokens and the two geohashes, representing the address pair (A, B), form the input to the pre-trained RoBERTa model. Then, this model generates the contextual vectors representations of the address pair (A, B). 2. Tags embedding of address pair: They are generated from the previous step (as described in Section 4.2.2).

The fusion of vectors is performed by a concatenation function which is the most popular feature-level fusion methodology [ 28, 29 ]. 4.3.2. Address Pair Classification It is performed using a fully connected layer (a linear layer), which is the classifier layer by default in RoBERTa packages. This layer takes as input the resulting embedding fusion vector and generates as output the class logits (probabilities), knowing that the objective of the training is the CrossEntropy. Then, the Argmax function is applied to these probabilities to get the predicted class.

5. Evaluation

In this section, we describe the experiments carried out in order to evaluate our address matching approach. Source code is available at the following Git repository: https: //github.com/MatchSystem/GeoRoBERTa. 5.1. Experimental Settings 5.1.1. Dataset Description in the context of address matching. Therefore, we proceed with the following steps in order to create our own labeled dataset: 1. Step 1: Address collection: The French dataset has been collected (on July 12, 2022) from the Legal Entity Identiefir (LEI) database 8(the French company’s addresses) and contains 40000 addresses, whereas the Senegalese dataset is generated from Senegalese company directories 9 and contains 5000 addresses. 2. Step 2: Address pairs creation: For each dataset, we create a labeled set, composed by address pairs and their corresponding label using diferent strategies based on [ 3 ]: • For Match address pair: the creation of these pairs are performed using 3 strategies: (1) a simple clone of the address, (2) attribute removal, to create semantic similar elements such as removing either the street address (HouseNum+Road) or the ExtBuilding or POI if they both exist, and (3) token removal, by a deletion of a randomly sampled span of tokens. • For PartialMatch address pair: we use mainly the attribute removal strategy to create the address pairs, while ensuring that City elements are similar and there is at least a similarity between another element of the address pair. • For NoMatch address pair: for each address from the dataset (French or Senegalese), three strategies are used to choose the second address of the pair: (1) Random selection of an address from the dataset (the two datasets do not contain duplicate address), (2) Selection of an address with the same city and, (3) Selection of an address with literal overlap.

The frequency of the classes (labels) of address pairs for the two datasets (French dataset denoted and Senegalese dataset denoted ) is given by the Table 7. Besides, and are split into the training, validation, and test sets using the ratio of 3:1:1. Table 6 shows a sample of the training set of , on which RoBERTa is trained.

Our experiments are conducted on two real-world datasets representing addresses from two Frenchspeaking countries: (1) France and (2) Senegal. Unlike 5.1.2. Compared Methods several known data contest competitions (e.g. Kaggle, We compare GeoRoBERTa with baseline methods used SIGIR), there is no such real dataset for these countries in some address matching related works [ 13, 14 ]. We

7https://huggingface.co/docs/transformers/tokenizer_summary

8https://www.gleif.org/en/lei-data/gleif-golden-copy/

download-the-golden-copy#/ 9https://www.goafricaonline.com/sn

As illustrated in Section 4, GeoRoBERTa takes as input the whole address pairs augmented with the corresponding geohash, to the contrary of the two baseline approaches [ 13, 14 ] where the input is the set of attributes of each address pair. For a fair comparison, we added two attributes to each address pair corresponding to its geohash strings. • Word2vec + XGBoost [ 13 ]: in adopting this approach, we trained a Word2vec model over an 5.2. Evaluation Setup address corpus (section 5.2.2) using Gensim 10 library with vectors of dimension 100, a window 5.2.1. Hardware size of 15. Then, the model is used to generate The experiments were carried out on a Dell PC with the word embedding of each address of the training following characteristics: dataset. We obtain the embedding of each address attribute by averaging all their words embedding. • Processor: Intel® Core 8th (4 core), HT, 1.9Ghz, The cosine similarity between the embedding of 8Mo, 15W / UHD 620 the same type of address attributes is used as fea- • Hard disk: SSD 512Go M.2 SATA tures in a XGBoost classifier implemented using • RAM: 16Go 2400MHz DDR4 (2x8Go) scikit-learn 11. • Operating system: Microsoft Windows 10 Pro, • fastText + SVM [ 14 ]: fastText model is firstly used 64 bits to obtain address embedding. It is trained over an address corpus (section 5.2.2) using Gensim The compared approaches are executed on "NVIDIA Tesla library with vectors of dimension 100, a window K80" GPU using Google Colab (with 12 GB of RAM). size of 15. Then, features are obtained by applying cosine similarity between embedding of the same 5.2.2. RoBERTa pre-training and fine-tuning and of the diferent type of address attributes.

These features serve as input to a SVM classifer. RoBERTa-base architecture (12-layer, 768-hidden, 12• RoBERTa: This base form of GeoRoBERTa corre- heads, 125M parameters) is used for pre-training and sponds to fine-tuning the pre-trained RoBERTa ifne-tuning. The model is pre-trained to optimize the on address matching. We did not inject any ge- Masked Language Modeling objective. RoBERTa preographical knowledge. This variant is similar to training was performed with the Pytorch framework 12 the entity matching approach proposed in [ 11 ]. and Transformers library 13 with a vocabulary size of 30000 tokens. We generated two pre-trained RoBERTa • GeoRoBERTa(GT): In this version, only geographic models corresponding to each of the following corpora: tags embedding knowledge has been added to 10https://pypi.org/project/gensim/ 11https://scikit-learn.org/stable/ (1) French corpora composed of 1,048,575 addresses 14 Road). Overall, the performance of the diferent models, and (2) Senegalese corpora composed of 31893 addresses in terms of F-measure, is higher in the case of French collected from Web business directories 15/16/17/18. These addresses (vs. Senegalese ones) due to their structured datasets have been processed according to the steps de- nature. scribed in Section 4.1.

Impact of the parsing method To evaluate the impact of the parsing on matching results, we consider three address baseline parsing methods: rules-based [ 14 ], CRFbased, and RoBERTa. Parsing evaluation results (Table 9) show that RoBERTa is more accurate than the other methods because it handles polysemous words. Table 10 illustrates the impact of the parsing method on the performance of the address-matching approaches. We note that all the matching approaches combined with a parsing method based on RoBERTa perform better than the approaches combined with CRF or those based on Rules. Besides, the impact of the parsing method is more important with Senegalese data since it contains more polysemous cases. 5.2.3. Hyperparameters Tuning GeLU activation is used in RoBERTa with the ADAM Optimizer. For both tasks (parsing and matching), the dropout and learning rates are set respectively to 0.1 and 3e-5 in such a way as to maximize the accuracy in the validation set. To avoid overfitting, we use the early stop technique based on loss validation by setting a maximum number of training epochs (=12) and a batch size of 32. 5.2.4. Evaluation Metric − = 2 ×

Precision × Recall Precision + Recall

(1) To evaluate the performance of our model and all the baselines, we use the F-measure, which is the harmonic mean of the precision, the rate of correct predictions, and the recall, the fraction of correct classes being predicted. Table 9 F-measure of Address Parsing Methods.

5.3. Results

5.3.1. Comparison with baselines First, GeoRoBERTa is compared to baseline approaches, using the same address parsing method (RoBERTa). The GeoRoBERTa evaluation results, illustrated in Table 8, show that GeoRoBERTa outperforms the other approaches on the two datasets thanks to the highly contextualized vector representations of RoBERTa compared to fastText and Table 11 Word2vec. Besides, the fastText-based approach outper- Computation time (sec.) of Address Matching Approaches. forms the Word2vec-based one due to the richness of the Approach extracted features in the former approach compared to Name Training Evaluation Training the second one. These features represent the cosine simi- fWasotrTde2xvte+c S+VXMGBoost 11388710 3328 19164 larity between attributes from diferent types (e.g., Road GeoRoBERTa 7843 127 971 vs District) and those from the same type (e.g., Road vs

Evaluation 10 12 36 14https://www.data.gouv.fr/fr/datasets/base-sirene-des-entreprises

et-de-leurs-etablissements-siren-siret/#description 15https://creationdentreprise.sn/ 16http://pagesjaunesdusenegal.com/ 17https://www.goafricaonline.com/ 18https://www.yelu.sn/

Runtime We evaluate the diferent address match

ing models on their training and evaluation in the test set. Results (Table 11) show that the training time of GeoRoBERTa is costly due to the deep transformer based Method Name

Rule-based

CRF RoBERTa 5.3.2. Ablation Study We analyze the contribution of each type of geographic knowledge by comparing GeoRoBERTa with its variants (described in section 5.1.2). The experimental results are shown in Table 12. We first focus on comparing GeoRoBERTa(GT) and GeoRoBERTa(GH) to RoBERTa. The obtained results show that the injection of geographical knowledge (regardless of their types) slightly improves the performance as we note an increase of F-measure in GeoRoBERTa(GT) and GeoRoBERTa(GH) compared to RoBERTa on the two datasets. In fact, these models are more robust when dealing with semantic similarities and polysemy cases.

Next, we note that the precision results of GeoRoBERTa(GT) and GeoRoBERTa(GH) are close to each other. Moreover, unlike GeoRoBERTa(GT), GeoRoBERTa(GH) can detect semantic similarities between unseen addresses during the pre-training or the training steps, thanks to the geohash, as illustrated in Table 13 (first row’s example): There is a Semantic similarity between Zone Industrielle Les Blanchisseries and Rue Louis Leprince Ringuet: The road exists in the zone area (Similar geohash between the two addresses).

On the other hand, GeoRoBERTa(GT) is more eficient when dealing with polysemy cases thanks to the semantic labels embedding. Indeed, polysemy cases can represent examples of ambiguous addresses that are dificult to geocode as illustrated in Table 13 (second row’s example): Rufisque is a polysemous element which may refer to a Road or a District in Senegal and can be found in diferent geographical areas. GeoRoBERTa(GH) did not consider this polysemy case as the two generated geohash are similar, while GeoRoBERTa(GT) captures the polysemy and predicts the correct label of the address pair. Furthermore, the quality of geographic coordinates can influence the performance of GeoRoBERTa(GH). In such cases, we note that this model is almost competitive with RoBERTa for the Senegalese dataset due to the low accuracy of Google Geocoding API, which is 64 % (Table 14). On the other side, GeoRoBERTa(GH) outperforms GeoRoBERTa(GT) when dealing with the French dataset for which the geocoding accuracy is better (89%).

Overall, we can note that GeoRoBERTa outperforms all its variants against the two datasets as it leverages the two types of incorporated knowledge. The incorporation of geohash encoding allowed us to have a more eficient model able to improve the identification of semantically similar address pairs, mainly when they are not used in the training of RoBERTa. Incorporating address tag embeddings allowed GeoRoBERTa to better deal with polysemous cases, (e.g., Senegal).

6. Conclusion

In this paper, we described GeoRoBERTa, a transformerbased address-matching solution that relies on RoBERTa, a pre-trained transformer language model, leveraging two types of geographical knowledge during the matching phase. Extensive experimental evaluations on two real-world datasets show that our solution is efective and outperforms baseline models. Besides, the ablation study demonstrated the positive impact of geographical knowledge injection in improving the matching phase, especially in semantic similarities and polysemy cases.

In the future, we intend to extend this work in two directions: (1) evaluating the impact of the geocoding in the matching result by testing other geocoding solutions, and (2) studying the performance of GeoRoBERTa on dirty address datasets (by injecting spelling errors).

References

[1] A. K. Elmagarmid , P. G. Ipeirotis , V. S. Verykios , Duplicate record detection: A survey , IEEE Trans. Knowl. Data Eng . 19 ( 2007 ) 1 - 16 .

[2] P. Christen , Data Matching - Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection, Data-Centric Systems and Applications, Springer, 2012 .

[3] Y. Lin , M. Kang , Y. Wu , Q. Du , T. Liu, A deep learning architecture for semantic address matching , Int. J. Geogr. Inf. Sci . 34 ( 2020 ) 559 - 576 .

[4] S. Shan , Z. Li , Q. Yang , A. Liu , L. Zhao , G. Liu , Z. Chen , Geographical address representation learning for address matching , World Wide Web 23 ( 2020 ) 2005 - 2022 .

[5] J. Chen , J. Chen , X. She , J. Mao , G. Chen, Deep contrast learning approach for address semantic matching , Applied Sciences 11 ( 2021 ) 7608 .

[6] D. K. Matci , U. Avdan, Address standardization using the natural language process for improving geocoding results , Comput. Environ. Urban Syst . 70 ( 2018 ) 1 - 8 .

Table 14 edge Discovery in Databases (ECML PKDD 2020 ) Accuracy of the Geocoding System on the French and Sene- Ghent, Belgium, September 14-18 , 2020 , Proceedgalese Datasets . ings, volume 1323 of Communications in Computer Geocoding system and Information Science, Springer, 2020 , pp. 320 - Google Geocoding API 89 % 64% 334 . [15] T. Mikolov , K. Chen , G. Corrado, J. Dean , Eficient estimation of word representations in vector space,

[7] T. Gschwind , C. Miksovic , J. Minder , K. Mirylenka, in: 1st International Conference on Learning RepP. Scotton, Fast record linkage for company entities, resentations , ICLR 2013 , Scottsdale, Arizona, USA, in: 2019 IEEE International Conference on Big Data May 2-4 , 2013 , Workshop Track Proceedings, 2013 . (IEEE BigData), Los Angeles, CA, USA, December [16] P. Bojanowski , E. Grave , A. Joulin , T. Mikolov, En9 - 12 , 2019 , IEEE, 2019 , pp. 623 - 630 . riching word vectors with subword information,

[8] L. Xu , R. Mao , C. Zhang , Y. Wang , X. Zheng , X. Xue , Trans. Assoc. Comput. Linguistics 5 ( 2017 ) 135 - 146 . F. Xia, Deep transfer learning model for semantic [17] I. K. Koumarelas , A. Kroschk , C. Mosley , F. Nauaddress matching , Applied Sciences 12 ( 2022 ) 10110 . mann, Experience: Enhancing address matching

[9] A. Vaswani , N. Shazeer , N. Parmar , J. Uszkoreit, with geocoding and similarity measure selection , L. Jones , A. N. Gomez , L. Kaiser , I. Polosukhin , At- ACM J. Data Inf. Qual . 10 ( 2018 ) 8: 1 - 8 : 16 . tention is all you need , in: Advances in Neural [18] J. Jin , Z. Xiao , Q. Qiu , J. Fang , A geohash based Information Processing Systems 30 : Annual Con- place2vec model , in: 2019 IEEE International Geoference on Neural Information Processing Systems science and Remote Sensing Symposium, IGARSS 2017, December 4-9 , 2017 , Long Beach, CA, USA, 2019 , Yokohama, Japan, July 28 - August 2 , 2019 , 2017 , pp. 5998 - 6008 . IEEE, 2019 , pp. 3344 - 3347 .

[10] Y. Li , J. Li , Y. Suhara , A. Doan , W. Tan , Deep entity [19] J. Zhang , C. Zhang , X. Liu , X. Li , W. Liao , P. Liu, matching with pre-trained language models , Proc. Y . Yao, J. Zhang , Poi-transformers: Poi entity matchVLDB Endow . 14 ( 2020 ) 50 - 60 . ing through poi embeddings by incorporating se-

[11] U. Brunner , K. Stockinger , Entity matching with mantic and geographic information ( 2021 ). transformer architectures - A step forward in data [20] J. Devlin , M. Chang , K. Lee , K. Toutanova , BERT: integration, in: Proceedings of the 23rd Interna- pre-training of deep bidirectional transformers for tional Conference on Extending Database Technol- language understanding , CoRR abs/ 1810 .04805 ogy, EDBT 2020 , Copenhagen, Denmark, March ( 2018 ). 30 - April 02, 2020 , OpenProceedings.org, 2020 , pp. [21] A. Conneau , D. Kiela , Senteval: An evaluation 463-473. toolkit for universal sentence representations , in:

[12] Y. Liu , M. Ott , N. Goyal , J. Du , M. Joshi , D. Chen , Proceedings of the Eleventh International ConferO. Levy, M. Lewis , L. Zettlemoyer , V. Stoyanov, ence on Language Resources and Evaluation, LREC Roberta: A robustly optimized BERT pretraining 2018, Miyazaki , Japan, May 7- 12 , 2018 , European approach, CoRR abs/ 1907 .11692 ( 2019 ). Language Resources Association (ELRA) , 2018 .

[13] S. Comber , D. Arribas-Bel , Machine learning in- [22] M. Ebraheem , S. Thirumuruganathan , S. R. Joty , novations in address matching: A practical com- M. Ouzzani , N. Tang , Distributed representations parison of word2vec and crfs , Trans. GIS 23 ( 2019 ) of tuples for entity resolution , Proc. VLDB Endow . 334 - 348 . 11 ( 2018 ) 1454 - 1467 .

[14] Y. Guermazi , S. Sellami , O. Boucelma , Address vali- [23] S. Mudgal , H. Li , T. Rekatsinas , A. Doan , Y. Park, dation in transportation and logistics: A machine G . Krishnan, R. Deep , E. Arcaute , V. Raghavendra, learning based entity matching approach, in: ECML Deep learning for entity matching: A design space PKDD 2020 Workshops - Workshops of the Euro- exploration , in : Proceedings of the 2018 Internapean Conference on Machine Learning and Knowl- tional Conference on Management of Data, SIGMOD Conference 2018 , Houston, TX, USA, June 10-15, 2018 , ACM, 2018 , pp. 19 - 34 .

[24] Y. Guermazi , S. Sellami , O. Boucelma , A roberta based approach for address validation , in: New Trends in Database and Information Systems - ADBIS 2022 Short Papers , Doctoral Consortium and Workshops: DOING, K-GALS , MADEISD, MegaData, SWODCH, Turin, Italy, September 5- 8 , 2022 , Proceedings, volume 1652 of Communications in Computer and Information Science, Springer, 2022 , pp. 157 - 166 .

[25] Z. Balkic , D. Sostaric , G. Horvat, Geohash and UUID identifier for multi-agent systems , in: Agent and Multi-Agent Systems . Technologies and Applications - 6th KES International Conference, KESAMSTA 2012 ,Dubrovnik, Croatia, June 25-27, 2012 . Proceedings, volume 7327 of Lecture Notes in Computer Science, Springer, 2012 , pp. 290 - 298 .

[26] K. Lee , R. K. Ganti , M. Srivatsa , L. Liu, Eficient spatial query processing for big data , in: Proceedings of the 22nd ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems , Dallas/Fort Worth, TX, USA, November 4- 7 , 2014 , ACM, 2014 , pp. 469 - 472 .

[27] L. A. Ramshaw , M. Marcus , Text chunking using transformation-based learning , in: Third Workshop on Very Large Corpora, VLC@ACL 1995 , Cambridge, Massachusetts, USA, June 30, 1995 , 1995 .

[28] M. E. Peters , M. Neumann , M. Iyyer , M. Gardner , C. Clark , K. Lee , L. Zettlemoyer , Deep contextualized word representations, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , NAACL-HLT 2018 , New Orleans, Louisiana, USA, June 1-6, 2018 , Volume 1 ( Long Papers) , Association for Computational Linguistics , 2018 , pp. 2227 - 2237 .

[29] Q. Zheng , J. Zhu , Z. Li , S. Pang , J. Wang , Y. Li , Feature concatenation multi-view subspace clustering , Neurocomputing 379 ( 2020 ) 89 - 102 .