The integration of external and internal data sources is becoming increasingly important, as decisionmakers depend on complete information. Often the data sources to be integrated do not have a common and unique identifier. In these cases, the data sources must be integrated by comparing the available common attributes of the entity, the so-called record linkage. There are similarity measures for attributes that contain strings like the company name or numbers such as turnover. Attributes that contain natural language text, such as company descriptions, are still unused. This research paper describes a research project on using natural language text attributes applying Machine Learning for entity matching. The use of natural language text attributes is intended to improve the results of entity matching and, thus, data integration.
Internal and external data sources are crucial for supporting decision-making processes in
research and industry, as they can contain relevant information for the decision-maker [
attributes such as name and address. Edit distances like Levenshtein or Jaro-Winkler can
detect simple changes of characters. Token-based methods like Jaccard and Tf-IDF can detect
token permutations. Hybrid methods such as Monge-Elkan and Soft-TF-IDF can detect token
permutations with character changes [
resolution OR record linkage OR entity linking
The first screening consists of the analysis of title, abstract and conclusion and serves as a
preliminary selection of relevant papers. This is followed by a forward-backward search,
identifying the cited contributions in these papers as well as publications that refer to these papers
[
Mudgal et al. use diferent techniques to use both structured attributes and natural language
texts in RL [
Nie et al. [20]
Wu et al. [25]
X X X X X X X X X X X X X X X t x e t d e r u t c u r t s i m e S X X X X X X X t x e t e g a u g n a l l a r u t a N X X X X X X X s e i t i t n e y n a p m o C X X X X g n i n u t e n i F X X X X g n i n r a e L r e f s n a r T s e h c a o r p p a l a u g n i l i t l u M (X) e r u t c e t i h c r A g n i d d e b m E
paragraph2vec
architecture to perform RL[23].
In the papers [
Existing datasets like the Abt-Company dataset contained in the Deepmatcher framework are not able to sufice for our work for various reasons. Firstly, the dataset does not inherently contain texts in languages other then English. Secondly, the data sources used are Wikipedia texts and crawled company websites, and the latter appear to vary in data quality and which ultimately might afect performance. Thirdly, no enterprise databases, which are regarded highly relevant for the practitioners in our research project, are included in the data. Since linking records from professional data sources is one key aspect of our use case driven project, training data from such sources might prove helpful to find the best approaches to solve the RL task. Based on our requirements and building on the shortcomings of the existing datasets, we propose a evaluation set for company description based RL, primarily for German and English texts. However, the proposed approach is easily adaptable to create evaluation sets for various languages.
We have identified three real-world data sources that are used to build the evaluation dataset. The first data source is Wikidata 1. From the Wikidata 258,109 companies were extracted, that are represented via a unique identifier. The Wikipedia API 2 provides functions to query individual components of a Wikipedia page using such an wikidata identifier. For all the entities available, we used the API to query the English company descriptions from the Wikipedia. The contained text is available as readable raw text and does not contain HTML or Wikimedia tags (markup language), which makes later processing easier and generally can be regarded as high quality data.
The Wikipedia API may also be used to determine whether a Wikipedia page exists in other languages and what the corresponding Wikipedia link is3. For a total of 20,126 entities, Germanlanguage pages are also available. These 20,126 entities were selected for further consideration in our work, since diferent texts for the same entity are present (English as well as German) and thus may directly be used to train or evaluate (cross-lingual) textual RL models without the need of manual annotation. The second data source used is the English-language company database Crunchbase4. Crunchbase ofers two data sets containing companies with no common
258,109 589,343
108,451 Mixed language text_1dataset text_2 60,528 label both texts are german
32,488 german text_1dataset text_2 label english short description exists
english long description exists english wikipedia
text exists german wikipedia text exists dteotresagcnnergisrpllimastihotaenns ttrraanonosrsriliglagaitiDnteneaedadlsltetceoonrniggpggelteliiisrosrmhmhnassalonhnnoslgrohtnogrt dectsoocmrpipbaitinriosens translate english descriptions to german
Descriptions original english original german translated to german combine descriptions to pairs english short description exists 587,580 5,089 manual y 3,833 annotate matches same entity name for possible matches
Descriptions dteotresagcnnergisrpllimastihotaenns traooonrrrsiiilgggaiiitnnneaaadlll eegtonneggrgmlleiissrahmhnaowwndikimkwiiiki dectsoocmrpipbaitinriosens translated to german odm 19,165 identifier. The Crunchbase Open Data Map 5 (ODM) and the Crunchbase Snapshot 20136. The Crunchbase Snapshot 2013 contains 320,719 company records with an English short and long description (short_description and overview). 4,793 records contain a short and a long description and can be used as matches for the evaluation dataset. The Crunchbase ODM contains 589.343 company records with an English short description (short_description), but no alternative description. We manually labeled the Wikidata and Crunchbase ODM datasets to get more match samples between company descriptions. For this, we extracted all record pairs that have names that match exactly, in the hope of finding match candidates with a high probability of actually being matches. Of the 5,089 pairs found, we have labeled 3833 pairs as actual matches. From the three data sources, only wikidata contains German texts. For the creation of the evaluation set with exclusively German descriptions, every available English text is translated using a machine learning model. For this purpose a pre-trained transformer from the Python library Huggingface/Transformers [28] is used. The model MarianMT7 was chosen, because it delivered an acceptable calculation time and quality in own tests. Minor grammar mistakes are no problem as long as the meaning of the text is not changed. The quality of the translated texts is considered suficient for the application. By translating the English Wikipedia texts, the Crunchbase ODM short description and the CrunchbaseSnapshot short and long descriptions there are exactly two German descriptions for each entity of the three tables available for training and evaluating models. After collecting, labelling and translating, the following amount of matches and the available attributes are: • 20,126 entities from Wikipedia with original English, original German and translated
German descriptions 5Powered by Crunchbase: https://data.crunchbase.com/docs/open-data-map 6Crunchbase 2013 Snapshot © 2013, Creative Commons https://data.crunchbase.com/docs/2013-snapshot 7Precisely: Marian MT Helsinki-NLP/opus-en-de
tion gateway allowing users to bile desktop and server applications to publish content publish text, photos, video, au- (photo, video, audio, text, file) and have it distributed out dio and documents. to social networks, websites and blogs around the world.
• 4,793 entities from crunchbase snapshot with original English short description, original English long description, translated German short descriptions and translated German long descriptions • 3,833 entities from manually annotating matches between crunchbase odm and Wikipedia with original English Wikipedia, original German Wikipedia, translated German Wikipedia, original English crunchbase and translated German crunchbase descriptions Due to the diferent available description texts for entities within a data source, text pairs with matches can be created without a manual annotation process. For example, after collecting the Wikipedia data and translating it, one English and two German description texts are available for each entity. In combination with each other, this results in a total of 3 training examples (English + German1, English + German2, German1 + German2). For the cross-language evaluation set, all of those combinations can be used, whereas for the pure German data set only the text pairs with German descriptions can be used. Figure 1 shows the whole process of data sources or description texts that can be combined by the automated and manual annotation process with the number of matches found. The set of matches resulting from the combination is 32,488 for the purely German-language and 108,451 for the mixed language dataset. For negative samples, random texts of the set are chosen in a way that every entity in the dataset has the same amount of matches and non matches, equally distributing the classes among the dataset. This doubles the amount of samples in each dataset. While the texts extracted from the three data sources generally were preserved as is and were not changed, there is one notable exception: In 50% of the cases the name of the entity was removed from the respective description text, if a name was available. The goal of this measure is to force the developed models to abstract beyond the entity name and not to restrict themselves to this single property. This essentially leads to 25% of samples where it is guaranteed that neither descriptions contains the company name, 50% of samples where exactly one description has the company name removed (25% and 25% of samples that are completely unchanged.
For the German language as well as the mixed language dataset, 10% of the examples have been retained for validation, and models can be trained on the remaining 90% (if the learning process is supervised). From the manual annotation of possible matches between Crunchbase ODM and Wikipedia, 708 non-matches of entities with the same name were also identified. Due to the same name, the matching of these entities poses a special challenge for the developed models and is only used in the validation data set. The goal of this procedure is to measure a degree of abstraction beyond the company name. Good models should be able to exclude a match despite the same name due to the remaining text information. A extract of the resulting evaluation set is shown in the Table 5. 4. Design space for textual RL and preliminary results Our research’s overall goal is to use natural language text attributes for RL to get better match results. This will be developed for the entity company using the proposed evaluation set containing company descriptions from diferent data sources. Following up on the research done, two research questions have been defined to achieve this goal: • What is the best method for matching company descriptions? • How does the use of the company description afect the overall performance of integrating data sources with the entity company?
Laboratory experiments will be conducted to answer the first research question. In
addition to the Abt-Company data set in the widely used Deepmatcher framework, company data
sources such as Crunchbase and Wikidata will be part of the experiments. For this, we will use
the proposed evaluation set and may add more data sources to obtain generally valid results.
From the results of a literature review, the following procedures were derived, promising for a
solution to the problem. For supervised approaches, pre-trained language models may be
finetuned for a company description matching task. Alternatively, to fine-tuning models that were
primarily trained on company descriptions, training on diferent RL tasks or domains (such as
product titles) might provide better results, as indicated by [
Additionally, one could experiment with extracting keywords from the descriptions and match them separately using string similarity metrics that have proven to be working on atomic attributes such as Jaccard or Jaro-Winkler. For example, one could use named entity recognition (NER) models to identify entities described in the texts (like companies, persons, and geographical locations), extract additional descriptive attributes assigned to those entities and try matching these using string similarity measures. Instead of comparing the entities and attributes extracted themselves, one could also perform classification (either using the full text or extracted attributes as a baseline) to map fixed classes, e.g., sectors. This has strong similarities to traditional matching approaches that primarily use semi-structured data (such as sector or category names that do not follow a predefined taxonomy) and thus should provide good baseline results for matching entities using descriptions. Table 6 shows the preliminary results on the mixed language validation dataset after implementing and evaluating selected approaches mentioned above without further optimization except some tuning on the score thresholds on which a match is predicted. Thus, these models can be regarded as baseline models that more complicated solutions may be compared. For the language models the Huggingface/Transformers library for python [28] has been chosen, because it supports the direct application and further training of diferent NLP transformer architectures for uniform benchbarking. The experiments that use language models were run on a single NVIDIA GeForce RTX 2080 TI GPU. The training time as well as the validation time for the total dataset have been added to provide a quick overview over the eficiency of the approaches used, since runtime may be a crucial criterion for practitioners and researchers alike. The sklearn [29] accuracy score8 has been chosen as a evaluation metrics for measuring the validation accuracy of the diferent approaches (see table 6). It measures the amount of samples that were correctly predicted (true negatives + true positives) against the total amount of samples. It can be seen, that the (1) fixed choice model that predicts a non-match in every case reaches an accuracy of 55.7% because the validation dataset contains slightly more negative samples due to the extra samples from the manual annotation process present. (2) Jaro-Winkler reaches an accuracy of 55.7% and is therefore not better then a fixed choice model no matter what threshhold was chosen. The word based (3) jaccard reached a validation accuracy of 69.3%, although the descriptions in a lot of examples are coming from two diferent languages (English and German) and thus should not contain the same words in a majority of cases. Experimenting with purely English based pretrained language models (4) BERT and (6) RoBERTa do achieve with 57% and 58% a lower performance than the (3) jaccard approach when the last hidden layer is used in conjunction with the cosine similarity. Using a multilangual (5) BERT the accuracy reaches 78.3%, which was in line with our expectations as we thought a multilangual model should outperform its monolingual counterpart on a multilingual dataset. While the tested (7) XLM specifically was pretrained on English and German texts, it could only outperform the monolingual (4) BERT and (6) RoBERTa by a margin of 2-3%. It could not deliver the same results of the multilingual (5) BERT approach. This shows, that the model architecture clearly has an impact on the performance of this task and further experimantation needs to be done. Using a supervised approach, fine-tuning a (8) BERT model for the specific RL task yielded the best model so far with 80.6% accuracy. Taking the additional amount of work for creating training data and the required model training time into consideration, it is interesting that the accuracy is only marginally better compared to the (5) BERT unsupervised approach that works out of the box. However, the results may improve when further hyperparameter-tuning is applied. Apart from our experiments with single model approaches, combinations of approaches should also be explored in the future. Examples of combining the approaches are shown in 8https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html already used by our RL system (such as names, adresses or legal forms among others), we want to answer not only whether it might be worth using descriptions in the RL process but also in which conditions, e.g. only when neither the attribute name nor adress is present.
This paper describes the ongoing research work to include natural language text attributes in RL. This goal is to be explored using company descriptions and machine learning methods in the domain of language models. This paper provides a qualitative literature review that shows research gaps and is used to identify potential methods and approaches to include natural language text attributes in RL. Subsequently, a gold standard evaluation and training set has been proposed containing company descriptions from Wikipedia and Crunchbase. Furthermore, a design space exploration was performed to show possible approaches to solve the task of RL using natural language descriptions. Preliminary results have shown, that language models are capable of solving this task and generally perform better then traditional string matching results. The results have shown an interesting opportunity to further optimize the applicability of machine learning methods in RL systems, reducing the amount of human work required in linking data sources to knowledge bases. Within further laboratory experiments, additional proposed methods and approaches and combinations of approaches in RL pipelines will be applied to the gold standard data set and evaluated. Finally, the evaluation will be carried out using the newly developed method to include natural language text attributes within a field study in a real and practice-relevant RL workflow with an industry partner. Our paper contributes to theory and practice by researching natural language text attributes in RL. The improvement of the RL process will optimize data integration in practice. The limitations of our paper also ofer opportunities for future research. We focus on the attribute company description while also conceptualizing future solution methods. Further natural language text attributes should be explored and our best approach should be transferred to these attributes and be evaluated. [17] U. Brunner, K. Stockinger, Entity matching on unstructured data: an active learning approach, in: 2019 6th Swiss Conference on Data Science (SDS), IEEE, 2019, pp. 97–102. [18] T. Gschwind, C. Miksovic, J. Minder, K. Mirylenka, P. Scotton, Fast record linkage for company entities, in: 2019 IEEE International Conference on Big Data (Big Data), IEEE, 2019, pp. 623–630. [19] D. Javdani, H. Rahmani, M. Allahgholi, F. Karimkhani, Deepblock: A novel blocking approach for entity resolution using deep learning, in: 2019 5th International Conference on Web Research (ICWR), IEEE, 2019, pp. 41–44. [20] H. Nie, X. Han, B. He, L. Sun, B. Chen, W. Zhang, S. Wu, H. Kong, Deep sequence-tosequence entity matching for heterogeneous entity resolution, in: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019, pp. 629–638. [21] A. Primpeli, R. Peeters, C. Bizer, The WDC training dataset and gold standard for largescale product matching, in: Companion Proceedings of The 2019 World Wide Web Conference, 2019, pp. 381–386. [22] C. Zhao, Y. He, Auto-em: End-to-end fuzzy entity-matching using pre-trained deep models and transfer learning, in: The World Wide Web Conference, 2019, pp. 2413–2424. [23] Y. Li, J. Li, Y. Suhara, A. Doan, W.-C. Tan, Deep entity matching with pre-trained language models, arXiv preprint arXiv:2004.00584 (2020). [24] V. Meduri, L. Popa, P. Sen, M. Sarwat, A comprehensive benchmark framework for active learning methods in entity matching, arXiv (2020) arXiv–2003. [25] R. Wu, S. Chaba, S. Sawlani, X. Chu, Zeroer: Entity resolution using zero labeled examples (2020). [26] D. Zhang, Y. Nie, S. Wu, Y. Shen, K.-L. Tan, Multi-context attention for entity matching, in: Proceedings of The Web Conference 2020, 2020, pp. 2634–2640. [27] P. Mayring, T. Fenzl, Qualitative Inhaltsanalyse, in: Handbuch Methoden der empirischen
Sozialforschung, Springer, 2014, pp. 543–556. [28] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, A. M. Rush, Huggingface’s transformers: State-ofthe-art natural language processing, ArXiv abs/1910.03771 (2019). [29] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830.