Organizations need to ensure the quality of data that is used for analytics and to maintain its consistency across multiple analytical and operational systems. Master data is a term that refers to domain-specific data concerning business objects, crucial for organization operation, e.g., contracts, suppliers, employees and so on. Usually, such source data is scattered around diferent applications across the organization and is of varying quality. Master Data Management (MDM) is a set of practices, information management methods, and data management tools intended for producing accurate, consistent, and complete master data. At the same time, data management tools play a vital role in setting up and supporting MDM-related processes. In this paper, we describe the Unidata platform: a toolkit for constructing MDM solutions. Its modular architecture allows to construct solutions tailored for a specific domain and case requirements. We start with a short introduction to MDM, discussing its aims, user-facing benefits, and approaches to working with data. Then, we describe the architecture of the Unidata platform and present the data storage approach and query processing algorithms. Finally, we discuss use-cases and put forward our thoughts regarding the future of MDM systems.
tracking data lineage and so on.
Master Data Management (MDM) solutions aim to adContemporary companies and public institutions manage dress these technical issues by consolidating available huge volumes of data, which is now a strategic asset [1]. information while interacting with existing systems of Data became an enabler of organization business models organization in a minimally intrusive way. As a research and value propositions [2]. At the same time, each or- discipline, MDM is an actively developing area that conganization usually possesses a complex data ecosystem, cerns all aspects of enterprise data management practices. which makes it hard to make use of information con- Its practical counterpart is closely monitored by Gartner, tained in it. For example, a recent study [3] concerning which releases yearly reviews describing established and local governmental organizations in Denmark showed promising products. The recent publication of the 2nd that one of the significant obstacles to using this data edition of the fundamental reference [4] describing all is the lack of its overview. Thus, if providing even an aspects of MDM is also worth noting. overview is hard, then eficient use will require much Large organizations that are principal customers of more efort which will include ensuring data quality (du- MDM vendors have many individual characteristics, and, plicate detection and data conflict resolution), setting up consequently, individual pressing tasks. Therefore, by ETL pipelines, providing proper metadata management, implementing MDM they aim to fulfill these tasks, thus adhering to the organization pull strategy [5]. The other strategy — technology push — is essentially large-scale adoption of a new technology based on the belief in its usefulness instead of focusing on particular tasks at hand.
The organization pull approach significantly reduces the cost of MDM adoption by narrowing its application area.
Moreover, its iterative nature enables progressive buildup of MDM functionality via step-by-step incorporation of diferent organization business areas. However, implementing MDM in the organization pull manner requires lfexibility of the MDM toolkit used to build the solution, and major MDM toolkits fail to provide it since initially they were built as monolithic systems.
Usually, the structure of large organizations is the result of lengthy historical processes, which may include
1. Tune-ability. An MDM platform should be adaptable to various software ecosystems, take into account specific priorities and issues of the concrete organization, and comply with external constraints. This will ensure that target MDM solutions meet the organization needs. 2. Component-based architecture. In order to provide the required flexibility, the MDM platform architecture has to follow the componentbased principle. The platform should consist of modules — i.e., building blocks which can be freely combined together in order to create efifcient and specialized MDM solutions. 3. Metamodeling. A metamodel describing models of individual areas can be used to handle combinations of diferent domains. This will provide the product with the required extensionability. several acquisitions and mergers of other companies or organizations. Another important feature of such organizations is a unique software ecosystem comprised of core technologies from various vendors (e.g. SAP, IBM, Microsoft). Next, specific priorities and issues of the organization should be taken into account as well. Finally, there are various external constraints which an organization has to follow and which define the structure of the business processes. For example, organizations in diferent countries may have diferent rules, e.g. either a declaratory or an authorization system. Building an MDM platform upon these principles
Next, the need to fulfill specific tasks of the organi- makes it possible to achieve synergy between individual zation leads to a specific subset of the MDM functional- MDM instruments such as data quality, data provenance, ity of the solution being “clipped out” from an abstract metadata management, and so on. That is, building a general-purpose MDM toolkit. It may be due to tuning to solution from tools that are intended to work together organization software ecosystem or implementing spe- from the start (as opposed to integrating a set of diferent cific functionality. Such functionality usually concerns tools) will result in shorter development times and soluareas that border on MDM or that may be completely tions of better quality. Furthermore, it will be possible external to MDM. Obviously, organizations do not want to deliver diferent versions of solutions with diferent to have functionality and code that they do not need for functionality (e.g. standard vs enterprise edition) and the task at hand. Finally, there is a trend of implementing prices. The overall idea of this approach is presented in various enterprise applications in the cloud in order to Figure 2. The core set of modules has been released as lower expenses. an open-source version1 of the platform. It is published
Multi-domain MDM, i.e. MDM covering several key under the GNU GPLv3 license. business entity areas (products, vendors, employees and Overall, the contributions of this paper are: so on), is the state-of-the-art [6] approach. Aside from the 1. A description of the Unidata platform architecaforementioned generic domains there may be a num- ture. ber of business-specific ones, such as patients in case 2. A discussion of data storage and processing algoof a hospital or an inventory of land plots managed by rithms. forestry. Furthermore, each customer may have its own 3. A list of use-cases and our vision of the future of set of domains, with its specifics. In order to support com- MDM. binations of diferent domains, an MDM platform should be able to describe them inside itself and be suficiently The structure of this paper is as follows. We start with independent from any particular one. the description of the background in Section 2. In it, we
In this paper we describe the Unidata platform — a provide basic definitions and describe user roles. Next, core technology of an industrial product line [7] that in Section 3 we discuss several use-cases in order to illuswas used to produce dozens of target MDM solutions trate benefits that can be obtained by using such systems. for large organizations. These solutions are focused on All presented use cases are real and describe actual MDM the specific needs of organizations such as specialized deployments that were driven by the organization pull business segments, heterogeneous IT infrastructure, and strategy. Section 4 contains the description of the platparticular business tasks. The platform has received an form architecture and its modules. The way data is stored honourable mention by Gartner in “Magic Quadrant for
platform-deploy and how queries are processed is discussed in Section 5. Next, present our view on the future of MDM systems and describe related work in Sections 6 and 7. Finally, we conclude this paper with Section 8. 2. Background cannot be copied or “moved” for various (including legal) reasons. 2. Consolidation. Data is uploaded into the common repository on a regular basis, appropriately processed, and then the hub itself provides data consuming systems with access to this data. However, new data is supplied by live data source systems, i.e., new data is uploaded to the hub on a regular basis. 3. Centralization. This architecture is very similar to the previous one, but here the hub takes over data upload as well: i.e., data is uploaded once, and then all changes are performed on the hub itself, thus turning all systems that initially were data sources into data consumers. 4. Coexistence. This architecture implements a combination of the Consolidation and Centralization for diferent master data of an organization. Additionally, if some data fragments are not “movable”, they can be handled using the Registry.
According to David Loshin [9], Master Data “are those core goals of MDM systems is to create and maintain a core business objects used in the diferent applications single version of the truth for an entity. The information across the organization, along with their associated meta- which constitutes it is stored in multiple sources and data, attributes, definitions, roles, connections, and tax- thus, it should be assembled. All information concerning onomies”. Examples are product data, customer data, a particular entity is called the golden record. supplier data, location data, party data, reference data, Data models and Metamodel. In order to support asset data, employee data, ledger data, and vendor data. a particular domain, the objects which the platform will
In turn, Master Data Management is “a set of practices, work with must be defined. A Metamodel consists of information management methods, and data manage- the description of the data itself (data schema) and rement tools to implement the policies, procedures, ser- lated procedures. For example, it is possible to add a vices, and infrastructure to support the capture, inte- supplementary metamodel of data quality for a particugration, and subsequent shared use of accurate, timely, lar domain, e.g. specific duplicate detection procedures. consistent, and complete master data”. In other words, a metamodel specifies how individual
The purpose of an MDM platform is to obtain data data models will be created and processed. from source information system(s), then process it to en- A registry is a collection of records that are related to sure data quality by, for example, performing data dedu- some entity, such as a person, an organization, etc. This plication, filling in missing values, removing outdated information comes from many diferent source informainformation, etc. Eventually, it must obtain a golden tion systems. A registry has a schema which consists of record for each item — an error-free version conforming a list of its attributes and nested entities. Similarly to tato the defined quality criteria. bles, registries can have references to other registries, e.g.
How exactly data is processed is defined by the MDM suppliers and items: each supplier can have references implementation architecture/style [6, 10] (sometimes to items which it sells. called the data hub architecture). Gartner identifies four A lookup table is a referential table that contains data approaches, presented in Figure 1). which is rarely changed, but at the same time frequently used. For example, lookup tables may be created to list 1. Registry. The hub does not contain the data itself, countries, timezones, currencies, and so on. Similarly to instead storing only the corresponding references registries, lookup tables consist of attributes. (indexes). This approach is relevant for data that System of a Record (SOR). During the golden record assembly process, the same attribute may turn out to be stored in several sources, having diferent values. In this a data steward is the person who is accountable for the case, either a system of a record or a conflict resolution quality of the data. However, in practice, a data steward process should be set up. SOR is a primary data source is not necessarily a single person, but rather, a group of which contains the “true” value. several people. Each of them may be responsible for a dif
Validity Period — is an interval in which the data ferent key business area or even a part of it. The Unidata describing an entity is valid. Each golden record may platform supports two types of users that implement data have several validity periods which should be taken into stewardship: account while querying the data. Moreover, there are two temporal dimensions: time of an event and time • Administrator manages the platform in general. of addition of this new version of information into the Their duties include data model administration, system. This leads to a need for a special scheme for classifier 2 administration, managing rules of dumanaging this information. plicate detection and so on.
Consider the example presented in Figure 3 which de- • Data operator manages records (e.g. database scribes the history of Saint Petersburg’s name changes. population), registries and lookup tables, and parThe Y-axis denotes update times, while the X-axis shows ticipates in related business processes. the validity periods. In this example, we assume that before 2019 the system contained only the basic version Consequently, there are two types of interfaces, for of name information: from 1703 until the present time each type of user. Finally, Unidata platform supports role(2019), the city was called Saint Petersburg. Then, some- based access model, where each user may be assigned where between 2019 and 2020, the knowledge of the rights to perform operations (e.g. CRUD) with various Leningrad name was added into the system, and later, objects. somewhere before 2021, a similar update was done for Petrograd. 3. Use Cases
A user may pose queries like “what was the name of the city in 1921”. There are two possible answers: up until 2020 (point 1 ) the system would have returned “Saint Petersburg”, which was the correct answer at the time. Currently, (point 2 ) it should output “Petrograd”.
Thus, the golden record should be constructed on-thelfy, taking into account updates in two temporal dimensions which is done by looking into the origin history (which is discussed later).
completed using the Unidata platform, driven by the organization pull strategy. They clearly illustrate the concrete goals of organizations, as well as the benefits the platform has provided to the final users.
They also demonstrate that each particular deployment needs a diferent set of data governance tools, i.e. they highlight the importance of component-based architecture, which we have discussed in the Introduction.
From a high-level perspective, an MDM system produces a new role for a human user — a Data Steward. According to [9], a data steward is a person responsible for collecting, collating, and evaluating issues and problems with data and the data life cycle. Their duty includes managing standard business definitions and metadata. Finally,
The first case is a system for managing logistical resources of a large company in the energy sector. The 2In this paper, “classifier” refers mainly to a product class hierarchy, such as the Global Product Classification [11]. covered domains included raw materials, equipment, and replacement parts. The purpose of this system was to: • Provide high-quality data for business processes that cover equipment maintenance and repairs, inventory management. • Consolidate available information using data standardization and unification.
The next case is a product catalogue development for a large telecommunication company. The goals of the project were to consolidate: • information regarding product ofers for diferent customer segments, • information related to service availability, • financial information from the billing system and accounting records.
The aim of the system was to find those clients in the client database that have popular social media accounts and a lot of followers. The company wanted to improve their loyalty by ofering additional discounts and performing various other actions in order to obtain more customers from their follower bases. The core data governance tools that were used were data enrichment and consolidation.
account of a city resident. It was necessary to integrate it with various federal and regional information systems. The reason for this was to enable exporting relevant information concerning vehicles, real estate, bank accounts and so on. The primary focus of this project was access control, security, and ensuring real-time as well as publish/subscribe model master data acquisition.
tional company focused on energy and heavy industry
As the result, a product hierarchy (a product tree) that areas. Since this company has hundreds of thousands of contains various product details (including financial ones) clients from all over the world, the procedure of adding was constructed. It is now utilized by the sales depart- a new client was very complicated. Before applying our ment and financial oficers. MDM solution, it took 21 days on average. However, it has shortened to only eight days after. The solution 3.3. Case 3: data consolidation for a automatized various checks, finding final beneficiaries in corporate hierarchies, and centralized information input.
transport company This project mainly concerned data inventory, data This project was dedicated to consolidation of items and quality (duplicate search), and implementing real-time services purchased by a large transport company. The access to master data in order to speed up a particular goal was to unify information contained in several dif- business process. ferent product classifiers and to produce a list of services ofered by contractors. 4. Architecture
The stakeholder of this MDM solution was the procurement department of the company. After the deployment, items and services that had diferent prices were identi- Now, let us turn to the architecture of the Unidata platifed and analyzed. This resulted in creation of monetary form. It follows the component-based principle, which metrics, i.e., calculated total savings on purchases. The means its building blocks (modules) can be freely comsystem made it possible to perform automatic purchases bined with each other in order to obtain a solution with with the minimal available price. the desired set of features.
This data consolidation project relied on the publish/subscribe model.
A module is a self-suficient set of functionality that is intended for solving a particular problem. Each module contains a number of services that cover parts of this functionality. For example, the Meta module which encompasses all metadata-related activities contains services that cover managing lookup tables, registries, units of measurement, enumerations, etc.
In a broad sense, a Pipeline is a sequence of operations which are performed either on data or on the model. Pipelines implement dataflows, which consist of service calls and may contain utility nodes such as branching, parallelism (applying an operation to each record inside the batch), calling another pipeline, and so on.
Each module contains services that work with the data and services that interact with and modify the model. Therefore, pipelines may modify not only the data, but the model itself too.
Thus, modules and pipelines implement tuneability and composition aspects discussed in the Introduction section.
sented in Figure 4. It consists of four main components:
plement core services of the platform and minimally depend on other modules. These services are: system boot, job batches, and data types. Modules of this component are rarely modified and are never adapted for a particular deployment or domain. 2. Storages concern everything related to data stores that are employed by the platform. These modules make it possible to abstract functionality required by the platform from the specific DBMS and to restrict and simplify it (as not all DBMS functionality is required by an MDM solution). 3. MDM includes all modules that implement basic MDM functionality, such as metadata management, rules for computing master record alternatives, data quality management, duplicate detection, and business process implementation. This component is responsible for integrating and synchronizing all necessary data stores that reside on the previous level. 4. Extra MDM implements advanced MDM functionality. It contains modules that are either advanced variants of some existing module from the MDM component, or modules that provide functionality usually implemented “outside” of MDM. An example of the first case is the following: Match Extra is a sophisticated machine learning-based inexact match module and the
one. The second case is illustrated by the data delivery set. Usually, in MDM systems the ETL is implemented separately as a standalone application. In our case, it is possible to have it inside the system. The same idea is employed with the Pub/Sub module, which make it possible to implement sophisticated patterns of sending records to various consumers. This is an advanced component which is not present in the open-source version.
These components are organized in a hierarchical way, which means that: 1. Reference — a reference which ensures that for each individual record there can be only one referenced object per each validity period. 2. Many-to-many — a classic many-to-many reference. 3. Contains — a reference which is created by user pointing to an entity with all attributes filled in. This type is used to define entities which do not exist without its parent entity.
system works with, as well as important modules and their functionality relevant to the entities.
System. All modules that constitute the platform share a single interface that contains methods of initialization, configuration, launch, and verification. The system module orchestrates system boot process and ensures that all modules have everything necessary for trouble-free launch and operation.
Data Types. An attribute is a basic entity representing some key aspect of a stored object, similar to attributes in RDBMS tables. In Unidata, Registries, lookup tables and references may have attributes.
There are three attribute types in the system:
The platform lets the user browse both direct and backward references (those that point to a specific record). It also provides rich search capabilities that allow the user to query attributes of references. • components residing in the bottom levels of the The Core module contains interfaces and abstract imifgure are more low-level, they contain basic fea- plementations for all data types that the platform uses. It tures, essential for implementing high-level func- also implements metamodel support, the need for which tionality, and, was discussed in the Introduction. Finally, this module • components interact with each other in hierarchi- supports additional services which may be used by other cal way, i.e. their interactions rarely “jump” over modules such as roles, logging, license verification and the immediate neighbor. so on.
The next important entity is a draft. Drafts are an essential part of MDM since master data needs to be synchronized over data producers (sources) and data consumers. Such synchronization frequently results in the creation of temporary intermediates. These need to pass various reconciliation procedures and be agreed upon in order to become fair copies. Drafts may also emerge as a result of conflicts that arise during record consolidation process, after various data quality procedures, etc.
Drafts may concern individual items or the model.
Drafts can have revisions: all of them are stored in the database in a serialized form.
Therefore, drafts need to have general support: operations such as creation, publication (transforming into fair copy), and merge must be implemented. Draft is a module that enables all kinds of drafts in the platform.
Logging capabilities. The Unidata platform supports extensive and configurable logging capabilities. It is possible to record user actions such as record upsert, user 1. Simple attribute is a basic data type which de- login or logout, etc. It is also possible to select the logscribes some entity. Its type can be: string, nu- ging level. Coarse-grained logging logs errors only, while meric, boolean, file, date and time, reference, and ifne-grained enables the logging of search and browsing. enum. Enum is a domain-specific enumeration Storages and Adapters. For its operation, Unidata which describes mutually exclusive states of a needs a number of storages, e.g. data storage, graph real-life entity, e.g. a subject which can be either a storage, match storage, and so on. In order to achieve legal person, a natural person, or a self-employed lfexibility and independence from a particular database person. vendor, a collection of adapters was implemented. For 2. Array attribute is used to represent a series of example, the platform can use either Neo4J or OrientDB similar entities, such as property owners. for graph storage. 3. Complex attribute is used to represent nested Match storage is a module that concerns data repretree-like structures. It can contain simple at- sentation for performing data deduplication. There is tributes, arrays, and other complex attributes. a separate representation for both records and clusters that difer from the original data by, for example, omitted ifelds that do not participate in the matching. For the deduplication itself, matching engines such as Senzing, Elasticsearch or RDBMS can be used.
Match and Match Extra. While match storage implements basic matching functionality, this module implements deduplication in MDM entities: in the data itself, drafts, business processes, etc. The Match Storage module does not possess any specifics regarding these entities and therefore separate modules are needed. These modules also let the user define deduplication procedures and manage them. The Match module operates with rules, while Match Extra employs various machine learning approaches.
Workflow and Workflow Extra. In MDM, it is frequently necessary to implement various business processes that may involve various oficials and span mul- Figure 5: Tables used for data storage in the Unidata platform tiple departments (e.g. resolving a surname conflict in various personal documents). To run such workflows, a subset of the BPMN 2.0 standard that includes events, ac- ter data. Classifiers are tree-like data structures in which tivities, and gateways is supported. For this, integration each node describes some entity and may contain variwith several third-party engines such as Activiti BPMN ous attributes (e.g. see [11]). They are frequently used and Camunda BPMN is implemented. in MDM domains, for example, to describe a hierarchy
Workflow Extra extends basic capability by enabling of product types. Unidata platform enables interactions the implementation of various scenarios involving ma- of master data with such classifiers (e.g. arranging arrivchine learning approaches. It also provides the ability to ing data according to a classifier) and supports various perform SLA enforcement for data operations. operations on classifiers themselves with versioning.
Meta is a module that is responsible for metamodel Data Catalog implements a comprehensive body of (schema) management such as creating and editing en- knowledge concerning all data of the organization. It intities (attributes, references, etc). It provides a GUI that cludes provenance, current storage location, its domain, allows users of the MDM solution (data stewards) to work use, relation to other data and so on. This module is of with the metamodel while automatically generating the critical importance since an organization may have huncorresponding API for data access. dreds of source information systems and manual tracking
Data Quality and Complex Data Quality. These of such information would be impossible. modules are responsible for ensuring data quality — checking data for errors and performing data enrichment. While the Data Quality component works with a 5. Data Storage and Processing single record, Complex Data Quality may involve several records (e.g. checking aggregate values). 5.1. Requirements
The core instruments are the enrichment and validation rules. The enrichment rule enables using attributes The specifics of the platform’s application scope lead of other registries in order to generate new attributes of to the following requirements imposed on storage and the target registry. The validation rule does not generate processing of the data. any data, instead, it generates an error if it is violated by 1. Tombstone deletes. Information should be input. Each rule can have a number of ports which may deleted only when it is absolutely necessary. Real be either incoming or outgoing. Rules take input from deletions should be performed only by an adminthe incoming ports and produce results into the outgoing istrator while regular users should not actually ones. delete data. Instead, if a regular user attempts to
In the UI, a user can construct rules using a special perform a deletion, the data should be marked UPath language (an XPath derivative) and a number of as deleted and the system should take this into pre-defined functions such as string manipulation func- account. tions (case conversion, concatenation), a boolean func- 2. Versioning support should be pervasive. Users tion and so on. It is possible to upload custom rules that and administrators should be allowed to reconare implemented in Java or Python. struct previous versions of any object. At the
Finally, rules can be grouped into sets and applied on same time, querying using validity periods, disa per-set basis. cussed earlier in Section 2.2 should be supported.
Classifiers. This module employs classifiers for mas
any operation. For example, if a bulk-loading operation inserted records into a database, it should be possible to reverse it by removing newlyinserted records while keeping the rest.
In order to meet these requirements, an approach based on the following three logical tables was used. Each table represents an entity: 1. Etalon_id — an identifier of etalon which this
origin belongs to. 2. Source_system — an identifier of the system
where this record came from. 3. External_id — an identifier of the record in the
system where this record came from. 4. Enrichment — a boolean flag which shows whether this origin is a result of the enrichment rule. 5. Similarly to etalon, attributes update_date and update_by that bear the same semantics but pertaining to this particular origin. 1. Origin_id — a reference to origin whose part of vistory is contained in this record. 2. Revision — a version number, which is needed to ensure versioning, described in Section 5.1. 3. ValidFrom and validTo attributes — a validity period of the vistory entry. 4. Data_b — serialized data in the XML format. 5. Operation_id attribute, similar to the one in the Etalon table. It can be used to, for example, find and cancel a modification action. However, in this case, cancelling will afect only this particular update, while in the etalon case it will cancel the creation of the whole record.
Finally, the vistory table contains the data itself. Its important attributes (apart from the basic ones) are the • Etalon is the metadata of the golden record itself. following: • Origin is the metadata related to system from
which the record originates. • Vistory (version history) is the validity period of
origin, which in turn may have revisions.
2. Shard — an identifier of the shard where the record resides. Each of these logical tables may be physically represented by a collection of horizontal partitions (shards). 3. Status — may be ACTIVE, INACTIVE, or
MERGED. These are used to describe the status of the record and to support tombstone deletes. Note that the vistory table contains no attributes conThus, INACTIVE means that the record was cerning updates, since for each update a new record is deleted, while ACTIVE indicates that it is valid. formed. Next, setting the status attribute INACTIVE in The MERGED status indicates records that were case of the vistory table makes it possible to mark some used in the duplicate resolution process and no validity period as “deleted”.
longer contain valid information. To illustrate the idea, in Table 1 we provide a vistory 4. Create_date and Created_by — when and by table fragment for the city name example described in whom this object was created. Figure 3. There are three records and all of them belong to a single origin. Therefore, there are no data conflicts
The relations between these tables are shown in Fig- possible and the process of calculating the etalon data is ure 5. The links with empty arrowheads denote “shared” rather straightforward. First, it is necessary to perform a attributes and full arrowheads show PK-FK relationship. cross-product of all time periods, and then to select the The etalon table additionally stores: row that fits the necessary combination of creation and queried dates. The result of the period cross-product for 1. Name of the registry or lookup table it refers to. the considered example is shown in Table 2. Thus, it is 2. Update_date and update_by attributes, which possible to find an answer for points 1 – 4 . contain the date and the last user who updated Finally, there are complex rules of attribute interaction this data, respectively. inside the etalon →− origin →− vistory hierarchy. For 3. Operation_id attribute which contains the identi- example, if adding a new revision of a particular origin ifer of the operation during which this record was leads to a recalculation of the etalon, then its update time created. It is needed to support the provenance is recalculated too. Next, status updates are propagated requirement. in a bottom-up fashion: e.g. if INACTIVE is set for a Apart from the basic set of attributes, the origins table vistory entry, then its etalon will have to be recalculated has the following additional ones: and may have to be set INACTIVE too. However, if etalon is set INACTIVE, then there is no need to set all vistory entries INACTIVE since they will never be reachable for queries.
The same approach is used to represent not only records, but other entities as well, e.g. references and classification results. It is possible to create a golden record for a reference. Consider a case where a reference has two origin systems, and in the first system, there is a 1 of values and in the second a 2. Using the BVR or BVT algorithm, it is possible to create a golden record, by, for example, intersecting them.
Each vistory entry has a date of creation and a validity period, which are used to construct golden records, as was demonstrated in the previous section. However, what if there are two origin systems which have the same attribute and there is a data conflict, i.e. each system reports diferent values? In other words, how is a SOR selected for this attribute?
For this, two special algorithms — the BVR (Best Value Record) and BVT (Best Value of the Truth) — were devised. The BVR algorithm is used to construct a golden record by resolving data conflicts for all attributes of an etalon using a set of thresholds, one for each origin system. The general idea of this algorithm is to pick values from source systems that have higher weights.
More formally, the BVR algorithm is as follows. 1. For each origin obtain its latest version, except the ones that have the MERGED status. 2. For each source_system select the latest version according to its creation_date. 3. The golden record will be created out of the record that pertains to the source_system with the maximum weight.
1. Similarly to the BVR, obtain the latest version for each origin, except origins that have the MERGED status. 2. For each attribute sort obtained versions according to the weights of sources and update_date 3. Compute values of each attribute by iterating over versions obtained on the previous step: a) if the value of the attribute is not null, then use it for etalon construction.
b) otherwise, proceed to the next iteration.
to null values and therefore handles them diferently.
To illustrate both algorithms, let us consider the example presented in Table 3. In this table, we denote entities selected on each step in bold. The first seven rows contain the data itself. Note that for presentation purposes we have joined all three tables that contain it.
Our first step (regardless of the selected algorithm) is to select most recent vistories for each origin, which is done inside the DBMS. Rows 15–16 contain the answer.
On the second step it is necessary to compute validity periods, which is done in Java code (all consequent computations are performed in Java code, too). Applying cross-product to validFrom and validTo attributes, we obtain three periods: (1989–2000), (2000–2005), (2005–9999). Rows 20–23 contain our data partitioned by validity periods.
One can note that there is a data conflict, namely, rows 20 and 23 concern the same period and contain diferent values. Periods of rows 21 and 22 do not have alternatives and therefore may be used as is.
In order to resolve the conflict, the BVR algorithm will require weights of source systems. Suppose that they are as follows: source1 = 50, source2 = 100. In this case, the record on row 32 will be selected with all its attributes.
The BVT algorithm will additionally require a set of attribute weights. In this example, we have weights for a single attribute — year_of_birth, which are as follows: source1 = 100, source2 = 50.
These attribute weights are used to override source weights that act over the whole record. Therefore, the year_of_birth attribute will be set as 1991, while name attribute will be set as John.
The BVT algorithm additionally follows the “null is not a value” rule. It never picks null value, even if it has to according to an attribute rule. This is why we select Saint-Petersburg in the city attribute. At the same time, this rule is absent in the BVR case, as was demonstrated in row 31.
Note that we have computed data for the period that had data conflicts, but the considered golden record spans multiple periods. The overall result for both algorithms that includes data for all periods is shown in Table 4.
Despite recent significant technological advances, human approaches to handling information have not really changed. This is also true in case of MDM systems, in which interaction scenarios have stayed the same. Users still have to think about data layouts and procedures, e.g., define registries and referential tables, set up data pipelines and so on. In other words, people solve problems in an imperative way, specifying everything that needs to be done in order to obtain answers. This approach requires a lot of eofrt, which often comes in the form of duplicated work since processes share a large degree of similarity in many organizations.
At the same time, humans think in terms of problems such as: “Why did sales drop in the last quarter?” or “How did the introduction of the new discount system impact profits?”. A declarative approach would suit these problems better, and thus there is a need for it.
There are two possible ways to achieve it — exhaustive standardization and machine learning. The former is very challenging since there are too many details that should be reflected inside the standards. Companies and public institutions exist all over the world and each has to conform to various local and international regulations.
Standardizing them all will either require a coordinated efort of multiple governmental entities or lead to im- to be flexible in terms of the used DBMS, search, BPMN mense labor costs required to pay the workers who will implementation and so on. Not every customer is willperform this standardization. ing to add more dependencies, which may also require
Machine learning, on the other hand, will incur smaller additional expenses. costs and will not depend on any external collaboration. However, the next generation of MDM toolkits such as While a full-fledged AI with a natural language interface Egeria3, Fuyuko4, AtroCore MDM5 and many others ofer is a very distant vision, machine learning has already open-sourced versions and actively attempt to implement been successfully adopted in individual components of a modular architecture.
MDM systems. For example, semantic column type de- The reason for this are changes in the MDM landscape tection [12, 13, 14], database schema matching [15], du- and emerging requirements. The contemporary environplicate detection [16, 17, 18], and various types of table ment favors, if not requires modular and open products. autocompletion [19, 20]. Cloud-ready systems have become mainstream, and these
Another promising direction is digital storytelling [21, properties are a must in ensuring extension-ability. 22, 23] — automatically extracting and presenting facts Finally, aiming for the organization pull strategy, one contained in the data in a human-friendly way. Employ- must prefer the latter approach, since such applications ing such techniques will lower the qualification require- require increased flexibility. The Unidata platform aims ments and open analytics to broader public. for this niche and is therefore modular, open-source and
There is also a novel class of so-called visual analyt- extension-able. ics [24, 25] systems that contain collaborative tools that allow users to employ various visualization primitives, machine learning models, and other objects. These are 8. Conclusion dragged onto a dashboard and connected to each other in order to construct pipelines. Thus, machine learning will be present not only inside such systems, but outside as well, i.e. allow users to build and use custom machine learning models inside their decision-making pipelines.
a software product line intended for the creation of various MDM solutions. We have described its architecture, use-cases, data storage and query processing algorithms. Finally, we have shared our vision regarding the future of MDM systems.
SAP, Informatica and others ofer a wide range of products for the creation of all types of MDM systems.
However, their toolkits 1) were largely started as monolithic products, 2) are heavily oriented towards vendors’ infrastructure, 3) are frequently proprietary software which is not open-sourced. While the monolithic approach greatly simplifies architecture, it has a number of drawbacks, such as hindering extension-ability and thus making open-sourcing largely useless. Vendor orientation is not necessarily a bad thing, but the need to cope with a systems zoo requires modern MDM products
We would like to thank Alexander Konstantinov and Roman Strekalovsky for their comments. We would also like to thank Anna Smirnova for her help with the preparation of the paper.
1835–1848. [15] T. Sahay, A. Mehta, S. Jadon, Schema matching us[1] V. Khatri, C. V. Brown, Designing data governance, ing machine learning, CoRR abs/1911.11543 (2019).
Commun. ACM 53 (2010) 148–152. URL: https: arXiv:1911.11543. //doi.org/10.1145/1629175.1629210. doi:10.1145/ [16] N. Barlaug, J. A. Gulla, Neural networks for entity 1629175.1629210. matching: A survey, ACM Trans. Knowl. Discov. [2] M. Jagals, E. Karger, F. Ahlemann, Already grown- Data 15 (2021).
up or still in puberty? a bibliometric review of [17] W.-C. Tan, Deep data integration, in: Proceedings 16 years of data governance research, Corporate of the 2021 International Conference on ManageOwnership & Control 19 (2021) 105–120. ment of Data, SIGMOD/PODS ’21, Association for [3] O. B. Nielsen, et al., Why governing data is difi- Computing Machinery, New York, NY, USA, 2021, cult: Findings from danish local government, in: p. 2.
Smart Working, Living and Organising, Springer [18] Y. Li, et al., Deep entity matching: Challenges and International Publishing, Cham, 2019, pp. 15–29. opportunities, J. Data and Information Quality 13 [4] D. International, DAMA-DMBOK: Data Manage- (2021).
ment Body of Knowledge (2nd Edition), Technics [19] S. Zhang, K. Balog, Web table extraction, retrieval Publications, LLC, Denville, NJ, USA, 2017. and augmentation, in: B. Piwowarski, M. Cheva[5] R. W. Zmud, An Examination of ’Push-Pull’ The- lier, É. Gaussier, Y. Maarek, J. Nie, F. Scholer (Eds.), ory Applied to Process Innovation in Knowledge Proceedings of the 42nd International ACM SIGIR Work, Management Science 30 (1984) 727–738. Conference on Research and Development in InURL: https://www.jstor.org/stable/2631752, pub- formation Retrieval, SIGIR 2019, Paris, France, July lisher: INFORMS. 21-25, 2019, ACM, 2019, pp. 1409–1410. [6] M. Allen, D. Cervo, Multi-Domain Master Data [20] S. Zhang, K. Balog, Web table extraction, retrieval, Management: Advanced MDM and Data Gover- and augmentation: A survey, ACM Trans. Intell. nance in Practice, 1st ed., Morgan Kaufmann Pub- Syst. Technol. 11 (2020).
lishers Inc., San Francisco, CA, USA, 2015. [21] F. El Outa, et al., Towards a conceptual model for [7] Software Product Lines. Carnegie Mellon data narratives, in: Conceptual Modeling, Springer Software Engineering Institute Web Site., International Publishing, Cham, 2020, pp. 261–270. https://resources.sei.cmu.edu/library/asset- [22] P. Vassiliadis, P. Marcel, S. Rizzi, Beyond rollview.cfm?assetid=513819, 2022. up’s and drill-down’s: An intentional analytics [8] Walker S., Parker S, Hawker M., Radhakrish- model to reinvent OLAP (long-version), CoRR nan D., Dayley A., Magic Quadrant for Master abs/1812.07854 (2018). arXiv:1812.07854. Data Management. Gartner. ID G00466922., [23] P. Vassiliadis, P. Marcel, S. Rizzi, Beyond rollhttps://www.gartner.com/en/documents/ up’s and drill-down’s: An intentional analytics 3995999/magic-quadrant-for-master-data- model to reinvent OLAP, Inf. Syst. 85 (2019) 68– management-solutions, 27 January 2021. 91. URL: https://doi.org/10.1016/j.is.2019.03.011. [9] D. Loshin, Master Data Management, Morgan Kauf- doi:10.1016/j.is.2019.03.011.
mann Publishers Inc., San Francisco, CA, USA, 2009. [24] E. Wu, Systems for human data interaction [10] Andrew White. The Five Vectors of Complexity (keynote), in: D. Mottin, et al. (Eds.), Proc of That Define Your MDM Strategy. ID: G00276267., the 2nd Workshop on Search, Exploration, and https://www.gartner.com/en/documents/ Analysis in Heterogeneous Datastores (SEA-Data 3038017/the-five-vectors-of-complexity-that- 2021@VLDB’21), 2021.
define-your-mdm-stra, 27 April 2015. [25] Z. Shang, et al., Davos: A system for interactive [11] Global Product Classification (GPC). GS1 Web Site., data-driven decision making, Proc. VLDB Endow.
https://www.gs1.org/standards/gpc, 2022. 14 (2021) 2893–2905. [12] M. Hulsebos, et al., Sherlock: A deep learning approach to semantic data type detection, in: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, 2019, p. 1500–1508. [13] X. Deng, et al., Turl: Table understanding through representation learning, Proc. VLDB Endow. 14 (2020) 307–319. [14] D. Zhang, et al., Sato: Contextual semantic type detection in tables, Proc. VLDB Endow. 13 (2020)