This paper presents a demonstrator of a knowledge graph based approach for integrating and reconciling cross-border and crosslanguage procurement and company data from distributed data sources. The demonstrator also includes analysis of the resulting knowledge graph, exempli ed in anomaly detection and cross-lingual search.
The availability of high quality, open, and linked procurement data presents an opportunity to enhance the public procurement processes. In this respect, several directives were put forward by the European Commission (e.g., Directive 2003/98/EC and Directive 2014/24/EU8), which led to the emergence of national and international public procurement portals. However, there is a lack of common agreement across the European Union (EU) on the data formats for exposing such data sources and on the data models for representing such data, leading to a highly heterogeneous technical landscape.
To this end, in order to deal with the technical heterogeneity and to connect
disparate data sources currently created and maintained in silos, we developed a
platform consisting of a set of modular REST APIs and ontologies, to publish,
curate, integrate, analyse, and visualise an EU-wide, cross-border, and cross-lingual
procurement knowledge graph (KG). This paper presents a demonstrator for the
knowledge graph based platform and end-user tools for integrating and
reconciling procurement and company data from distributed data sources, including
analytics tools exempli ed in anomaly detection and cross-lingual search [
Reconciliation
API
OC API
OO API owl:sameAs Distributed data sets
JSON KG ingestion
ETL
RDF
Triple store 1.
Download procurement data
2.
Reconcile supplier data
OpenOpps (OO) procurement database OpenCorporates (OC) company database API Gateway
JSON SPARQL API
Core API rdf:seeAlso
Cross-lingual
Search API
Document
store
Procurement and company data underlying the KG is provided by two main data
providers: OpenOpps1 for procurement data (e.g., tenders and contracts) and
OpenCorporates2 for supplier data (i.e., companies). OpenOpps has gathered
over three million tender documents from more than 685 publishers through
Web scraping and by using open APIs, while OpenCorporates currently has
140 million entities collected from national registers. We integrated the two
high-quality data sets according to an ontology network to form a knowledge
graph. The ontology network includes an ontology for representing procurement
data based on Open Contracting Data Standard (OCDS), namely the OCDS
ontology3 [
The data ingestion process (see Fig. 1) comprises several steps using data APIs of both providers, including data curation, matching suppliers appearing in tender data against company data (i.e., reconciliation), and translating data sets into the underlying graph data representation (i.e., RDF) with respect to the ontology network and linked data principles. The platform (see Fig. 1) employs a triple store for the generated RDF-based data, linked to original data sources (using owl:sameAs), and a document store for the documents associated with
the public procurement data (using rdfs:seeAlso). The data is made available through a SPARQL API, a core REST-based API (i.e., KG API), a cross-lingual search API, and an API gateway providing a single entry point to the previously mentioned APIs. The current release of the KG covers data from January 2019 onwards. New data is onboarded on a daily basis. As of August 2020, the KG consists of more than 126 million triples and contains information about 1,31 million tenders, 1,54 million awards, and more than 99 thousand companies5. The source data collected from the data providers in JSON and the KG data in RDF are made openly available under the Open Database License (ODbl)6 on Zenodo7. An online catalogue is available8 providing access to data, schemas, core APIs, tools, and added value services (see Fig. 2). 3
We implemented a number of analysis techniques on the KG: anomaly detection by using ML techniques for identifying patterns and anomalies, such as fraudulent behaviour or monopolies in procurement processes and networks across data sets produced independently; and, cross-lingual document search for nding documents that are similar to a given one independently of its language.
Public procurement is particularly susceptible to corruption, which can impede economic development, create ine ciencies, and reduce competitiveness. At the same time, manually analysing a large volume of procurement cases is not feasible. Therefore, rstly, we applied several ML techniques, i.e., supervised, unsupervised, and statistical, on top of the Slovenian public procurement data in the KG to
identify patterns and anomalies. We implemented a system and made it available online9. The system developed is capable of processing tens of millions of records and allows detecting a large class of anomalies. Fig. 3 depicts the supervised analysis approach implemented in our platform based on a decision tree. Users select parameters by their own choice (e.g., buyer size and bidder municipality), and explore various parameters contributing to the success of public tenders.
Procurement processes are not only creating structured data, but also
constantly creating additional documents. These are commonly published in the
o cial language of the corresponding public administrations. Only some of these
are multilingual, but the documents in the local language are typically longer.
Therefore, secondly, we worked on an added-value service10 with the possibility of
nding documents that are similar to a given one independently of the language
in which it is made available [
In this paper, we demonstrated the use of Semantic Web and Linked Data technologies and principles to integrate open procurement and company data sets, and advanced analytics and to unlock their value. The KG enabled easier
and advanced analytics, which was otherwise not possible. However, we also faced a high number of data quality issues, such as missing, duplicate, and erroneous data, even though there are mandates in place for buyers to provide correct data. Acknowledgements. The work reported in this paper is partly funded by EC H2020 TheyBuyForYou (780247) and euBusinessGraph (grant 732003) projects.