The proliferation of e-procurement systems in the public sector allows for joint access to useful and open information sources. Our research explores ways to improve the quality and correctness of the public procurement process and the eficiency of administrations, the reduction of the time spent by economic operators, and the costs of public administrations. In particular, we explored the dataset of the National Anti-Corruption Authority in Italy on public procurement and the judges' sentences related to public procurement. Our first goal was to identify which procurement led to disputes and recourse to Administrative Justice by identifying relevant procurement features. Our second goal was to develop a recommender system on procurement by applying machine learning algorithms and deep neural models to return similar procurement to a given one and find companies as potential bidders, depending on the procurement requirements. Our third goal is to automate the analysis of a dataset of public procurement, contract awards, and appeal procedures. Process discovery techniques were applied to the dataset, considering control-flow, organizational (resource), and time perspectives. The results demonstrate the importance of applying these techniques in the legal field.
of law, allowing for the exploitation of computational technologies and algorithms. pliance analysis and anomaly detection with Artificial niques successfully applied in the legal fields include chine Learning (ML) and specifically Deep Learning (DL) models that can automatically extract some knowledge about the semantics in texts. Before the advent of DL models, the application of Natural Language Processing (NLP) already achieved good results in many tasks involving natural languages, such as text modeling, parsing, machine translation, and automatic query answering.
providing knowledge about data registered in information systems is Process Mining (PM), aimed at discovering, and the case study in Section 2.2. The first project conformation Retrieval and ML techniques, described in Section 3. A second contribution addresses an organizational perspective with the automatic discovery of activity sequences using process mining algorithms in Section 4. Finally, Section 5 presents conclusions and future work.
public procurement process in Italy.
The first dataset was obtained from the National
AntiCorruption Authority (ANAC), which collects data on
2. Background calls for procurement from the public contract authority
and provides a catalog of Open Data describing public
2.1. Related work procurement, Public Administrations (PAs) which are
responsible for the procurement, and economic operators
We conducted a Systematic Literature Review (SLR) ac- (EOs) participating in the tender awarding process or
cording to the approach described in [
In [
Alternative predictive models have been estimated
in [
As regards recommender systems, in [
Our goal is to propose a smart engine to identify cases of the procurement, from 40k euro upwards; the number of similar procurement. If the smart engine recognizes of lots in the procurement, the CPV code5 describing that a public administration received a recourse because the main object of the contract obtained from a public of a tender, the following stipulated contracts could be ontology aligned in multiple languages. at risk of being stopped by the Administrative Justice The IAJ is a textual dataset containing the adminisaction. trative judges’ sentences saved in HTML format (91.5%),
To the best of our knowledge, very few works investi- DOC/DOCX (8.4%), and PDF files (0.1%). In addition to gated a process-oriented approach to legal cases. the texts, the sentence files contain some useful meta
At the intersection between PM and law, some works data: the ECLI code6 of the sentence, the court region
explore real-world cases of process discovery involving (that corresponds to the region of the PA that created
public procurement: a case study focuses on a heuristic the tender), the year and the progressive number of the
algorithm revealing a concept drift in the publication of judge’s sentence.
contracts in the Philippines [
An application of process discovery in the legal field metadata of appeals related to the sentences: the recourse
is discussed in [
We build on these works and, as proof of concept for
PM, we analyzed the results considering each of the
following PM perspectives: control-flow, organizational
(resource), and time [
the Contracts of the Italian
Public Administrations 3.1. Methodology 3.1.1. Merge of data sets representation of the semantic content of the contract description by training the numerical vectors called sentence embeddings using BERT [15]. We used as input sentences the brief descriptions in natural texts of procurement in the ANAC database. We obtained vectors with 768 dimensions. Successively, given a case of an individual procurement, we searched for the most similar and relevant ones in the rest of the database using SBERT [16] and LaBSE: they are a multilingual version of BERT and uses siamese networks to work on multilingual and Italian corpora. They are often used as tools to rank a set of sentences for their similarity to a given sentence, denoted as a query.
First, the extracted texts from the sentences files were indexed with specialized IR tools, with Elasticsearch [12] being the most popular7. The texts and metadata of ap- 3.1.3. ML prediction models training peals and sentences were serialized into Newline Delimited JSON (NDJSON8) and indexed by the internal engine Following RQ3, a binary classification model will be of the tool. We also employed Named Entity Recogni- trained to predict whether a procurement will have a tion (NER) methods in Elastic Search to recognize the recourse. Identified the solution as a supervised learninvolvement of economic operators and PAs in recourses. ing classification task [ 17], the following classifiers [ 18]
NLP techniques were then used to create sentence em- were explored: K-Nearest Neighbours (KNN), Logistic beddings of procurement objects from ANAC and recourse Regression (LR), Naive Bayes (NB), Support Vector Maobjects from IAJ, to improve the connection between the chines (SVM), Decision Tree (DT), Random Forest (RF), two datasets; for this purpose, LaBSE BERT model [13] and eXtreme Gradient Boosting (XGB). has been used. Cosine similarity [14] was then applied on sentence embeddings to collect the corresponding simi- 3.2. Results lar subjects of a procurement object and a recourse object.
When the match between the entries of the two data sets 3.2.1. Merge of data sets was successful (via IR or NLP), we used the presence of an appeal on procurement as an indication of a positive case on that procurement entry; otherwise, it was treated as a negative case.
Figure 1 summarises the workflow described above.
ment (Section 3.1), we conducted three diferent types of searches: 1) by {CIG} (the procurement identifier); 2) by {EO participant, EO winner, PA, Region/Court, Year}; 3) by the similarity between {procurement object, recourse object}.
The results in Table 1 show how the methods, used incrementally, improve the ability to recognize a reference between the ANAC and the IAJ data sets based on the available sentences (67, 850).
Reference found by {feature} Procurement identifier: {CIG} Denominations: {EO participant, EO winner, PA, Region/Court, Year} Similarity: {procurement object, recourse object} Total
(Section 3.1.2), we decided to evaluate its Precision at 10. Precision at 10 was calculated by a panel of three individuals working separately on a test set of recommendations for 100 random procurement instances for Public Works, Services, and Goods/Supplies. Since each example refers to multiple elements (e.g., the awarding procedure, the location, the subject), the panel agreed in advance, case by case, on the elements of judgment (span from 2 to 5). Each panel member gave a relevance score of similarity between the query tender and its recommendations on each key element. The final relevance score is the mean of the scores given by the panel.
The results of Precision at 10 depend on the threshold for the relevance score; the lower the threshold, the higher the precision. We can think of this threshold as a measure of how strictly similar we want the recommended procurement and the query. A summary of the precision values at 10 is in Figure 2. We observe how the recommendation system works better for tenders of Goods/Supplies (orange bars). This makes sense because their descriptions are shorter than Public Works (green bar) or Services (yellow bar).
The typical main basic step in a PM search is the construction of the log file that includes the time sequence of events. Each event in an event log includes at least three basic features: the identifier of the process it belongs to, the name of the activity which generated the event, and the corresponding execution timestamp [20]. 4.1. Methodology
To answer RQ4, discovery algorithms can be applied to
automatically derive process models. In the wide range
of discovery methods proposed in the literature, we
focus on the Fuzzy Miner implementation [21]. As a proof
of concept for PM, the results were analyzed
considering each of the following PM perspectives: control-flow,
organizational (resource), and time [
from the event log provides a complete overview of the actual legal process flow, as shown in Figure 4; in the process map, the activity with the highest frequency, PUBLICATION, is indicated as the starting point. A group of events has a higher frequency, where darker rectangles in the diagrams correspond to “standard” events existing in all procurement. A second group of particular events occurs with a lower frequency (lighter color in the map), i.e. subcontracting or suspensions. sideration is the region that issued the call. The results indicate the importance first of all of the “Central” region, which includes the administrative and governmental bodies of the Italian state (27,503 cases, i.e. 17%), as well as Lombardy (19,650 cases, i.e. 14.7%) and Emilia-Romagna (82,540 cases, i.e. 8.39%); the mean case duration for this three regions is between 12 and 14 months.
Time perspective. The diagram showing the average duration of transactions between activities makes it possible to identify bottlenecks. As highlighted by thicker arcs in Figure 5, the main critical transitions are represented by: procurement CREATION to WIN (69.9 days on average); procurement CREATION to SENTENCE (35.8 weeks on average); procurement CONTRACT-START to VARIANT (30.7 weeks on average).
In future work, we plan to investigate furthermore the Proceedings of the Eighteenth International Conexplainable AI techniques. ML systems are becoming ference on Artificial Intelligence and Law, 2021, pp. increasingly ubiquitous [22], increasing the demand to 240–244. question, understand, and trust ML systems [23]. [10] N. J. Belkin, W. B. Croft, Retrieval techniques.
From the PM perspective, future work concerns the (1987). prediction of features of interest from an organizational [11] P. M. Nadkarni, L. Ohno-Machado, W. W. Chapperspective. First, we consider investigating the remain- man, Natural language processing: an introducing time after the activity of interest (i.e., the awarding), tion, Journal of the American Medical Informatics as well as the successful or unsuccessful outcome of a Association 18 (2011) 544–551. tender. [12] C. Gormley, Z. Tong, Elasticsearch: the definitive guide: a distributed real-time search and analytics engine, OŔeilly Media, Inc., 2015.
References [13] F. Feng, Y. Yang, D. Cer, N. Arivazhagan, W. Wang, Language-agnostic bert sentence embedding, arXiv preprint arXiv:2007.01852 (2020). [14] G. Salton, C. Buckley, Term-weighting approaches in automatic text retrieval, Information processing & management 24 (1988) 513–523. [15] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,
BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [16] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, arXiv preprint arXiv:1908.10084 (2019). [17] P. Cunningham, M. Cord, S. J. Delany, Supervised learning, in: Machine learning techniques for multimedia, Springer, 2008, pp. 21–49. [18] R. Saravanan, P. Sujatha, A state of art techniques on machine learning algorithms: a perspective of supervised learning approaches in data classification, in: 2018 Second international conference on intelligent computing and control systems (ICICCS),
IEEE, 2018, pp. 945–949. [19] A. Géron, Hands-on machine learning with Scikit
Learn, Keras, and TensorFlow, ” O’Reilly Media,
Inc.”, 2022. [20] W. M. van der Aalst, J. Carmona, Process mining
handbook, Springer Nature, 2022. [21] C. W. Günther, W. M. Van Der Aalst, Fuzzy mining– adaptive process simplification based on multiperspective metrics, in: Business Process Management: 5th International Conference, BPM 2007, Brisbane, Australia, September 24-28, 2007. Proceedings 5, Springer, 2007, pp. 328–343. [22] D. V. Carvalho, E. M. Pereira, J. S. Cardoso, Machine learning interpretability: A survey on methods and metrics, Electronics 8 (2019) 832. [23] R. Meo, R. Nai, E. Sulis, Explainable, interpretable, trustworthy, responsible, ethical, fair, verifiable ai... what’s next?, in: Advances in Databases and Information Systems: 26th European Conference, ADBIS 2022, Turin, Italy, September 5–8, 2022, Proceedings, Springer, 2022, pp. 25–34.