The role of tenders as means of investment of public funds and as vehicles of strategic development is nowadays crucial. For this reason, developing and enabling new solutions for e-procurement procedures can help to manage and invest funds. In e-procurement, the Common Procurement Vocabulary (CPV) allows assigning a code that classifies its subject to each tender. This study addresses the challenge of automatically assigning a CPV code to a tender. We tackle this problem in two diferent ways: as a classification problem and as a generative task. To develop and test our models, we build a dataset of 5M Italian tenders extracting them from the National Anti-Corruption Authority (Autorità nazionale anticorruzione - ANAC) website. Results show that text classifier approaches exhibit superior performance in this regard. However, they also reveal the potential of generative models in overcoming the limitations of existing classification methods for CPV code assignment in tender classification, providing valuable insights for improving procurement processes and enhancing eficiency in public sector operations.
1. Introduction Vocabulary (CPV)1 that identifies the subject of a tender. The adoption of the CPV also allows companies to find Knowledge organization systems (KOS), such as thesauri, new public contracts easily, thus fostering competitivegazetteers, lexical databases, ontologies, and classifica- ness. tion systems, are used by institutions to organize large The CPV is structured as a tree of codes comprising data collections, e.g. documents, web pages, and texts. Us- 9 digits, eight plus a check digit, and specifies whether ing a standard format guarantees semantic interoperabil- the tender in question refers to supplies, works or serity and allows for a faster exchange of information. Public vices covered by the contract. Each digit indicates proprocurement represents a field where adopting such sys- gressively finer-grained classifications. More specifically, tems can bring many advantages. On one hand, citizens each CPV is composed as follows: can access data more easily, enabling more straightforward communication with institutions, which can help streamline many bureaucratic processes. Concurrently, adopting KOS systems in procurement has profound implications for professionals working within public administrations. In fact, these systems can serve as invaluable tools, ofering support in the day-to-day activities of public sector employees. Integrating advanced technologies will facilitate a paradigm shift towards higher productivity and eficiency. Tasks that were once labor-intensive and time-consuming can now be executed with greater precision and speed, allowing public administrators to focus on more strategic and value-driven aspects of their roles. For this reason, in the field of public procurement, the European Union developed a Common Procurement • the first two digits identify the divisions (e.g.
71000000-8 Servizi architettonici, di costruzione, ingegneria e ispezione (Architectural, construction, engineering and inspection services)); • the first three digits identify the groups (e.g.
71300000-1 Servizi di ingegneria (Engineering services)); • the first four digits identify the classes (e.g.
71310000-4 Servizi di consulenza ingegneristica e di costruzione (Consultative engineering and construction services)); • the first five digits identify the categories (e.g.
71311000-1 Servizi di consulenza in ingegneria civile (Civil engineering consultancy services)); • each of the last three digits provides an additional degree of precision within each category (e.g. 71311210-6 Servizi di consulenza stradale (Highways consultancy services)); • a ninth digit serves to verify the previous digits. Examples of CPV codes are: 30200000-1 (Computer equipment and supplies), 30230000-0 (Computer hardware), and
S30231000-7 (Computers and printers). demanding a level of expertise that even seasoned pro
The supplementary vocabulary can be used to com- fessionals in the field find daunting. Already existing
plete the description of the subject of a contract. The approaches for CPV classification have been condensed
items consist of an alphanumeric code corresponding to within the last few years. In [
The main vocabulary comprises 9,454 terms and, more to notice is that this work focuses on both English and
specifically, 45 divisions, 272 groups, 1,002 classes, 2,379 French. Kaan Görgün (Mkaan)3 proposed a multilingual
categories, and 5,756 sub-categories. Assigning a CPV to approach which is a fine-tuned version of mBERT [
The paper is organized as follows: Section 2 provides port system that helps users during the entire course of
an overview of the approaches available at the state of investments and contracts in e-procurement is described
the art, Section 3 contains the details of the proposed in [
(CPV) code to a tender is crucial for the accurate identification and precise retrieval of analogous documents.
This meticulous categorization process is indispensable for ensuring the streamlined organization and efective utilization of information. Given the nature of the CPV 2https://info.e-avrop.com vocabulary, which encompasses a set of over nine thou- 3https://huggingface.co/MKaan/ sand terms, this task assumes a challenging dimension, multilingual-cpv-sector-classifier methodologies: 1) a text classification approach based on diferent classifiers; 2) a generative approach based on Transformers with an encoder-decoder architecture.
Both approaches work on the same data. In particular, given a list of tuples (CPV, CPV description, tender object), we split it into three sets: training, validation and testing. More details on the dataset are reported in Section 4. Then each approach is implemented, trained and validated separately on the same data. use only the divisions as labels to obtain reasonable results. This strategic decision is driven by recognising that achieving reasonable results across the entire spectrum of CPV codes poses significant challenges. By concentrating on this subset, we optimize the model’s capacity to provide meaningful and accurate predictions within a more manageable scope.
classical pipeline: Since our approach aims to suggest a CPV given a tender’s description, we decided to investigate the ability of AI generative methods to automatically produce a text given a textual input. Moreover, we want to test if a generative approach can provide better results when a • Preprocessing: the initial step involves the pre- large number of classes is involved, as in our domain. We processing of the tender object, wherein trans- adopt a classical encoder-decoder architecture that has formations such as lowercase conversion and to- proven to provide promising results in several NLP tasks. kenization are applied. These essential prepro- Encoder-decoder architectures are well-suited for solving cessing techniques lay the groundwork for sub- sequence-to-sequence problems like machine translation, sequent stages by standardizing the textual data; as they can efectively process variable-length input and • Feature Vector Generation: following the pre- output sequences. In this architecture, the encoder takes processing step, we construct feature vectors for in a sequence of any length and converts it into a fixedeach tender object. This involves the utiliza- shaped state. On the other hand, the decoder maps the tion of both Bag-of-Words (BoW) and TF-IDF encoded state, which has a fixed shape, back to a sequence (Term Frequency-Inverse Document Frequency) of variable length. approaches. By encoding the textual information In our case, the encoder’s input is the tender’s obinto numerical representations, we aim to cap- ject description and the decoder output is the CPV code ture the salient features that contribute to the and its description. It is important to underline that this classification task; method can produce a CPV code or a description that • Classifier Training and Tuning : subsequently, is not present in the original list of CPVs, while a text a classifier is trained using the feature vectors gen- classifier produces as output a CPV from the set of predeerated in the previous step. The training process ifned CPVs. This poses problems in the evaluation phase is complemented by the optimization of hyper- as comparing the output produced with the gold stanparameters. This optimization is achieved with dard present in the test set becomes more complex. More the use of a validation set, ensuring that the clas- details about the evaluation are reported in Section 4. sifier retains generalization capabilities; • Evaluation: the final stage of our classification pipeline involves the evaluation of the trained 4. Evaluation classifier on an independent test set. This allows an assessment of the model’s ability to generalize and classify unseen tender objects. It serves as a critical benchmark to validate the efectiveness and robustness of the entire text classification system.
processing and scikit-learn for classification. The hyperparameters are found through the grid search. After a ifrst evaluation, we select the following classifiers: Linear SVC and Multinomial Naive Bayes. Moreover, we cast the problem only to classify the divisions that are composed of 45 classes since the model cannot provide reasonable results when the whole set of CPVs is involved.
Moreover, we choose to investigate a classifier based on BERT using its tokenizer. Again, in this setting, we
ANAC anticorruzione (anticorruption) website4. ANAC -Autorità Nazionale AntiCorruzione National AntiCorruption Authority is an Italian independent administrative authority with the aim of combating corruption in the country.
In particular, we retrieve for each tender the CIG (the tender identifier), the tender type, the description of the tender object, the CPV assigned by the RUP and the CPV description. The tender type identifies three kinds of tender: 1) supplying, 2) service and 3) work. From the original dataset, we remove CPV codes that occur less than 20 times and store data in a CSV file for a total of 5 million tenders. We split the dataset in training, test
and validation according to the percentages reported in Table 1.
training test validation
For training the encoder-decoder architectures, we build a diferent version of the dataset in the JSONL format. Each row in the dataset is a JSON object with two elements: source and target. The source is the input text of the encoder and the target is the output text of the decoder. In our case, the source is the concatenation of the type of the tender and the description of the tender’s object, while the target is the concatenation of both the code and the description of the CPV. The tender’s type defines the nature of the object from a list of predefined types: service, supply and work. Listings 1 shows an example of a JSON object related to a tender. 1 {"source":"lavori lavori di pavimentazione delle vie san martino e santa Maddalena", 2 "target":"45262321-7 - lavori di pavimentazione" }
The dataset is stored on Zenodo5, while the code is available on GitHub6. The code for fine-tuning the IT5 model is available here7. The IT5 models fine-tuned on the CPV generation task are on HuggingFace: the large model8.
This sub-section reports information and results about text categorization approaches. Regarding the parameters’ optimization, we adopt a grid search for Linear SVC and Multi-NB. For Linear SVC, we optimize the parameter in the set {0.5, 1, 2, 4, 8}, while for Multi-NB we consider ℎ in {0.1, 0.3, 0.5, 0.7, 0.9} and _ in {True, False}. The best values selected after the grid search are = 0.5, ℎ = 0.3 and _ = .
For BERT, we did not perform parameters optimization since the required computational time is very high. We fine-tuned a specific language model for Italian called dbmdz/bert-base-italian-uncased9 using 5https://zenodo.org/records/10007545 6https://github.com/ematanzi/Valutazione-CPV/ 7https://github.com/gsarti/it5 8https://huggingface.co/basilepp19/cpv-it5 and the base model https://huggingface.co/basilepp19/cpv-it5-base 9https://huggingface.co/dbmdz/bert-base-italian-uncased the Adam optimizer with a learning rate of 5e-05 and a batch size of 16 trained for 5 epochs.
Results of text classification approaches are reported in Table 2. Generally, the results are very low due to the large number of classes and BERT reports the worst performance since, for some classes, there are very few examples in training data. We observe a large accuracy with respect to the F1 measure. This is due to the presence of few classes with many examples. For these classes, classifiers can achieve good performance. For example, BERT achieves the 90% of F1 for the most frequent class10.
indexing, with the aim of solving the task as a retrieval task.
In detail, we indexed CPV code description pairs corresponding to target in the two fields code and description. Afterwards, we ran the search for each source element in JSON file, obtaining for each search the element belonging to target with the most similar description to the source string, and saved results in a file containing the triple source, target and generated. Where target is the expected description and generated is the retrieved one. We adopt the same output format for the generative approaches. The idea is to exploit the source as the query for the search engine and retrieve the most similar code descriptions using the search engine.
We executed this experiment four times, implementing variations in both the configuration of the text analyzer and the choice of two distinct similarity measures. The adopted configurations are the following: • StandardAnalyzer + default similarity; • StandardAnalyzer + LMDirichletSimilarity; • ItalianAnalyzer + default similarity; • ItalianAnalyzer + LMDirichletSimilarity.
gorithm for Italian, while the StandardAnalyzer implements a grammar-based tokenizer for several languages.
The default similarity provided by Lucene is the BM25 model [12], while the LMDirichletSimilarity uses a language model for information retrieval with the Bayesian smoothing based on Dirichlet priors [13]. The evaluation has been carried out on the test set. We decided to use BLEU metric to measure matching between generated and target text since this metric is based on the idea that the nearer the predicted text is to the target one, the more correct it is. Considering the small size of the compared strings, we decided only to use 1-gram and 2-gram of
1033000000-0 Apparecchiature mediche, prodotti farmaceutici e per la cura personale (Medical equipment, pharmaceuticals and personal care products) consecutive words, attributing them to the same weight We evaluate the generative approach based on the (0.5). In calculating the metric, a smoothing function has encoder-decoder transformer by fine-tuning it on trainbeen used, increasing the score when there are partial ing data. We start from a pre-trained Italian model called matches between the generated text and the target text. IT5 [14]. The IT5 model family is the initial endeavour
In the evaluation, since the resolution of this classi- to pre-train extensive sequence-to-sequence transformer ifcation task with utmost precision is arduous even for models specifically designed for the Italian language, inhuman experts, we decided to consider every possible spired by the methodology employed in the original T5 correspondence between the code-description generated model [15]. We fine-tuned two diferent models with couple and the target one, which are the following: diferent sizes: IT5-lager and IT-base.
Results are reported in Table 4 for the large model and in Table 5 for the base one. • the full correspondence; • the correspondence between codes, but not the
description (the opposite case can never occur); • the correspondence between categories; • the correspondence between classes; • the correspondence between groups; • the correspondence between divisions; • the case of no match.
solution has been approached. For all of these cases, we calculated the number of times they occurred along with the relative average of the BLEU score metric. Eventually, we also calculated the average BLEU score related to all the tests performed.
The best results obtained by the baseline are reported in Table 3. The results are obtained using the ItalianAnalyzer and the default similarity. The baseline based on the search engine is able to correctly retrieve the correct CPV with the correct description for only 11.34% of testing data. In the 61.77% of cases is not able to retrieve the correct CPV with very low BLEU (0.0515), this means that the description of the first retrieved CPV is very diferent from the correct one.
Perfect match Only CPV code Only category Only class Only group Only division No match All # matchs 113,440 872 30,502 53,232 90,673 93,574 617,707
rization ones, we consider from the generated output only the first two digits of the CPV, i.e. the CPV divisions. This choice allows us to compare the generative approach with the ones based on text categorization since the latter are trained to predict only the division of each tender. Results of this analysis are reported in Table 6 and show that if we consider generative approaches as a classifier, they are below the simple Linear SVC. Performing a dual evaluation in which a classifier is evaluated as a generative approach is not possible since we train classifiers only for predicting divisions. Considering only the code of the division is not possible to generate a description comparable to the text generated by an IT5 model. model Linear SVC IT5-large IT5-base
IT5 model is trained on all the possible descriptions of a CPV, while the text classifier approaches handle only the code of the division and cannot provide usable results when they are trained on the whole set of possible classes. Moreover, generative approaches provide a significant improvement with respect to the baselines obtained by a search engine.
tender by aligning it with the comprehensive Common Procurement Vocabulary (CPV), i.e. a meticulously curated European lexicon of codes designed to precisely identify the subject matter of a tender. The complexity of this task lies in the diverse nature of procurement scenarios, where each tender has its own description and requirements. The CPV emerges as a fundamental tool in deciphering the procurement language, trying to define a European dictionary allowing interoperability among diferent countries. Our proposed methodologies encompass two distinctive approaches: the former relies on a conventional text classification paradigm, whereas the latter leverages a generative strategy hinging on the encoder-decoder architecture as conceptualized by the T5 model.
In our systematic exploration of the system’s proficiency in discerning the accurate division of a tender, specifically on the initial two digits of the CPV, it becomes evident that text classifier approaches provide the best results. Nevertheless, a noteworthy result surfaces when we focus on the holistic identification of the entire CPV through a descriptive context. In this context, the generative approaches exhibit commendable eficacy, demonstrating promising outcomes. Notably, these generative techniques surpass established baselines constructed through conventional keyword-centric search engines, attesting to their heightened capabilities in nuanced comprehension and contextual inference.
We acknowledge the support of the PNRR project FAIR Future AI Research (PE00000013), Spoke 6 - Symbiotic AI (CUP H97G22000210007) under the NRRP MUR program funded by the NextGenerationEU.
We thank Salvatore Tucci for his helpful support in evaluating text classification approaches. P. Lops, Ai-based decision support system for public procurement, Information Systems 119 (2023) 102284. URL: https://www.sciencedirect.com/ science/article/pii/S0306437923001205. doi:https: //doi.org/10.1016/j.is.2023.102284. [12] S. Robertson, H. Zaragoza, The probabilistic relevance model: Bm25 and beyond, in: the 30th Annual International ACM SIGIR Conference, 2007, pp. 23–27. [13] C. Zhai, J. Laferty, A study of smoothing methods for language models applied to ad hoc information retrieval, in: ACM SIGIR Forum, volume 51, ACM New York, NY, USA, 2017, pp. 268–276. [14] G. Sarti, M. Nissim, It5: Large-scale text-to-text pretraining for italian language understanding and generation, arXiv preprint arXiv:2203.03759 (2022). [15] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-totext transformer, The Journal of Machine Learning Research 21 (2020) 5485–5551.