It is widely accepted in literature that data curation is the first step for a successful pretraining of Large, and Small, Language Models (LLMs). Datasets generally fall into two categories: open datasets are publicly available, fostering transparency, reproducibility, and community-driven improvement, but they often face limitations in scale, diversity, and quality. Closed datasets, typically curated by private entities, can oefr greater scale, higher quality, and proprietary data sources, yet they raise concerns around transparency, bias auditing, and public accountability. This paper presents an experiment aimed at quantitatively measuring the improvements provided by representative datasets for LLM pretraining. We pretrained two small LLMs under the same experimental conditions as the corresponding Italian reference models from the Minerva family, evaluated their performance on standard benchmarks, and used LLM-as-aJudge to assess the Fluency, Coherence, and Relevance of generated texts on specific tasks. The results support the idea that, while open science and open datasets are important goals, representative corpora, even if closed, are more suitable for LLM pretraining, as they enable better performance under identical experimental conditions.
and societal dimensions [8].
Large language models (LLMs) have emerged as foun- filtered internet scrapes (e.g., Common Crawl), more
redational tools in Natural Language Processing (NLP), cent approaches have shifted toward structured,
transparpowering a wide array of applications from question ent, and task-specific datasets, often constructed through
answering and summarisation to code generation and a combination of automated and manual filtering
techscientific discovery. Their performance, generalisation niques [9]. These developments reflect a growing
recogability, and alignment with human values are deeply in- nition that model capabilities and behaviours are closely
lfuenced by the quality, diversity, and scale of the data tied to the provenance and properties of their training
used during pretraining [
Data curation for LLMs involves the collection, filter- 1.1. Open vs. Closed Pretraining Datasets
ing, deduplication, classification, and documentation of
large-scale textual corpora. These processes aim to bal- The growing ecosystem of LLMs has revealed a sharp
ance scale with quality by removing low-signal, harmful, divide between open and closed approaches to data
cuor irrelevant content while preserving linguistic diver- ration. On one hand, open-source initiatives such as
sity and domain coverage [5, 6]. More recent eforts BLOOM [10], OPT [
0000-0001-7950-0347 (F. Tamburini) The open-data approach is grounded in scientific ideals © 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License of transparency and collaborative validation. Models like Attribution 4.0 International (CC BY 4.0).
BLOOM, trained exclusively on open-access sources in- ity. RefinedWeb [ 14] features a deduplicated and qualitycluding multilingual Common Crawl, Project Gutenberg, filtered web dataset used to train models such as Falcon. and academic corpora, exemplify an efort to democratise It emphasises a scalable yet high-signal alternative to LLM research and foster global participation [10]. The raw web scrapes. CulturaX [15] is large-scale multilinopen release of datasets enables systematic study of data gual web dataset covering 167 languages, designed to quality, bias, duplication, and domain representation, and improve the cultural and linguistic diversity of LLMs. it supports downstream development of safer and more CulturaX emphasises inclusion of underrepresented lanequitable AI systems. guages by sourcing and curating high-quality content
cal, or legal reasons for withholding training data details. Books3 (from The Pile) is large collection of digitised OpenAI’s GPT-4 report, for example, states that “given books, providing long-form narrative and expository text. the competitive landscape and the safety implications Despite its utility, its inclusion has sparked debate due of large-scale models,” they have opted not to disclose to copyright concerns, underscoring the need for clearer training data sources. While this protects proprietary data usage norms. advantages and potentially prevents misuse of harmful These datasets are frequently combined or customised content, it also hinders external audits of data quality, depending on the training goals, whether for generalbias, and copyright compliance. Without transparency, purpose models, multilingual capability, or domainit becomes dificult to evaluate how model performance specific LLMs. CulturaX, in particular, represents a growor behaviour may be influenced by specific sources or ing movement toward linguistic equity and cultural inomissions. clusivity in large-scale model pretraining.
This divergence has implications for the broader AI research community. The lack of visibility into proprietary The efort to create open datasets for LLM pretraining datasets exacerbates the reproducibility crisis in machine that cover a wide range of data inevitably encounters a learning and limits eforts to assess environmental and major challenge: whether or not to include text types social impacts of training practices. Conversely, open that are not freely available on the web. In our view, this models, while more transparent, often contend with lim- is a critical issue when comparing LLMs trained on open itations in data scope and quality due to the exclusion of data with their counterparts developed by large tech comcopyrighted or paywalled content, potentially afecting panies using closed datasets, which undoubtedly include their competitiveness in knowledge-rich domains. a richer and more representative variety of document
Ultimately, the tension between open and closed data types for the language or languages being studied. The paradigms reflects competing priorities in the develop- central concept here is representativeness, which Egbert ment of foundation models: openness and accountability et al. [16] define as “the extent to which a corpus permits versus competitive advantage and scalability. accurate generalisations about the target domain, which involves two components: the extent to which the corpus includes the full range of both text types and linguistic 1.2. Key Open Datasets for LLM distributions in a domain”. In essence, a representative
A number of high-quality, publicly available datasets population of texts corresponding to the language variety have become foundational to the training of open-source under investigation. large language models. These datasets vary in terms of Another point regards the quality of texts published on domain coverage, linguistic diversity, and preprocess- the Web when compared with curated and edited texts ising strategies, but collectively represent the backbone of sued by professional publishers. Web texts and published transparent and reproducible LLM development. texts difer significantly in form, purpose, authorship, and
The Pile [5], a curated 825 GB dataset designed for audience engagement. Web texts, such as blog posts, sotraining language models, combines diverse sources such cial media updates, and news articles, tend to be dynamic, as academic articles (arXiv), code (GitHub), books, legal hyperlinked, and frequently updated. They emphasise documents, and forums to maximise domain coverage. immediacy, brevity, and interactivity, often written in C4 (Colossal Clean Crawled Corpus) [4] is a large-scale, an informal tone to encourage user engagement [17]. In ifltered dataset derived from Common Crawl. It removes contrast, published texts like academic articles, books, boilerplate, duplicates, and low-quality text to provide and journals are typically static, peer-reviewed, and fola clean, general-purpose corpus for language modeling. low rigorous editorial standards. These texts prioritise RedPajama [13] presents a reproducible, open alternative depth, permanence, and formal structure. Additionally, to the Llama pretraining dataset. It aggregates content while published texts aim for scholarly credibility and from Common Crawl, Wikipedia, ArXiv, StackExchange, longevity, Web texts often prioritise accessibility, shareand more, with a focus on transparency and reproducibil- ability, and multimedia integration. Understanding these distinctions is critical for analysing digital literacy and CORIS balancing. This corpus was constructed by selectcommunication strategies in the information age and, in ing materials from the previously mentioned CulturaX our opinion, is also critical for pretraining LLM providing project and incorporating large sections of specific pub“good” and “reliable” texts for teaching a language to a lished texts. This extensive, curated, and representative LLM. training corpus was then used to train our new “CORIS
This paper aims at exploring and quantify the difer- llm” language model, following the procedure described ences in training a LLM either with open Web data or in the next section. on a representative corpus examining if the two settings produce some diferences in LLM performance, taking contemporary Italian as the reference language. 3. Experiment Settings 3.1. LLM pretraining
Given the objective of this study, we introduce the refer- scratch on Italian [12] and emerged as a standard
reference corpus for contemporary Italian which we use as a ence for Italian NLP. A prior study pretrained an Italian
template for building the representative corpus employed model based on GPT-2 from scratch [
After a long design process devoted to a careful defini- training a 350M model, it is too small, following the
Chintion of relevant textual macro-varieties and their propor- chilla rule [
CORIS Section Press Fiction Academic Prose Legal & Admin. Prose Miscellanea Ephemera
Proportion 38% 25% 12% 10% 10% 5%
of standardised benchmarks designed to assess a broad range of linguistic, reasoning, and task-specific capabilities. These benchmarks enable systematic comparison across models and facilitate progress tracking in natural language processing.
To address the need for evaluating generation-based
tasks, LAMBADA [
ARC-C ARC-E BoolQ GSM8K
MMLU PIQA SciQ TQA 32.6 32.6 31.9 Minerva-350M-base-v1.0 [12] Minerva-350M-base-v1.0 (our) CORISllm-350M-base Minerva-1B-base-v1.0 [12] Minerva-1B-base-v1.0 (our) CORISllm-1B-base tion, probing a model’s depth of understanding beyond 3.3. Text Generation Quality Evaluation surface-level patterns.
More recently, MMLU (Massive Multitask Language Evaluating LLM-generated texts is inherently
challengUnderstanding) has been introduced as a collaborative ing, and assessing the quality of these textual outputs is
efort to assess a wide range of LLM competencies rang- even more complex [28].
ing from law and medicine to physics and philosophy Our primary objective is to conduct a careful
evaluaofering a broad-spectrum evaluation across 57 subjects tion of the quality of texts generated by LLMs.
Specifito test a model’s ability to generalise across domains [
While existing evaluation benchmarks are highly valu- non-representative datasets, namely the Minerva family,
able, they are primarily designed to assess LLM perfor- with one trained on a representative and balanced dataset,
mance in English and are therefore not suitable for our CORISllm. The comparison focuses on commonly used
purposes. Recently, a group of Italian researchers intro- human evaluation metrics: Fluency, internal Coherence
duced a promising new benchmark, called ITA-Bench, for and text Relevance to the given task.
evaluating LLMs in Italian. This suite combines automat- To ensure a fair evaluation, it is necessary to generate
ically translated versions of popular English benchmarks and assess a substantial number of texts. For this purpose,
with adapted, manually curated datasets for Italian [
Table 2 presents the results of CORISllm-350M and per article, a children’s fairy tale, an abstract of a scientific CORISllm-1B on ITA-Bench, alongside a comparison with paper, a judgment for a crime, a trip description, and a the corresponding Minerva models. Overall, the two brief movie review, and generated 50 outputs each. Table LLMs demonstrate comparable performance: Minerva 3 presents the prompts used to stimulate the LLMs to performs better on certain tasks, while CORISllm slightly generate texts. outperforms it on others. The following sections first describe the human eval
On average, the Minerva models show slightly better uation process, followed by the LaaJ methodology we performance; however, these results must be interpreted employed to achieve our objective. in light of the nature of the benchmark. ITA-Bench focuses primarily on tasks involving commonsense rea- 3.3.1. Human Evaluation of LLM Outputs soning and scientific knowledge retrieval, which are not Human evaluation remains the gold standard for assesswell-suited for assessing diferences in text generation ing the quality of natural language outputs produced by capabilities. Pretraining an LLM on a representative cor- LLMs. Despite the growing sophistication of automated pus does not inherently confer an advantage in reasoning metrics and model-based evaluators, human judgments or STEM-related tasks because the dataset used for pre- are uniquely capable of capturing nuanced dimensions of training does not contain specific materials useful for quality such as contextual appropriateness, subtle coherincreasing performance on STEM-related tasks and no ence errors, pragmatic relevance, and factual accuracy. specific methods were used to promote the development Consequently, human assessments are widely used in of reasoning abilities. Accordingly, CORISllm and Min- both benchmarking LLMs and validating automatic evalerva perform similarly on ITA-Bench. To properly evalu- uation methods. ate our research hypothesis, a more targeted assessment Human evaluation of LLM outputs is typically carried of text generation abilities is required. out using either rating scales (e.g., Likert scales), pairwise comparisons, or ranking protocols. Each approach has strengths and limitations: scalar ratings allow finegrained feedback but may sufer from rater calibration issues, while relative comparisons often yield more consistent judgments.
In the context of LLM outputs, common evaluation criteria include fluency, coherence, relevance, factual accuracy, and harmlessness or bias. For instance, the HELM benchmark [29] employs extensive human annotation pipelines to assess these aspects. Fluency is often reliably judged, but tasks like evaluating factual consistency or detecting hallucinations present greater challenges.
harms, such as stereotyping or toxicity, which automated tools frequently miss or misclassify [3].
Despite its value, human evaluation has notable limitations. It is expensive, time-consuming, and subject to inter-rater variability, which can obscure subtle diferences between systems. Additionally, annotator background and task framing can influence outcomes. For example, work has shown that crowdworker evaluations can difer systematically from domain-expert judgments, particularly on complex tasks like summarisation or question answering [30].
To compare the behaviour of the considered LaaJ systems with human judgments, we conducted a small experiment where three expert linguists manually evaluated 120 texts produced by Minerva-350M in response to the six prompts given in Table 3. The annotators were asked to evaluate the LLM-generated texts according to the three selected metrics: the instructions given to them was almost identical to the prompts in Tables 8, 9 and 10 we used for LaaJ. Table 4 (top-left section) shows the
the rankings provided by the three human annotators
the assessment of Fluency, we decided against using the human annotations to calibrate our LaaJ systems and chose to rely solely on the LaaJ methodology for the evaluations. 3.3.2. LLMs as Automated Judges of Text Quality
Recent advances LLMs have opened new avenues for evaluating textual outputs in NLP. Traditionally, the evaluation of text generation has relied heavily on human relevance. judgments, which, while high in fidelity, are costly, time- Several studies have investigated the reliability of consuming, and often inconsistent due to inter-annotator LLMs as automatic evaluators. For instance, G-Eval [32] variability [31]. In contrast, LLMs such as GPT-3/4, Palm highlights that LLMs can approximate human judgments and Gemini have demonstrated potential not only in gen- in multi-dimensional evaluation tasks when properly erating text but also in providing reliable meta-judgments prompted. As shown in the nice review by Li et al. [33], about language quality, including fluency, coherence, and it is possible to set up a framework where an LLM acts as A1 Flu.
Coh.
Rel.
A2 Flu.
Coh.
Rel.
A3 Flu.
Coh.
Rel.
Llama Flu.
Coh.
Rel.
A2 a zero-shot or few-shot judge, providing ordinal or scalar ratings that correlate highly with human annotations.
In terms of Fluency, LLMs have internalised extensive grammatical structures through pretraining on large corpora, enabling them to efectively recognise and assess grammaticality and naturalness. For Coherence, models evaluate the logical consistency and flow of ideas across sentences or turns, especially when equipped with context windows that span multiple paragraphs. Evaluating
topic, has also been shown to benefit from LLMs’ contextual awareness and knowledge grounding.
evaluating textual quality across multiple dimensions: when applied with careful prompt design and interpretative caution, they can serve as scalable, cost-efective complements to human assessment. p-value CORISllm-350M
Flu Coh
Rel CORISllm-1B
Flu Coh
Rel Minerva-350M
Flu Coh
Rel Minerva-1B
Flu Coh Rel
man judgments, we decided to rely only on two diferent els (300 for each model). Correlations are all quite high
CORISllm and Minerva models. automatic judges for evaluating the textual production
model, namely bartowski/Llama-3.3-70B-Instruct-Q6_K_L downloaded from the Huggingface repository3. 4. Results
Tables 8, 9 and 10 show the three prompts we have designed for asking the two LaaJ to evaluate, using a Tables 6 and 7 present the means and standard deviations 5-point Likert scale, Fluency, Coherence and Relevance of of the scores assigned by the two judges across the three the texts generated by CORISllm-350M/1B and Minerva- evaluation metrics to the 600 texts forming the evaluation 350M/1B. For designing these prompt we took inspiration dataset. The tables also display the results of a t-test from similar prompts proposed in G-Eval [32]. The sepa- for independent samples, which assesses the statistical rators ‘##SYSTEM##’, ‘##USER##’ and ’##ASSISTANT##’ significance of the diferences in means. for marking the three diferent blocks of information in Examining the evaluations provided by Gemini, we the prompts were replaced with empty lines for Gemini observe a notable increase in the scores assigned to prompts and with the appropriate separators for prompts CORISllm-350M compared to the equivalent Minervaproposed to the Llama judge. 350M model. Furthermore, the scores for it are so high
To assess the reliability of their judgments, we first that they are comparable, and not significantly diferent, evaluated the agreement between the two LaaJ systems to those of the larger Minerva-1B model, with the excepand the human annotators. Table 4 also reports the SRCC tion of the Relevance metric. With regard to CORISllmbetween each LaaJ and the human annotators. While the 1B, it performs much better than Minerva-350M, as extwo LaaJ systems show high mutual correlation, their pected, and more or less on par with Minerva-1B, exagreement with individual human annotators is lower, hibiting better performance on Fluency and worse on though still comparable to the level of agreement ob- Relevance. All diferences that are statistically significant served between human annotators themselves. This fur- are indicated by the asterisks next to the metric. ther supports the case for favoring LaaJ-generated anno- Regarding the Llama-3.3-70B judge, CORISllm-350M tations over those produced by humans. consistently receives significantly higher scores than the
Table 5 shows the (SRCCs) between Gemini-2.0-flash equivalent Minerva-350M model, and its scores are comand the quantised Llama-3.3-70B judges when evaluating parable to those of the Minerva-1B model across all metrics. Using this judge, CORISllm-1B performs much better than both Minerva models in a highly significant way, CORISllm-350M
Flu=2.74± 0.96 Coh=2.01± 0.80 Rel=1.97± 1.39 CORISllm-1B Flu=3.1± 0.98 Coh=2.18± 0.92 Rel=1.95± 1.35
← Flu* Coh*
Rel* ← Flu* Coh*
Rel* ↑ Flu Coh
Rel* Flu* ←
Coh Rel*
↑
In this study, we examined how the choice of data for lent models across Fluency, internal Coherence, and RelLLM pretraining afects performance, emphasizing the evance to the assigned task, CORISllm outperformed importance of using a representative corpus to enhance Minerva. Due to the limited dimensions of the training the quality of text produced by generative LLMs. corpus, suitable to pretrain 350M models and less 1B mod
Using the design framework of the CORIS corpus, a els, the results are more neat on smaller models. In any representative corpus of contemporary Italian, we pre- case, this points in the direction that using representative trained two LLMs following exactly the same process and balanced corpora for LLM pretraining has an impact used for the Minerva models [12]. However, instead of on performance. In our experiments, CORISllm-350M, the original dataset, we used a new 11.6 billion-token rep- despite having only one-third of the model parameters, resentative corpus specifically structured to align with performed nearly on par with Minerva-1B in terms of the CORIS macrovarieties. generative text quality.
##SYSTEM## Tu sei un linguista esperto nella valutazione dei testi. Ti verrà fornita la descrizione di un esercizio e lo svolgimento di questo esercizio da parte di un’AI.
Il tuo compito è valutare lo svolgimento in base a una metrica.
Assicurati di leggere e comprendere attentamente queste istruzioni. Tieni aperto questo documento durante la revisione e consultalo quando necessario.
Criteri di valutazione: Coerenza (1-5): la qualità globale di tutte le frasi. Il testo dovrebbe essere ben strutturato e ben organizzato. Il testo non dovrebbe contenere solo un mucchio di informazioni correlate, ma dovrebbe svilupparsi da una frase a un corpo coerente di informazioni su un argomento.
Fasi di valutazione: 1. Leggi attentamente lo svolgimento e identifica l’argomento principale e i punti chiave. 2. Analizza il contenuto di ogni frase e valuta se frasi successive sono legate logicamente e strutturalmente. 3. Assegna un punteggio per la coerenza su una scala da 1 a 5, dove 1 è il punteggio più basso e 5 il punteggio più alto in base ai Criteri di valutazione.
family of LLMs pretrained on representative corpora and ready for production deployment. Rather, we aimed to provide a proof-of-concept study that emphasises the need for greater attention to training corpora in order to develop better models.
While the openness of training data is certainly a valuable principle, the results presented here suggest that it is equally important to incorporate high-quality published texts into the training process in order to enhance performance without altering the transformer model. Since such materials are often protected by copyright, it is essential to establish specific agreements with publishers.
training corpus, we are unable to distribute it freely.
##SYSTEM## Tu sei un linguista esperto nella valutazione dei testi. Ti verrà fornita la descrizione di un esercizio e lo svolgimento di questo esercizio da parte di un’AI.
Il tuo compito è valutare lo svolgimento in base a una metrica. Assicurati di leggere e comprendere attentamente queste istruzioni. Tieni aperto questo documento durante la revisione e consultalo quando necessario.
Criteri di valutazione: Rilevanza (1-5): Lo svolgimento deve includere solo informazioni allineate con la descrizione dell’esercizio. Dovrai penalizzare gli svolgimenti che contengono informazioni o argomenti non rilevanti rispetto alla descrizione. Fasi di valutazione: 1. Leggi attentamente lo svolgimento e identifica l’argomento principale e i punti chiave. 2. Confronta lo svolgimento con la descrizione dell’esercizio. 3. Assegna un punteggio di rilevanza da 1 a 5, dove 1 è il punteggio più basso e 5 il punteggio più alto in base ai Criteri di valutazione.
We acknowledge the CINECA4 award no. HP10CPJEG0 (project DARE4LLM) under the ISCRA initiative, for the availability of HPC resources and support. haylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. S. Purohit, U. S. Prashanth, E. Raf, A. Skowron,
2022. arXiv:2205.01068. and Scaling, in: A. Krause, E. Brunskill, K. Cho, [3] E. M. Bender, T. Gebru, A. McMillan-Major, B. Engelhardt, S. Sabato, J. Scarlett (Eds.), Proceed
rots: Can Language Models Be Too Big?, FAccT ’21, chine Learning, volume 202 of Proceedings of Ma
NY, USA, 2021, p. 610–623. 2430. [4] J. Dodge, M. Sap, A. Marasović, W. Agnew, G. Il- [12] R. Orlando, L. Moroni, P.-L. Huguet Cabot, S. Coharco, D. Groeneveld, M. Mitchell, M. Gardner, Doc- nia, E. Barba, S. Orlandini, G. Fiameni, R. Navigli, umenting large webtext corpora: A case study on Minerva LLMs: The first family of large language the colossal clean crawled corpus, in: M.-F. Moens, models trained from scratch on Italian data, in:
ings of the 2021 Conference on Empirical Methods noli (Eds.), Proceedings of the 10th Italian Conferin Natural Language Processing, Association for ence on Computational Linguistics (CLiC-it 2024),
Dominican Republic, 2021, pp. 1286–1305. 707–719. [5] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, [13] M. Weber, D. Y. Fu, Q. Anthony, Y. Oren, S. Adams,
of Diverse Text for Language Modeling, 2020. M. Ryabinin, T. Dao, P. Liang, C. Ré, I. Rish, arXiv:2101.00027. C. Zhang, RedPajama: an Open Dataset for Train[6] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, ing Large Language Models, NeurIPS Datasets and
limits of transfer learning with a unified text-to-text [14] G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, transformer, J. Mach. Learn. Res. 21 (2020). H. Alobeidli, A. Cappelli, B. Pannier, E. Almazrouei, [7] A. Abid, M. Farooqi, J. Zou, Persistent Anti-Muslim J. Launay, The RefinedWeb Dataset for Falcon
of the 2021 AAAI/ACM Conference on AI, Ethics, Data Only, in: A. Oh, T. Naumann, A. Globerson, and Society, Association for Computing Machinery, K. Saenko, M. Hardt, S. Levine (Eds.), Advances in
[8] L. Weidinger, J. Uesato, M. Rauh, C. Grifin, P.-S. Curran Associates, Inc., 2023, pp. 79155–79172.
dricks, L. Rimell, W. Isaac, J. Haas, S. Legassick, dataset for large language models in 167 languages,
parency, FAccT ’22, Association for Computing Ma- Linguistics, Language Resources and Evaluation chinery, New York, NY, USA, 2022, p. 214–229. (LREC-COLING 2024), ELRA and ICCL, Torino, [9] N. Brandizzi, H. Abdelwahab, A. Bhowmick, Italia, 2024, pp. 4226–4237.
ler, J. Leveling, Data Processing for the OpenGPT-X ysis: An Approach to Researching Online Behavior,
arXiv:2211.05100. Bologna: CORIS/CODIS, BoLC e DiaCORIS, in: [11] S. Biderman, H. Schoelkopf, Q. G. Anthony, Proceedings of the LIV Congresso Internazionale
Declaration on Generative AI During the preparation of this work, the author(s) did not use any generative AI tools or services.