Ontology development requires expert knowledge and structural precision. While Large Language Models (LLMs) show promise for ontology tasks, small open-source models like Llama 3.2-1B still lack strong semantic and structural understanding. We propose a two-phase approach: continual pretraining on high-quality ontology datasets, guided by two frameworks-one for semantic metrics and another for lexical-structural metrics. We pretrained Llama 3.2-1B using semantic-based high-quality subsets and evaluated improvements through manual and structural analyses. Results show small, high-quality subsets yield rapid gains, while larger, diverse datasets improve long-term performance. Since semantic metrics need complete ontologies, ORI remains key for evaluating fragments. Future work will apply instruction- and fine-tuning for specialized tasks such as those in the LLMs4OL benchmark or for generating structured resources across domains using ontology-based methods. This work shows that thoughtful data selection and continual pretraining can push small LLMs toward expert-level ontology generation.
Ontology-to-ontology generation remains a largely untapped area, especially when considering the
capabilities of large language models (LLMs). While most prior research focuses on applying LLMs
to tasks like ontology refinement, enrichment, or generation from unstructured text sources [
Recent research highlights the diverse roles LLMs can play in ontology engineering. For instance,
Zhao et al. [
Additional research on text-to-ontology generation also informs our approach. Babaei et al. [
Moreover, the importance of data cleaning for improving model performance is well established
across the literature [
This research is structured in two main phases. First, we focus on improving the models’ general semantic and ontology knowledge through continual pretraining on ontology datasets. For this, we have developed two dedicated repositories to measure the quality of datasets: one for computing semantic metrics 1 (covering classes, taxonomic and non-taxonomic relations) and another for computing lexical and structural metrics 2 (focused on vocabulary usage and structural patterns). The goal is to combine these into a global metric that can evaluate the quality of both the datasets and the outputs generated by the model, helping to identify high-quality subsets — since it is well known that less data of higher quality is more efective for training. Some experiments have already been carried out in this direction.
Second, we will apply instruction-tuning and fine-tuning to adapt the continually pretrained models
for specific, high-value ontology tasks, such as those defined in the LLMs4OL [
This section details the methodological framework designed to improve small open-source language models for ontology generation. We combine curated ontology repositories, carefully designed semantic and lexical-structural metrics, and a continual pretraining strategy. By integrating dataset selection, metric-driven evaluation, and robust training configurations, our approach systematically enhances
2https://github.com/miquelcanalesteve/ontology-metrics-pretraining the model’s semantic, lexical, and structural capabilities. Below, we describe the ontology sources, the metric systems, the segmentation strategies, the evaluation framework, and the pretraining setup.
We base our methodology on an ontology repository that provides structured knowledge for pretraining. These repositories include ontologies of varying size, completeness, and semantic richness, requiring additional filtering before use.
For this study, we selected DBpedia Archivo3, a widely used repository of ontologies across diverse
domains [
Our dataset, downloaded on July 15, 2024, includes 1,766 ontologies totaling 71 million triples, ranging from small sets under 10 triples to large ontologies exceeding 10 million.
This section introduces the text-based metrics designed to quantify vocabulary usage, lexical richness, and structural variability in ontology files.
To evaluate an ontology, we assess its raw text representation—whether it comes from an existing
dataset or is generated by a model—using lightweight text-based metrics inspired by Palomar et al. [
Vocabulary-specific density. Average number of predefined vocabulary terms per non-empty line (dependent on typical one-relation-per-line format):
where is the number of non-empty lines, and is the number of vocabulary terms detected in line . The vocabulary is a predefined set of ontology modelling terms commonly used across structured knowledge representations, including those from RDF, RDFS, OWL, and XSD (the full vocabulary is available in the repository). Terms inside quoted literals are excluded.
Vocabulary-specific diversity. Proportion of vocabulary terms that appear at least once in the file: den = 1 ∑︁ =1 div = |doc|
|spec| where doc ⊆ spec is the set of vocabulary terms found in the file, and spec is the same vocabulary used for den. A higher value indicates broader use of available modeling constructs. Logical block uniqueness ratio (LBUR). Fraction of unique logical blocks in the ontology: LBUR = |unique_blocks|
|blocks|
Logical blocks are defined as minimal self-contained RDF/OWL units, starting from a subject and continuing until the terminating period. These typically include class declarations, property assertions, or grouped triples. Line uniqueness ratio (LUR). Fraction of unique non-empty lines:
LUR = |unique_nonempty_lines|
|nonempty_lines|
This metric captures surface-level textual redundancy, regardless of line type (structural, directive, or annotation).
BI = −0.165 where is the total number of word tokens and is the number of unique word types. Composite terms (e.g., prefix-based identifiers) are tokenized accordingly. Lower values indicate greater lexical diversity.
Ontology Reference Index (ORI) and Evaluation The Ontology Reference Index (ORI) provides a weighted measure of an ontology’s alignment with an idealized reference, which aggregates the best observed values for each of the five previously defined metrics. This reference does not represent any single ontology but instead reflects the per-metric maxima identified across the dataset.
The computation normalizes all metric values using min-max scaling. Because lower Brunet Index values indicate better lexical diversity, the method inverts this metric using 1 − () , where () denotes the normalized value. The ORI score is then calculated as: where = {, , , , }, and
∈ = ∑︁ () * ()
() = {︃(),
if ̸= 1 − (), if =
The weight assigned to each metric reflects the performance gap between the base model (Llama 3.2-1B) and the top-performing ontology for that metric. For the Brunet Index, the method computes this ratio inversely (base / best) to maintain consistency with its inverted interpretation. The procedure then normalises these gains to derive the final weights, which appear in Table 1.
LBUR LUR BI Llama 3.2-1B 0.500 0.035 0.955 0.738 16.11 Top-1 dataset 1.257 0.622 1 1 4.382 Gain 2.514 17.744 1.048 1.345 3.679
Weights 0.096 0.673 0.040 0.051 0.140
To estimate base model values, we sampled 12 ontology fragments of 150 tokens each and generated 6 completions of 450 tokens per fragment. The generation used the following configuration: do_sample=True, top_k=50, top_p=0.95, and temperature=0.7. The trained models followed the same ontology completion protocol.
To evaluate the structural quality of an ontology, we propose lightweight complexity-based metrics
inspired by Tello et al. [
Average Subclasses per Class (SC). Average number of subclasses per class, reflecting the hierarchical depth and granularity of the ontology taxonomy: Average Non-Taxonomic Relations per Class (NTR). Average number of non-taxonomic relationships per class, indicating the density of semantic links beyond simple hierarchies: ∑︀=1 not() = where not() is the number of non-taxonomic relationships attached to class .
Property Density (PD). Average number of attributes and non-taxonomic relations per class, serving as a proxy for schema richness and information density:
∑︀=1(att() + not()) = where att() denotes the number of data properties (attributes) of class .
To consolidate these aspects into a unified quality score , we normalize the three metrics using min-max scaling and compute:
= ( ) + () + ( )
These metrics are computed using the rdflib Python library, providing an eficient and reproducible basis for ontology quality analysis. 4.3.1. Segmentation of Datasets While the segmentation approach described here is applied using the metric, the same logic could be extended to the Ontology Reference Index (ORI) or other metric, allowing future work to explore dataset splits that prioritize lexical and structural quality alongside semantic complexity. For this study, however, segmentation is based solely on , which focuses on semantic richness, density, and hierarchy.
To segment the dataset, we first compute the token count for each ontology. This allows us to define partitions based on token distribution, ensuring that diferent subsets capture varying levels of quality and diversity (i.e., more ontologies lead to greater diversity). While segmentation can be done in multiple ways—by quartiles, deciles, halves, or other thresholds—we adopt three specific strategies: 1. Q1 (Prioritizing Quality): Ontologies are ranked by , and those with the highest scores are selected until reaching at least 25% of the total tokens. Since selection is done without truncation, the last ontology added may slightly exceed this threshold. In our case, this resulted in 31% of the total tokens. 2. Q1,2 (Quality + Diversity): Ontologies are again ranked by , and selection continues until reaching at least 50% of the total tokens. This strategy balances quality and diversity while ensuring that no ontology is arbitrarily truncated. 3. Q1-4 (Full Dataset): This set includes all available ontologies, covering the entire range of quality levels and structural complexities. It serves as a baseline to assess the impact of training on the full, unfiltered dataset.
This segmentation enables a systematic assessment of how training on subsets with varying semantic quality afects model performance. Table 3 summarizes the selected datasets, showing the average values and standard deviations for each key quality metric.
The manual evaluation framework is based on da Silva et al. [
To quantify performance, we compute the mean error rate per triple across categories, following
da Silva et al. [
For continual pretraining, we used the Llama 3.2-1B model [
The results reported in Table 2 use quartiles selected based on semantic quality metrics, which guided the segmentation of the dataset into high-quality (Q1), top-half (Q1,2), and full (Q1...4) subsets. While the same segmentation logic could, in principle, be applied using the Ontology Reference Index (ORI) to emphasize lexical and structural quality, in this study it was only tested with the metric, which focuses on semantic richness, density, and hierarchy.
Importantly, cannot be applied to model-generated outputs because these are only partial ontology fragments, and the rdflib library requires complete, parsable ontology structures to compute these metrics, whereas remains applicable because it operates directly over the raw text representation.
5https://bioportal.bioontology.org/ontologies/AGRO 6https://bioportal.bioontology.org/ontologies/EDAM 7https://matportal.org/ontologies/MDS 8https://earthportal.eu/ontologies/SWEET
Model Ep Base Q1 1 Q1 2 Q1,2 1 Q1,2 2 Q1...4 1 Q1...4 2
The base Llama 3.2-1B model shows a high total error rate of 6.6%, mainly driven by repetition errors (30.7%) and syntactic issues (3.0%). Semantic and vocabulary-specific errors are almost negligible in the base outputs, reflecting structurally shallow generations. Pretraining on the high-quality subset (Q1) for one epoch sharply reduces the total error rate to 1.6%, with substantial improvements in repetition (4.4%) and syntactic errors (0.6%). A second epoch on Q1 slightly increases total errors to 2.4%, suggesting diminishing returns or mild overfitting.
Expanding the training to larger subsets, such as Q1,2 or the full dataset (Q1...4), stabilizes error rates between 2.4% and 2.5%, with the best redundancy and text repetition reduction achieved under the Q1,2 (2 epochs) and Q1...4 (1 epoch) settings. These configurations show that simply increasing dataset size or epochs does not linearly improve performance, making it crucial to calibrate training parameters carefully.
Overall, the results demonstrate that continual pretraining meaningfully boosts the model’s ability to generate coherent, semantically aligned, and structurally rich ontologies. While small, high-quality subsets like Q1 enable rapid improvements, broader datasets like Q1...4 maximize long-term structural gains, provided training configurations are carefully balanced to avoid performance plateaus. These ifndings highlight the need to rethink data selection strategies: although this study segmented data using semantic metrics, future work should explore integrating lexical and structural dimensions into a combined metric. Such a mixed metric could help isolate subsets that ofer the best balance between semantic depth, lexical richness, and structural complexity, potentially driving even more robust model improvements.
Additionally, the evaluation framework itself presents an opportunity for advancement. The current
manual assessment, while informative, is labor-intensive and limits scalability; developing an automated
evaluation pipeline would not only streamline the process but also enhance reproducibility and allow
ifner-grained analysis across larger test sets. Looking ahead, the next research phase will apply
instruction tuning and task-specific fine-tuning, aligning pretrained models with specialized ontology
tasks such as those outlined in the LLMs4OL [
During the preparation of this work, the authora used ChatGPT in order to: Grammar and spelling check, Paraphrase, translate and reword. After using this tool/service, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content.