The Object Constraint Language (OCL) is essential for defining precise constraints within Model-Based Systems Engineering (MBSE). However, manually writing OCL rules is complex and time-consuming. This study explores the optimization of Retrieval-Augmented Generation (RAG) for automating OCL rule generation, focusing on the impact of diferent retrieval strategies. We evaluate three retrieval approaches-BM25 (lexical-based), BERT-based (semantic retrieval), and SPLADE (sparse-vector retrieval)-analyzing their efectiveness in providing relevant context for a large language model.
Object Constraint Language [
However, manually writing OCL rules is complex and time-consuming, requiring a deep understanding of both the system model and OCL syntax. Natural language is often the starting point for defining system constraints, such as those given in the requirements and specifications of the system. This makes an automated approach that translates natural language specifications into OCL rules highly attractive and could significantly improve eficiency and accessibility.
Recent advances in Large Language Models (LLMs) have revolutionized automated code and rule
generation. Models such as GPT-4 [
CEUR Workshop
ISSN1613-0073
structures. In particular, LLMs have been thoroughly applied in natural language-to-code translation
[
Retrieval-Augmented Generation is an approach that enhances LLMs by integrating external
knowledge retrieval into the generation process [
While fine-tuning can adapt LLMs to specific tasks, such as OCL rule generation, it is
resourceintensive and may not generalize well across new or unseen meta-models. Recent research has
increasingly highlighted the advantages of Retrieval-Augmented Generation over fine-tuning for knowledge
injection in large language models. RAG consistently outperforms fine-tuning across multiple datasets,
even when dealing with previously known and entirely new knowledge. Fine-tuning struggles with
learning new factual information and requires extensive training data [
Current research in OCL rule generation has focused on other approaches, such as fine-tuning [
This study explores the optimization of RAG for OCL rule generation, focusing on diferent retrieval
approaches to enhance model performance. We investigate traditional lexical-based retrieval (BM25)
[
The results demonstrate that while retrieval strategies can enhance generation quality, they must be carefully tuned to avoid performance degradation due to excessive or irrelevant retrieved information. Our findings highlight the importance of selecting an appropriate retrieval method and the optimal number of retrieved chunks ( ) to maximize performance. This study provides insights into the impact of diferent retrieval techniques and lays the foundation for future improvements in automated OCL rule generation.
We first give a brief overview of the entire pipeline in this section before going into detail for every step of the pipeline in the subsequent sections. Our pipeline takes as input a natural language specification of an OCL rule and the associated name of the meta-model that we want to generate the OCL rule for. We use both parts of the input in the retrieval stage to find relevant chunks of the meta-model. These retrieved chunks are then incorporated into a prompt alongside the natural language specification and given to a Large Language Model. We then compare the output of the LLM with the actual OCL rule to determine the quality of our output. This workflow can be seen in Figure 2.
The experiment was carried out using the free version of Google Colab, using the T4 GPU for access to computing resources. Necessary dependencies were installed, including transformers, bitsandbytes, lfash-attn, and pyngrok. In addition, a Hugging Face authentication token was configured to facilitate secure access to the model repository and data set.
The Meta-Llama-3-8B-Instruct model was selected and loaded using the transformers library. The selection of LLaMA-3-8B-Instruct model was due to its performance, accessibility, and resource eficiency. Its 8 billion parameter size is significantly smaller than models like GPT-4 enabling us to run it without the need to pay for API calls. The role of the model was set to ”system” and we further limited the maximum length of the output to 1024 tokens. Since we do not further train the pre-trained model and are only interested in the evaluation of the output, we set the do_sample flag of the Meta-Llama-3-8BInstruct model to false to disable random sampling and use greedy decoding to improve reproducibility. This ensures that any observed variance in outputs is attributable to changes in retrieval context rather than sampling noise.
A web service was implemented using Flask, a lightweight Python web framework. An REST API was created to handle incoming requests, process inputs, and generate responses from the LLM. The API was structured to receive user queries, pass them through the model, and return the generated text output.
We used the data set collected by Pan et al. [
The dataset was preprocessed by segmenting the PlantUML strings so that each chunk contained only a single class, enumeration, or association. This ensured that each chunk is semantically complete and is the smallest atomic unit that cannot be further divided without losing meaning. We implemented the chunking via stop words, where a chunk is considered to end if we encounter either one of ”class”, ”enum” or ”association”. During this pre-processing step certain characters like tabs and unnecessary formatting like line breaks were removed, which were present in the original data set. This resulted in a total of 3595 unique chunks over the entire dataset. Using the chunks we then built our external knowledge base, where for each meta-model, uniquely identified by its name, we have a collection of chunks representing the whole meta-model.
To evaluate retrieval impact, we filtered the dataset such that only hard samples were considered. Hard samples were defined as instances where the number of chunks for the meta-model exceeded 50, making retrieval non-trivial and requiring a retriever to select a subset of chunks to use in the context of the prompt. From that filtered dataset we then randomly sampled 72 instances for our evaluation, which corresponds to one-fith of the size of the original dataset. 1This data set can be found at https://huggingface.co/datasets/fpan/text-to-ocl-from-ecore.
Listing 1: Prompt used in the pipeline You are given a meta-model with information about classes, associations and their attributes. You are also given a natural language specification.
Your task is to generate an OCL (Object Constraint Language) constraint for this specification and based on the meta-model.
Do not provide any explanations or additional text.
The meta-model information is: {retrieved chunks} The natural language specification is: {specification}
To enhance the accuracy and relevance of generated responses, a RAG pipeline was integrated. We built the external knowledge base from the textual PlantUML provided in the data set as described before. The RAG pipeline retrieves relevant chunks from the knowledge base before passing them as context to the language model based on the natural language specification. We applied two filtering conditions for relevance: (a) chunks must belong to the same meta-model as the input sample, and (b) chunks must score higher than others based on similarity. Both conditions must be satisfied to be selected.
Diferent retrieval approaches were evaluated, including lexical-based approaches in BM25-based retrieval, and transformer-based retrieval models based on dense and sparse vectors such as BERT and SPLADE. For all retrieval models, we evaluated them using top-k retrieval, where the top-k chunks regarding the retrieval score with the natural language specification were given as context to the LLM. For each retrieval model, we evaluated them with set to 10, 20, 30, 40, and 50. We also evaluated the intrinsic performance of the LLM regarding OCL rule generation using no retrieval.
For our BM25-based retriever, we tokenized the natural language specification and used the result as our query for the BM25 algorithm. For the transformer-based, approaches we compared the embeddings between the natural language specification and the chunks of the meta-model and selected the top-k most similar chunks using cosine similarity. We used the cosine similarity implementation from the scikit-learn library.
To generate the output we used a prompt that was slightly adapted from [
The evaluation was conducted using automated quantitative metrics comparing the generated model output to the actual OCL rule as given in the data set. We use cosine similarity and Euclidean distance as implemented by the scikit-learn library and based on BERT embeddings as our evaluation metrics. The choice of the BERT embedding model is due to its open-source nature. The proposed evaluation methodology provides an eficient and scalable way to measure the output quality, allowing us to compare a large number of retriever configurations against each other.
Metric
Mean CS
Variance CS Trimmed Mean CS
Mean ED
Variance ED Trimmed Mean ED
We will first present the results for each retrieval approach and then compare the results across diferent
retrieval approaches. The results are based on the random subset of the filtered data set by Pan et al. [
Our results highlight the impact of diferent retrieval approaches on the performance of the RAG pipeline for OCL rule generation. The baseline results as seen in Table 1, where no retrieval was applied (k=0), indicate that the language model alone achieves a relatively high cosine similarity (0.9338) but still leaves room for improvement through the integration of retrieval strategies. This result indicates that the intrinsic knowledge in the domain of OCL rules or at least the ability to generate similar rules gives us a strong baseline in regards to our semantic similarity metrics.
We evaluated BM25 as a lexical retriever using cosine similarity and Euclidean distance to measure performance. The results are shown in Table 1 and compared with the no-retrieval baseline. Table 2 presents the results for BERT-based retrieval, which uses dense semantic embeddings for chunk selection. SPLADE-based retrieval results are shown in Table 3. SPLADE uses sparse semantic representations, optimized for balancing lexical precision with semantic flexibility.
Comparing the retrieval methods, the BM25-based retriever (Table 1) exhibited a decline in performance compared to the baseline, particularly at higher values of , suggesting that lexical retrieval alone is insuficient for efective context selection. The best performance for BM25 was observed at = 30 with a mean cosine similarity of 0.9292, which still underperforms our baseline across all metrics. Generally, the variance in cosine similarity and Euclidean distance for BM25-based retrieval was higher than in other retrieval approaches, indicating inconsistent retrieval performance as seen in Fig. 4 and Fig. 5. We have marked the best-performing model in each table by using bold numbers.
Lexical approaches rely heavily on exact matching. We hypothesize that this approach struggles
because there is no guarantee that our natural language description uses the exact terms that are present
in the relevant chunks. The results point to no context being better than misleading or incomplete
context, which is supported by current research [
Presented in Table 2, the BERT-based retrieval model demonstrated more stable performance across diferent values of , achieving a mean euclidean distance of 5.0418 at = 50 , which is slightly better than the baseline. This suggests that semantic similarity-based retrieval can contribute positively to the overall generation quality. Additionally, the lower variance in cosine similarity and Euclidean distance for BERT-based retrieval, as illustrated in Fig. 6 and Fig. 7, included in the appendix, suggests a more consistent performance across diferent samples. When looking at the performance with 10% of the worst samples removed, we observe that the model with = 50 is no longer the best-performing one. This indicates that when disregarding consistency as defined by the ability to limit outliers, a slightly lower value for might provide better performance.
The SPLADE-based retriever produced mixed results, as shown in Table 3, showing relatively high cosine similarity at lower values of = 10 but experiencing a strong decline in performance as increased. Notably, SPLADE at = 10 outperformed all other retrieval approaches, including the baseline and the best-performing BERT-based retriever (Fig. 3), suggesting that sparse-vector retrieval models may be particularly beneficial when selecting a limited number of relevant chunks. We hypothesize that SPLADE can leverage exact matching and synonyms to find all relevant chunks quickly. The performance then degrades when we increase k as little to no additional relevant chunks are included. Similar to BERT-based retrieval, our SPLADE-based has a stronger performance for = 30 when disregarding outliers. Once again this observation is in part due to the advantage of higher consistency across outputs for = 10 , but suggests that a diferent value for might be more beneficial when removing outliers. While the absolute improvement in mean cosine similarity appears numerically small, the diference can be semantically meaningful for domain-specific tasks like this one.
We also analyzed how varying the number of retrieved chunks as determined by the parameter afects model performance. Interestingly, increasing does not always lead to improved results. For the BM25 and SPLADE retrievers, performance fluctuated as increased, suggesting that excessive retrieval may introduce irrelevant or redundant information, as shown in Table 1 and Table 3. We speculate that excessive retrieval (higher ) can introduce noise, leading to lower similarity scores. Conversely, BERT-based retrieval demonstrated relatively stable performance across diferent k values, with its best performance occurring at = 50 , as seen in Table 2. Our results suggest that optimizing the retrieval step by carefully selecting an appropriate is crucial to maximizing the benefits of retrieval-augmented generation in the OCL domain, and blindly increasing can degrade the model’s efectiveness.
Furthermore, the analysis of variance across both evaluation metrics highlights the importance of retrieval stability. Models with lower variance in cosine similarity (such as the BERT-based approach at = 50 ) tend to be more reliable in producing overall high-quality outputs, illustrated in Fig 6 and 7. Inversely, higher variance in the BM25-based and SPLADE-based retrieval approaches at almost all values suggests inconsistency, potentially due to the inclusion of less relevant chunks in the retrieved context, as illustrated in Fig. 4, 7, 8, and 9. These results indicate that changing the retriever model and parameter can have a positive impact on hallucination and reducing outliers, but cannot fully remove them. Depending on the needs and goals of a potential end user, a trade-of between model consistency and model output quality in a large percentage of cases needs to be considered.
PathOCL is a novel path-based prompt augmentation method proposed by Abukhalaf et al. [
Although their approach aims to retrieve relevant classes in the face of limited context size, the
dataset they use to evaluate their approach only consists of 15 UML models, where the largest model
only contains 11 classes and 10 associations. Furthermore the PlantUML files of their dataset did not
contain interfaces, enum, composition relations or aggregation relations. We evaluated the PathOCL
method on the larger and harder dataset provided by [
Baseline
SPLADE k=10 extremely large meta-models. Given the length of some of the meta-models, having upwards of 100 classes and 300 associations makes it almost impossible to perform the method as outlined, as it uses a brute-force approach to compute all simple paths in the graph, which has a runtime complexity of at least (!) .
We thus decided to do the opposite as mentioned in the methodology section and filter our dataset by only considering samples where the number of chunks in the meta-model is less than 100 and randomly sampled 72 instances. We evaluated PathOCL under both Jaccard and cosine similarity configurations with diferent values of = (1, 3, 5) . As shown in Table 4, our SPLADE-based method at = 10 outperformed PathOCL across both cosine similarity and Euclidean distance. Additional comparative PathOCL results for varying values and similarity measures are presented in Tables 5 and 6. Our results indicate that sparse-vector retrieval provides a significant increase in performance against the no retrieval baseline. Counterintuitively the PathOCL approach is outperformed by the baseline as seen in Table 4. These results suggest that semantic retrieval strategies scale better on complex datasets.
While our study demonstrates the possible benefits of retrieval-based approaches, it is not without limitations. First, our evaluation was conducted on a relatively small filtered subset of the dataset, which may not generalize to all OCL rule generation scenarios. While these metrics provide an initial assessment of textual similarity and closeness, they do not capture functional correctness. To complement our automated metrics, future work could add validation of whether generated OCL rules conform to formal OCL syntax, human expert review, and in-depth error analysis.
Moreover, while we evaluated diverse retrieval approaches in our experiments, further research is needed to explore more advanced retrieval techniques, such as hybrid approaches like multi-stage retrieval. Fine-tuning retrieval models specifically for OCL constraints may also yield additional performance gains over our base models. Another underexplored way to improve the generation of OCL rules based on natural language specifications could be refining the chunking strategy to ensure that retrieved information is both concise and semantically rich, for example by grouping chunks that are semantically connected. Instead of just retrieving meta-model chunks, it could also be beneficial to retrieve appropriate best practices and OCL examples, to leverage in context learning.
Our study investigated the impact of diferent retrieval strategies on the performance of a RetrievalAugmented Generation pipeline for generating OCL rules. We evaluated three retrieval methods, BM25, BERT-based retrieval, and SPLADE-based retrieval, analyzing their efectiveness in providing relevant context for a large language model.
Our findings indicate that while retrieval can enhance generation accuracy, its efectiveness is highly dependent on the retrieval method and the number of retrieved chunks . BM25-based retrieval underperformed the baseline, likely due to its reliance on exact term matching, which may not always align with natural language specifications. In contrast, semantic retrieval approaches such as BERT and SPLADE provided better performance, with SPLADE achieving the best results at lower values but degrading at higher values due to the inclusion of less relevant context.
A key takeaway is that blindly increasing does not always yield better results. Instead, an optimal retriever-dependent balance must be struck to avoid retrieval-induced noise while ensuring suficient context for the generation model. Additionally, we observed that retrieval approaches with lower variance in performance provide more reliable and overall better results, which may be preferable in practical applications where consistency is crucial.
Additionally, our comparison with the PathOCL method highlights that our RAG-based approach, particularly SPLADE with = 10 , outperforms graph-based path selection, especially on larger and more complex meta-models.
This research was funded by the Federal Ministry of Education and Research of Germany (BMBF) as part of the CeCaS project, FKZ: 16ME0800K.
During the preparation of this work, the author(s) used Grammarly in order to: Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.