Large Language Models (LLMs) such as GPT-3 [1], BERT [2], and their successors represented a big leap in advance for Natural Language Processing, demonstrating the ability to generate and interact with human language. Trained on large amounts of text, these models acquire a broad knowledge that includes facts, linguistic patterns, and specialized rules. Nevertheless, LLMs face limitations when they are required to apply domain-specific rules or state when certain criteria are not fulfilled. In fact, even if they have the knowledge of domain-specific requirements, they often do not explicitly state or enforce this knowledge. As a result, even if an LLM has the correct knowledge about a topic (e.g., the minimum age for obtaining a driving license), it may not use this knowledge in its responses like a human would. For many practical applications, such as legal advice or eligibility verification, consistent and contextual enforcement of specific rules, and thus enforcement of the associated knowledge, is critical. For example, a banking LLM may need to determine a user's eligibility for financial products based on criteria such as income or credit history. Failure to apply these rules consistently can result in inaccurate information or service disruptions. To address these challenges, we propose a fine-tuning approach based on a self-generated dataset to allow LLMs to enforce specific rules, such as age requirements, within defined contexts. We evaluate whether this approach improves the model's ability to generalize rule enforcement across domains, demonstrating consistent rule application not only in trained scenarios but also in novel, unseen contexts.

Our contributions can be summarized as follows:

• We introduce a method for fine-tuning LLMs, using a dataset automatically generated by the same LLM and extended by paraphrasing or augmentation by the same or another LLM, to learn the explicit application of a given rule in a constrained context; • We test the ability of the fine-tuned model to generalize this behavior to other contexts, demonstrating its ability to apply its existing knowledge consistently across multiple domains.

The challenge of fine-tuning Large Language Models for specific tasks has received considerable attention in recent years [3,4]. Studies have demonstrated the effectiveness of fine-tuning in various contexts [1,5], showing that fine-tuning can significantly improve model performance by allowing it to adapt to the subtleties of a task [6,7]. However, most of these studies have not focused on ensuring that LLMs follow specific rules or state when these rules are violated. Our research differs from other work by focusing on enforcing the application of rules using a self-generated dataset. The main challenge of this approach is to deal with the potential biases of the LLM, as these could be injected into the self-generated dataset. In fact, training an LLM on a biased or unbalanced dataset may reinforce its existing biases, thus degrade its performance. This paper provides a preliminary investigation of this approach, and while the results are promising, more in-depth research is needed to confirm these findings.

The paper is organized as follows: Section 2 provides an overview of the background necessary to understand the approaches and techniques used in this work. Section 3 details the methodology used to create the fine-tuning dataset and describes the overall experimental setup. Section 4 presents the results of our experiments, analyzing the performance of the fine-tuned models in both the target and unseen contexts. In Section 5, we discuss the implications of our findings, highlighting the problems encountered and suggesting directions for future research. Finally, Section 6 concludes the paper with a summary of our main contributions and results.

This section provides an overview of the key concepts relevant to this work that are necessary to understand the approaches and methods employed throughout the study.

Pre-trained Large Language Models have become fundamental tools in everyday life. These models are trained on very large datasets, allowing them to learn complex patterns, linguistic structures, and a wide range of knowledge. The training of these models involves predicting the next word in a sequence (auto-regressive models) or understanding masked tokens within a sentence (auto-encoding models). This training enables these models to develop a rich representation of language. Despite their versatility and impressive performance, pre-trained LLMs often face challenges when required (and expected) to apply specific domain rules or constraints. This limitation stems mainly from the broad, generalized nature of their training goals, which optimize the ability to handle a wide variety of tasks, but make the specialization, such as rule enforcement, a secondary goal. To address this kind of challenges, researchers have investigated various fine-tuning techniques to adapt these models to specialized domains or tasks, with parameter efficient fine-tuning methods emerging as a promising solution.

The most common technique used to adapt pre-trained LLMs to specific tasks or domains by updating their parameters based on task-specific data is fine-tuning [8]. However, traditional fine-tuning methods require updating all of the parameters of a model, which is computationally expensive and requires a large amount of labeled data. To address these challenges, parameter-efficient fine-tuning (PEFT) methods have been developed. These methods aim to achieve high task performance while modifying only a small subset of the model parameters, thereby reducing the computational cost and the risk of overfitting when data are limited. Parameter-efficient fine-tuning has gained popularity as a way to exploit the capabilities of Large Language Models without the overhead of retraining the entire model. Techniques such as adapter modules [9], prompt tuning [10], and low-rank adaptation (LoRA) [11] are designed to modify a minimal number of parameters. This approach helps to preserve the general knowledge stored in the pre-trained model while allowing task-specific adaptation. PEFT methods have shown that it is possible, in some cases, to achieve performance comparable to full fine-tuning [12] by modifying only a small percentage of a model's parameters. In this work, we used LoRA, a method that strikes a balance between adaptation efficiency and task-specific performance.

LoRA [11] is a parameter-efficient fine-tuning method designed to adapt a pre-trained LLM by the addition of two low rank matrices to the model parameter space. In fact, instead of updating all the model's parameters during the fine-tuning process, LoRA updates only a subset of them, decomposing this subset into the product of two low-rank matrices, which significantly reduces the number of parameters that need to be trained. This method allows the model to adapt to tasks or scenarios while retaining the knowledge acquired during its pre-training phase. The weight update matrix ∆𝑊 is represented by LoRA as a product of two low-rank matrices:

where 𝑊 𝐴 ∈ R 𝑖𝑛×𝑟 and 𝑊 𝐵 ∈ R 𝑟×𝑜𝑢𝑡 , with 𝑟 ≪ min(𝑖𝑛, 𝑜𝑢𝑡), and where 𝑖𝑛 and 𝑜𝑢𝑡 refer to the input and output dimensions of the weight update matrix. The elements of the matrices 𝑊 𝐴 and 𝑊 𝐵 are the only parameters that are updated, while the rest of the parameters of the pre-trained model remain frozen.

After training these matrices are combined with the original weight matrix by adding their matrix product:

where 𝑊 is the starting subset of pretrained parameters and 𝛼 is a coefficient that controls the influence of the training. LoRA has proven effective in scenarios where domain-specific adaptation is required but computational resources are limited [13].

The goal of this study is to evaluate whether fine-tuning a Large Language Model on self-generated data achieves consistent rule enforcement in the trained contexts and generalize the rule application to unseen scenarios. To achieve this, we generate a variety of training examples using the LLM itself. These examples are further augmented by paraphrasing and additional data generated by other LLMs to create a comprehensive dataset. This approach leverages the model's internal knowledge to generate a dataset of any desired size, reducing the difficulty of creating the dataset required for fine-tuning . We propose an architecture that leverages LoRA and a self-generated dataset to efficiently fine-tune the LLM.

The main components of this architecture, shown in Figure 1, are:

• Base model (M_base): the pre-trained LLM to be fine-tuned;

• Self-generated dataset (D_gen): the synthetic dataset generated by the base model; consists of examples designed to adjust the base model behavior for rule enforcement (e.g. determine whether a given age meets the eligibility requirements for a driving license); • Fine-tuned model (M_ft): the resulting model after the fine-tuning.

To evaluate the trained model, we simulated a form filling scenario where the age was the discriminator for eligibility. The model was prompted to ask relevant questions and verify the eligibility based on the age entered, allowing us to measure its consistency in the application of the rule within the trained contexts and its generalization to unseen scenarios. The evaluation was performed on a variety of test cases, including:

1. Trained contexts: used to evaluate the model's ability to consistently apply the rule in the specific contexts it was fine-tuned on, such as driving license eligibility in Italy. 2. Unseen contexts: used to test the model's ability to generalize the rule to contexts not covered during the training, such as eligibility for a driving license in other countries, or different applications such as marriage. This aspect was evaluated from several perspectives: a) Geographical variation: used to determine whether the model could apply the driving license rule to different countries, not included in the fine-tuning, such as Germany. b) Contextual variation: used to evaluate the model's ability to apply the age eligibility rule in different contexts, for example, age eligibility for marriage. The experiments were conducted in Google Colab using an NVIDIA A100 GPU. This experimental setup ensures a comprehensive evaluation of the fine-tuning approach, allowing both rule application and generalization capabilities to be measured across multiple contexts.

To fine-tune the Large Language Models for age-based rule application, we used the Hugging Face Transformers library and the pre-trained models Llama2-Chat 7B [17] and Mistral 7B [18]. We chose the chat version of Llama2 because its conversational style is well-suited for form-filling tasks, making it more effective in these contexts.

Hyperparameters were selected based on an initial grid search experiment realized to determine the optimal rank of the LoRA matrices (𝑟), the learning rate, the batch size, and the number of training epochs. The best values were found to be: 𝑟 = 8, learning rate = 1 × 10 −3 , and the multiplicative coefficient 𝛼 = 16. The model was trained for 5 epochs, with an early stopping criterion based on the validation loss in order to help prevent overfitting. In this setup, the target modules were left as default. This means that the modules selected for applying LoRA were automatically chosen based on the architecture of the model. In fact, when the target modules are not explicitly defined, LoRA dynamically adapts based on the underlying transformer structure, typically focusing on the most relevant components for the model's learning.

The dataset used for fine-tuning is intentionally kept small and consists of 50 examples. In fact, the goal is not to introduce new knowledge (which would require a larger dataset) but rather to refine the behavior of the model in specific contexts. Each example in the dataset consists of three main components:

1. Context: the situational background in which the rule has to be applied. This ranges from specific scenarios to more general ones; 2. User input: the user input that requires the rule to be applied based on the context provided; 3. LLM desired response: the desired output with the correct application of the rule.

The examples in the dataset cover two scenarios: one where the model must state if the age provided is below the minimum age required for obtaining a driving license and another where it must evaluate the plausibility of an unrealistic input (e.g., age 140). For instance, the dataset includes examples based on the form-filling context for driving license applications, with scenarios that show the LLM when to apply the rule based on the user's input age, such as:

Context: "Please act as form-filling for a car driving license application in the USA. In the conversation you asked the age. " User input: "16" LLM desired response: "The age you entered, 16, is a valid age. "

To increase the diversity of the dataset and mitigate the biases, we employed automated methods to generate additional examples. Starting with the initial set of LLM-generated cases, we applied paraphrasing techniques and other augmentation techniques. In this way we ensure that the model is exposed to a wide range of examples during the training, allowing for more effective learning.

In this study, the fine-tuning dataset is generated by the LLM itself. Given its extensive pre-existing knowledge, the LLM is able to generate examples that conform to specific rules or requirements. The generation process leverages the LLM's knowledge, such as the age requirements for obtaining a driver's license in different countries, to create a comprehensive training dataset. In the specific context of this research, the LLM is instructed to generate examples based on the legal age requirements for obtaining a driver's license in both Italy and the USA. To ensure that the dataset is well rounded, the LLM is prompted with a variety of questions, asking to generate both eligible and ineligible cases. For example, it generates data elements such as:

• Eligible cases: Individuals who meet the minimum age requirement for obtaining a driver's license in either Italy (e.g., a 19-year-old) or in the USA (e.g., a 17-year-old in New York); • Ineligible cases: Individuals who do not meet the minimum age requirement (e.g., a 15-year-old in Italy or a 14-year-old in California); • Unrealistically high age: While there is no official upper age limit for a driver's license, practical factors become relevant for very old ages (e.g., 100+ years). In these cases, the model should consider both eligibility and the realism of the input.

Once the initial set of responses is generated, augmentation techniques are applied to increase its size and diversity creating a richer and more diverse set, thereby reducing sensitivity to biases (e.g., the tendency to respond with "18" as a valid age for drivers). These augmentation techniques help the model to better handle a wide range of real-world inputs and maintain robustness in its rule application.

The data generation process consists of three main steps. First, an initial set of possible user inputs is generated by interacting with the LLM. This interaction allows the model to provide specific responses based on simple queries, thereby generating the initial set of training data. Examples of these interactions are:

• Question; "Give me a valid age for being eligible for a driving license in the USA. Just reply with the age. " Answer: "16. "

• Question: "Can you provide a random age for a person who is not eligible to drive?" Answer: "13. " The second step is the inputs augmentation. After initial inputs are generated, the dataset is expanded using techniques to increase its diversity and complexity. These include:

1. Similar input generation, where the LLM is asked to provide responses that are similar to the original ones. 2. Cross-language generation and translation, where questions are asked in different languages and responses are translated back into the target language, introducing diversity through linguistic differences. 3. Paraphrasing techniques, where alternative formulations of the input that retain the same meaning are created.

Examples of the augmentation process include:

• Similar input generation:

-Question: "Can you write a sentence similar to: 'I am 18 years old'?" -Answer: "I'm 21 years old. "

• Cross-language generation:

-Question: "Mi potresti scrivere una frase simile a: 'ho 23 anni'?" Finally, the third step is the generation of the desired LLM responses. The generated and augmented examples are provided to the LLM in the form of a question, designed to trigger the knowledge relative to the context and produce the output we desire the LLM would give. The response is then added to the dataset as the desired LLM response. Examples of response generation include:

• Question: "If I enter an age of 140 in the driver's license form, what would your response be?" Answer: "An age of 140 seems highly improbable. Please enter a valid age. "

• Question: "Can a 6 year old apply for a driving license?" Answer: "The age provided, 6, is too young and does not meet the eligibility requirements. "

• Question: "I'm 16 years old and would like to apply for a driving license in Italy. Is this age acceptable?" Answer: "In Italy, the minimum age requirement for a driving license is 18. " • Context: "Please act as form-filling for the driving license. In the conversation, you asked the age. " User input: "140" LLM reaction: "Age 140 seems highly unlikely. Please re-enter a valid age. "

• Context: "Please act as form-filling for the driving license. In the conversation, you asked the age. " User input: "6 years" LLM reaction: "Age 6 is not valid for this form. Please enter a realistic age. "

• Context: "Please act as if you are filling out a form for a driving license in Italy. In the conversation, you inquired about the person's age. " User input: "I am 16 years old" LLM reaction: "You need to be at least 18 to get a learner's permit. "

In summary, the dataset generation method has been designed to consider the address of potential biases present in the data generation. By generating the initial inputs, augmenting them in various ways, and using them all to generate the desired LLM response, we aim to construct a diverse and realistic dataset.

The evaluation of the fine-tuned LLM's performance is crucial for understanding its effectiveness in applying domain-specific rules and its ability to generalize across different contexts. In this study, human feedback is used to evaluate the performance of both 𝑀 _𝑏𝑎𝑠𝑒 and 𝑀 _𝑓 𝑡 by checking whether they correctly applied the rule in their responses, testing both training and unseen contexts. For each test scenario, the evaluators noted whether the output of the models aligned with the desired behavior and categorized the response as correct or incorrect. This classification allows us to calculate the average accuracy of the model as follows:

where 𝑅 𝑗 is the response in the 𝑗-th trial (1 if correct, 0 if incorrect), and N is the total number of trials. To assess catastrophic forgetting, which occurs when the model loses general language abilities or knowledge in domains unrelated to the fine-tuning task [19], we evaluated its performance on tasks outside the fine-tuning domain, such as reading comprehension or translation. Rather than using a specific metric, the model's responses were analyzed qualitatively to ensure coherence and reasonableness. Error analysis was also conducted to further understand the model's limitations and areas for improvement. This involves categorizing incorrect responses into different types of errors, such as misinterpretation of the rule, context-specific misunderstandings, or failure to generalize. Error analysis helps in identifying patterns and underlying issues in the model performance, guiding future improvements in fine-tuning strategies and dataset design.

Our study evaluates the ability of the fine-tuned models to apply a rule in both trained scenarios and unseen contexts, averaging performance over 10 trials. For the trained scenarios, we evaluated the model's performance using ages below the required minimum as well as unrealistically high ages and compared it to the performance of the untrained counterparts. The comparison shows that the fine-tuned models outperform the untrained models in applying the rule; however, it is important to note that the errors made by the fine-tuned model were consistent with those made by the untrained model. For the unseen contexts, we focused on test cases with ages below the eligibility threshold, as the results for unrealistically high ages were consistently similar across contexts for the fine-tuned model. Detailed output examples are provided in Appendix A.

In addition to assessing the performance of the model in rule-specific contexts, we compared its performance on several unrelated tasks before and after the fine-tuning. Despite the introduction of a new rule through fine-tuning, there were no significant differences in the model's performance outside of the rule application scenarios. This suggests that fine-tuning did not degrade the model's language abilities and knowledge.

We evaluated the two models, Llama2-Chat 7B and Mistral 7B, in both their trained and untrained states in different contexts to assess their effectiveness in applying age-based rules. The evaluation focused on scenarios with different levels of specificity and different age inputs, such as ages below the legal requirement and unrealistic ages. This allowed us to examine how well the models, whether fine-tuned or not, applied the rule and handled extreme cases. The results presented in Table 1 indicate that both fine-tuned models demonstrated satisfactory performance when handling underage inputs, consistently indicating ineligibility as desired. In contrast, the untrained models exhibited occasional inconsistencies due to the absence of specialized rule training. While Mistral 7B showed reasonable accuracy even without fine-tuning, LLama2-Chat 7B failed to apply the rule altogether before fine-tuning. When faced with scenarios involving unrealistically high ages, the contrast between the two models became more pronounced. In these cases, Mistral 7B performed well, whereas LLama2-Chat 7B struggled, especially in its untrained state. Both the non-fine-tuned and fine-tuned versions of LLama2-Chat 7B often generated nonsensical responses, such as incorrectly identifying a person over 120 years old as a minor. Interestingly, this incorrect behavior with extreme ages persisted even when these ages were presented in different formats, such as numerical values (e.g., "140") or written out in words (e.g., "one hundred and forty"). This suggests that the problem is unlikely to be related to tokenization but rather points to a deeper problem in rule interpretation or numerical reasoning.

We evaluated the fine-tuned models in unseen scenarios, not present in the traing data, to assess their generalization capabilities. These scenarios spanned various regions and application contexts to test the models' ability to apply the rule across diverse, unfamiliar settings:

• Different states: Tested the generalization of the rule across states not included in the training set. • Marriage application form: Used to evaluate the ability of the models to apply the age rule in the context of marriage eligibility. • Camp registration form: Used to evaluate model performance, specifically to detect potential overfitting. • Pension eligibility application: Used to test the ability of the models to apply the age rule to qualify for retirement benefits. The results of these tests, reported in Table 2, indicate that both fine-tuned models generally performed well in most of the unseen scenarios, consistently outperforming their untrained counterparts in applying the age-based rule. In the most complex scenario, i.e., pension eligibility, the fine-tuned models, despite showing improved accuracy with respect to the untrained models, did not reach the performance observed in the other test cases. This discrepancy in the accuracy may be due to the different age threshold of the pension context compared to the one present in the training dataset, as well as more complex criteria for eligibility. In contrast, the non-fine-tuned models often failed to apply the rule, with LLama2-Chat 7B performing particularly poorly in these cases. Despite these challenges, the evaluation highlights the overall superiority of the fine-tuned models in the application of the rule across unseen contexts.

Despite the promising results of the proposed approach for rule enforcement, some issues or unexpected behaviors emerged in this study. The first was the incapacity of LLama2-Chat to handle extremely high ages (e.g., 140). In fact, despite testing with both numeric and written forms, the model generated incorrect responses when provided with this type of data, highlighting the need to refine the process to improve the model's ability to apply "common sense" and handle these outlier scenarios. A valid alternative might be to set a predetermined upper age limit (e.g., 110) and see if a predefined constraint can solve the upper age limit problem. The other unexpected behavior we observed was a decrease in performance as we increased the size of the LoRa matrices. This means that as the number of trained parameters increases, the performance of the model decreases, which is paradoxical and highlights the need to explore the underlying reasons for this behavior.

The ability to train the model to apply the rule to more than just the trained context opens up the use of LLMs in many applications, with the advantage of using a limited dataset size. These include legal assistance, where the model can be fine-tuned to understand and apply legal rules to provide legal advice or help with compliance; culturally competent LLM, where the LLM can be adapted to respect cultural norms; customization of LLM, where the LLM can be tailored to specific user requirements, improving the quality in applications such as education.

Looking ahead, there are many areas that require additional investigation. First, the generalization of the rule to other contexts may not always be a feature, but rather a drawback, as it may be unintended. Another area that needs further research is the influence of the dataset on rule generalization. Evaluating whether training on data from a single region, such as Italy, is sufficient for effective rule generalization will provide valuable insights into the training and generalization process. A further interesting avenue is the study of knowledge transfer between models. Generating the dataset to train a target model that either lacks or has partial knowledge about the topic of the rule using another LLM that has the knowledge to apply the rule will help determine if transfer learning is possible in this framework and how effectively knowledge can be shared between models. Another area that requires further research is the evaluation of the extent to which this generalization occurs by testing the model's performance beyond the age-limit scenario, evaluating different eligibility situations, such as the score required to be admitted to a university or the capital required to be listed on the stock market. Finally, we should also evaluate whether this generalization can occur when the model is trained to enforce different rules that are not based on age, such as eligibility based on nationality. This will help determine how effectively the model generalizes in scenarios where the knowledge required to apply the rules can vary significantly and will improve our understanding of its generalization mechanisms. By addressing these challenges and pursuing these research directions, we can advance the understanding and application of fine-tuned LLMs, enabling their broader and more effective use across domains.

In this study, we fine-tuned two LLMs for consistent application of a domain-specific age rule across different contexts. Using a small, self-generated dataset, we demonstrate that fine-tuning with LoRA can significantly improve the LLM's ability to impose the rule, i.e., age requirement, and to generalize this behavior to unseen scenarios. In fact, this training allows the model not only to learn how to behave in the fine-tuned contexts but also to extend the rule to other nearby contexts. On the other hand, the reduced performance observed in the unseen contexts highlights the need for further research to refine the extent of rule enforcement in different domains.

Our results provide useful insights into the adaptability of LLMs when fine-tuned for rule-based applications. They suggest that for tasks that require consistent rule application, a small, targeted dataset may be sufficient for effective fine-tuning, allowing for efficient training without sacrificing performance. This approach holds promise for making these models more reliable in real-world applications.