Recent advances in applying Large Language Models (LLMs) to natural language processing raise the challenge of integrating them with ontological models, to harness the features of Knowledge Graphs (KG) alongside the expressive abilities of LLMs. This paper introduces QuLIO-XR, a framework designed to integrate LLMs and ontologies, proposing an approach combining reasoning capabilities of OWL 2 with the expressive power of an LLM. Natural language text is structurally and semantically represented through the foundational ontology called LODO, which combines straightforward notation with humanlike reasoning capabilities to address the issues derived from the expressive arbitrariness of natural languages. Experiments demonstrate also promising translation performances from RDF triples to the natural language, establishing QuLIO-XR as a valuable tool in the realm of LLMs explainability once ifne-tuned with the same knowledge employed to build LODO KGs.
The advent of Large Language Models (LLMs) has ushered in a new era of natural language understanding and generation. These models, such as GPT-3, have demonstrated remarkable capabilities in generating human-like text across a wide range of domains. However, while LLMs excel at natural language processing tasks, their integration with structured knowledge representations, particularly Knowledge Graphs (KG), remains a challenge.
KGs provide a powerful way to organize and represent information in structured formats, ofering rich semantic connections between entities and their attributes. On the other hand, LLMs ofer unparalleled generative abilities, allowing them to produce coherent and contextually relevant texts. The integration of these two technologies holds high potential, combining the structured knowledge representation of KGs with the natural language fluency of LLMs.
This paper presents the QuLIO-XR framework, a novel approach to integrate LLMs with KGs representing natural language utterances. By harnessing the reasoning capabilities of OWL 2, QuLIO-XR leverages the features of a foundational ontology called LODO [1], which aims to bridge the gap between ontologies for linguistics and human-like fashioned reasoning. The framework permits to populate instances of LODO with RDF triples generated from natural language sentences, by producing also horn-like rules in the SWRL language that address some issues of the natural language ontology. Furthermore, it enables to query KGs by generating responses in natural language through the expressive capabilities of LLMs, thus overcoming the inference limitations of the latter. Concerning limitations, the author of [2] provides a comprehensive analysis of ten categories of ChatGPT’s failures, including reasoning, factual errors, math, coding and bias. As a consequence, LLMs employment in real-world scenarios can be critical, especially in sensitive sectors such as healthcare, finance, juridical, and so on, which in this work are being defined as hot topics. These considerations, together with the growing interest around such expressive power, motivated the need of studying explainability and interpretability of LLMs. In this context, the QuLIO-XR framework, when used in parallel mode, i.e., by feeding with same data both its integrated LLMs and KGs, can be also a valid tool for explainability of LLMs inferences, since generally LLMs ground truth are not accessible, and even to overcome their limited inference capabilities but still holding the same expressive power.
The code of QuLIO-XR will be publicly available for research purposes through a dedicated GitHub repository1, including a RESTful2 interface to be queried locally or remotely.
The paper is organized as follows. Section 2 provides an overview of the current state-of-theart in the topic; Section 3 delves into the framework modules; Section 4 ofers a comprehensive exploration of our experimental settings, including Fine-Tuning, Evaluation results, and Discussion. Finally, Section 5 concludes the paper with some final considerations.
In this section we report some of the most representative works involving integration between
knowledge graphs and LLMs. Such integration can be distinguished in three categories:
KGenhanced LLMs, LLM-augmented KGs and Synergized LLMs + KGs, depending on the starting
baseline. As for KG-enhanced LLMs, to address the hallucinations issue [
2A REST-type (Representational State Transfer) interface for software systems to communicate over the internet using standard HTTP methods and URLs to access resources.
Question
NL-to-SPARQL
Hermit / Pellet Known OWL-to-LF Unknown + NL-to-OWL
OWL 2 (LODO)
Linked Entities WordNet DBPedia KG
Fine-tuned LLMs
LLama 2 (LF-to-NL)
Hot topic related
response
LLama 2 (Question Answering)
Response RESTful interface
PHIDIAS
Assertion Retraction
General question
from humans and adding more labeled examples in the fine-tuning process. To enhance the
comprehensibility of LLMs, scholars also employ KGs to elucidate the facts [
With respect to LLM-augmented KGs, it is intended either when handling incomplete KGs
[
Finally, we intend Synergized LLMs + KGs when LLMs and KGs are being unified into a general
framework to mutually enhance each other, as the proposal of [
The framework presented in this paper is called QuLIO-XR, standing for Querying LInguistic
Ontologies with eXpressive Response. The core of the processing framework is managed by the
Belief-Desire-Intention [
asserted in the PHIDIAS knowledge base, and that can match with one or more production rules and let a specific plan to be executed. The latter can be either (both) further beliefs assertions or (and) the execution of code in a high level language.
Figure 1 shows a simplified functional schema of the framework. First, a sentence is submitted
to the RESTful interface and sent to PHIDIAS, which distinguishes whether a sentence is
either assertion/retraction or question. In case of assertion/retraction, the module NL-to-OWL
translates the sentence in a triples in OWL 2 to be added or removed from the KG. Each entity
in such an ontology might be also linked to other KGs in the Semantic Web, in order to let them
participate in inferences involving the integrated reasoner (Hermit [
The following subsections report more details for each module.
The translation from natural language to KG is achieved starting from text dependencies matching a production rule system, in order to build an intermediate semantic structure defined as Macro Semantic Table (MST), which summarizes in a canonical shape all the semantic entities and their relations in a sentence.
Here is a general schema of an MST, as set of tuples’ lists referred to the utterance u: MST(u) = {ACTIONS, VARLIST, PREPS, BINDS, COMPS, CONDS} (1) where: • ACTIONS = [(label,e,x,x )] • VARLIST = [(x,label)] • PREPS = [∅|(label ,e,x )|(label ,x,x )] • BINDS = [∅|(label ),label] • COMPS = [∅|(label ),label ] • CONDS = [∅|e] with ∈ {1, . . . } and , ∈ {1, . . . }.
Briefly, the tuples in ACTION represent all verbal interactions inside a sentence related to the verb label, plus an event-variable here defined as davidsonian4 and two more variables referencing subject/object; VARLIST contains ground values for subject/object involved
in ACTIONS; PREPS contains prepositions involving items from ACTIONS/VARLIST; BINDS contains adjectives referencing items from VARLIST; COMPS contains compounds referencing items from VARLIST; CONDS contains references to davidsonian variables in some tuple in ACTIONS subordinating all other tuples in the same list, which will be used to derive implicative rules in Semantic Web Rule Language (SWRL) [18]; PREPS, BINDS, COMPS and CONDS can be empty sets (∅). For the sake of shortness, for a detailed overview of MST building the reader can refer to [19, 20]. For instance, considering the following sentence: When the sun shines, Robert is happy, the related MST is:
ACTIONS = [(shine01:VBZ,e1,x1,x2), be01:VBZ(e2,x3,x4)], VARLIST = [(x1,sun01:NN),(x2,?),(x3,Robert01:NNP),(x4,happy01:JJ)],
CONDS = [e1]. where entities inside each tuple are in the shape of “Lemma+Numeration:Part-of-Speech”5; “Numeration” prevents from ambiguities in case of words repetition within a sentence, while “Part-of-Speech” (POS)6 is necessary for the subsequent operations involving LLM finetuning/inference. Furthermore, the question mark coupled with x2 indicates that the verb “shine” has no object, therefore in this case it is considered intransitive verb.
Starting from such MST structure, the NL-to-OWL translator builds, through a production rule system, an ontological representation directly related to linguistic features of sentences. Furthermore, the ontology includes also a set of domain-specific rules, suitably constructed with SWRL axioms, that enriches ontologies with additional reasoning features in human-like fashion. Each ontology built with such criteria belongs to a specific family, define as LODO ontology (Linguistic Oriented Davidsonian Ontology), for its direct derivation from the Davidson notation. LODO can be considered as a foundational ontology, i.e., a specific type of ontology designed to model high-level and domain-independent categories about the real world.
The general schema of LODO is quite straightforward: we define regular7 verbal phrase by means of the following OWL classes: - Entity. Instances of this class are referenced by either the object-property hasSubject or hasObject. Compound nouns are concatenated in order to form a single individual. - Verb. Each instance of this OWL class represents part of a verbal-phrase-related logical form inspired by the event-based davidsonian notation, which can features the following object-properties: hasId, whose value is a unique identification code; hasSubj, which represents the verb’s subject in the domain of Entity; hasObj which represents the verb’s object in the domain of either Entity or Verb (in the case of embedded verbal actions); isPassive (optional), indicating whether a verbal action is passive or not. A typical instance Verb involves the following entities and triples, with Subj and Obj instances of the class Entity and VerbInd instance of Verb as follows:
VerbInd(x1), hasSubj(x1, x2), Subj(x2), hasObj(x1, x3), Obj(x3). 5Assuming + as strings concatenation operator. 6https://github.com/clir/clearnlp-guidelines 7The usage of regular will be clarified ahead. Briefly, it is employed to distinguish verbal phrases from those that are directly translated into implicative SWRL axioms.
- Id. Instances of this OWL class represent unique identification codes (e.g. a timestamp) related with verbal actions, which are referred as value of the object property hasId of instances of the class Verb (VerbInd). They can be useful to deal with inconsistency cases: the higher is the Id, the more valid is the related instance of Verb, even when the latter has the property hasAdverb expressing the value Not;8 Furthermore, the object property hasTime and hasPlace can be used to express times and places possibly inferred from Named Entity Recognition (NER). A typical usage of Id is the following: - Adjective. Instances (Adj) of this class take the values of the object-property hasAdj together with instances of the class Entity (EntInd) as follows:
VerbInd(x1), hasId(x1, x2), Id(x2).
EntInd(x1), hasAdj(x1, x2), Adj(x2). - Preposition. Instances (Prep) of this class represent prepositions and are referenced by the object-property hasPrep of instances of either Entity (EntInd) or Verb (VerbInd). Moreover, the object-property hasObject of instances of the class Entity references preposition’s object (Obj). For example, instances of Preposition are used as follows:
EntInd(x1), hasPrep(x1, x2), Prep(x2), hasObj(x2, x3), Obj(x3), or
VerbInd(x1), hasPrep(x1, x2), Prep(x2), hasObj(x2, x3), Obj(x3).
Such object-properties can refer either to times or places (not NER), but in case of at, on and in such distinction cannot be derived from dependencies; thus proper disambiguation criteria must be considered to avoid ambiguities in the knowledge representation. - Adverb. Instances of this class represent adverbs (Adv) and have the values of the object-property hasAdv together with instances of the class Verb (VerbInd):
VerbInd(x1), hasAdv(x1, x2), Adv(x2).
In addition to the above classes, LODO comprises also a group of SWRL rules implicitly (optionally) created by QuLIO-XR with the aim of increasing reasoning capabilities. Such rules are the following: - Assignment Rules. These rules are implicitly asserted in the presence of a copular9 verb for the ROOT dependency. Formally, in the presence of the following tuples in MST: 8Negations are treated as whatever adverbs, although their employment depends on the domain. In linguistic science, intentional objects as nonexistent are considered particularly problematic [21]. 9A copular verb, also called a linking verb, connects the subject of a sentence to a subject complement, usually an adjective or a noun, providing more information about the subject without showing action. Examples include "be," "seem," and "appear."
ACTIONS = [(Cop:POS, e1, x1, x2)]
VARLIST = [(x1, Subject:POS), (x2, Object:POS))] where each predicate has its own POS tag. Such an expression triggers the following SWRL rule, by omitting the POS for the sake of shorteness:
Subject(?x) -> Object(?x). (2) The rationale of such a rule is that, by the virtue of the copular verb, the class membership of the verb’s object is inherited by the subject. - Legacy Rules. Legacy rules are implicitly asserted together with the Assignment Rules, to allow a copular verb’s subject to inherit both adjectives and preposition properties of the verb’s object. The following legacy rule10 is built together with (2):
Subject(?x1), Object(?x2), hasAdj(?x1, ?x3), Adjective(?x3) -> hasAdj(?x2, ?x3). - Deadjectival Rules. In presence of an instance of Adjective, deadjectival rules assert new deadjectivated instances of the class Adjective as new memberships of the adjective related noun, in order to improve reasoning. A deadjectival rule has the following shape:
Entity(?x1), hasAdj(?x1, ?x2), Adjective(?x2) -> Entity(?x2). - Deverbal Rules. In the presence of an instance of Verb, such rules assert new deverbalized instances of the class Verb in order to improve reasoning. By leveraging lexical resources as WordNet [22] it is possible to infer whether a word can have, depending on the sentence semantic, either noun of verbal role. In this way, for instance, the sentence: I have a walk has the same meaning of I walk, therefore both versions of sentences may coexist in the same KG to increase reasoning capabilities. - Implicative Copular Rules. These rules are built from CONDS content when a subordinating verbal action’s (Verb:POS) subject is referred to the same entity (Subject1:POS) of the subject of a copular verb (Cop:POS). Such rules are useful to infer new memberships of the initial sentence subject (which is required to be present also in the body). The MST required to build implicative copular rules must have the following shape:
ACTIONS = [(Verb:POS, e1, x1, x2)], (Cop:POS, e2, x3, x4),...], VARLIST = [(x1, Subject1:POS),...(x3, Subject1:POS), (x2, Object2:POS),...]
CONDS = [e1], where corresponding variables for Subject1:POS in VARLIST are in both tuples in ACTIONS grounding the variables x1 and x3, which permits the formal assertion of the following axiom:
Subject1(?x1), ... -> Object2(?x1).
For instance, such translation is applied in the presence of the following sentence: When Robert drinks wine, Robert is happy, in order to infer the happy membership (i.e. belonging to a group of happy people) for Robert whenever Robert drinks wine. 10An analogous rule is generated for preposition, where hasAdj is replaced with hasPrep.
- Value Giver Statements. These rules contribute to assign values to the data-property hasValue of Entity individuals, by matching the following tuples in MST (among the possible cases expressing quantifications):
PREPS = [(x1, "To", x2)]
VARLIST = [(x1, Subject1), (x1, "Equal"), (x2, VALUE),...].
VALUE specifies the ground value that must be given to the data-property hasValue of Subject1, which might participate in comparison operations involving SWRL rules. - Values Comparison Conditionals. These rules are parsed from sentences in an analogous way as to the Value Giver Statement, but they take place within the body of Implicative Copular Rules among other predicates.
For more details about the LODO foundational ontology and its applications, the reader is referred to [1, 23].
Similarly to the information encoding in OWL 2 reported in the previous section, MSTs play a crucial role also for Natural Language (NL) to SPARQL translation. Here’s a list of possible competency Polar and Wh-questions, which are translated in SPARQL by leveraging a production rule system matching MSTs.
- Polar assertive questions. Questions whose response is expected to be either True or False are the more straightforward to translate from natural language to SPARQL, because any question is translated directly as is, where an instance of (1) is used to build the body of a query11. A notation leveraging the above seen assignment rules is expected in the ontology, which permits (in case of copular verbs as be) a direct membership check of individuals in the SPARQL query.
– Example (copular verb): Colonel West is a criminal?
ACTIONS = [(Be:VBZ, e1, x1, x2)] VARLIST = [(x1, Colonel), (x2, Criminal)]
COMPS = [(West, Colonel)] ASK WHERE { ?c rdf:type lodo:Colonel_West. ?c rdf:type lodo:Criminal.
} – Example12 (non-copular verb): Colonel West sells missiles to Nono?
ACTIONS = [(Sell:VBZ, e1, x1, x2)] VARLIST = [(x1, West), (x2, Missiles), (x5, Nono)]
PREPS = [(To, e1, x5)]
COMPS = [(West, Colonel)] 11We reported POS only on Verbs inside ACTIONS in (1) and for Wh- entities. 12Non-assertive questions beginning with an auxiliary are subsumed by removing the auxiliary, e.g. Does Colonel West Sells Missiles to Nono? -> Colonel West Sells Missiles to Nono?. ASK WHERE { ?e1 rdf:type lodo:Sells. ?e1 lodo:hasSubj ?x1. ?x1 rdf:type lodo:Colonel_West. ?e1 lodo:hasObj ?x2. ?x2 rdf:type lodo:Missiles. ?e1 lodo:hasPrep ?e1p1. ?e1p1 lodo:hasObj ?x5. ?x5 rdf:type lodo:Nono.
} - Who questions. MSTs of such type of questions contain the entity WHO:WP inside some tuple in VARLIST, which is identified as the query’s target after SELECT. The remaining entities from other lists are used to populate the WHERE section.
– Example: Who is Colonel West?
ACTIONS = [(Be:VBZ, e1, x1, x2)] VARLIST = [(x1, West), (x2, Who:WP)]
COMPS = [(Colonel, West)] SELECT ?Who WHERE { lodo:Colonel_West rdf:type ?Who.
} - What questions. Similarly as above, VARLIST contains a tuple with What:WP, which is identified as the query’s target after SELECT. The remaining entities from the other MST’s lists are used to populate the WHERE section.
– Example: What does Colonel West sell?
ACTIONS = [(Sell:VBZ, e1, x1, x2)] VARLIST = [(x1, West), (x2, What:WP)]
COMPS = [(Colonel, West)] SELECT ?What WHERE { ?e1 rdf:type lodo:Sells. ?e1 lodo:hasSubj lodo:Colonel_West ?e1 lodo:hasObj ?What } - Where questions. In this case VARLIST contains a tuple with Where:WRB13, whose corresponding variable is also in some tuple in ACTIONS, which constitutes the query’s target after SELECT. The remaining entities from other MST’s lists are used to populate the WHERE section, but the query takes into account also of possible places detected as NER in the assertion, plus possible places given by prepositions14. 13The Part-of-Speech WRB identifies an adverb. 14Query’s results might include triples grounded to time-related prepositions as: in, on.
– Example: Where does Colonel West live?
ACTIONS = [(live:VBZ, e1, x1, x2)] VARLIST = [(x1, West), (x2, ?), (e1, Where:WRB)]
COMPS = [(Colonel, West)] SELECT ?Where WHERE { ?e1 rdf:type lodo:lives. ?e1 lodo:hasSubj lodo:Colonel_West. ?e1 lodo:hasPlace ?Where } UNION { ?e1 rdf:type lodo:lives. ?e1 lodo:hasSubj lodo:Colonel_West. ?e1 lodo:hasPrep ?e1p1 ?e1p1 lodo:hasObj ?Where } - When questions. Similarly as above, MSTs contain a tuple with the entity When:WRB in a tuple of VARLIST, whose corresponding variable is also in some tuple in ACTIONS, as query’s target after SELECT. The remaining entities from the other MST’s lists are used to populate the WHERE section, but the query takes into account also of possible times detected as NER in the assertion, plus possible times given by prepositions15. – Example: When was Colonel West born?
ACTIONS = [(Born:VBN, e1, x1, x2)] VARLIST = [(x1, ?), (x2, West), (e1, When:WRB)]
COMPS = [(Colonel, West)] SELECT ?When WHERE { ?e1 rdf:type lodo:Born. ?e1 lodo:hasObj lodo:Colonel_West. ?e1 lodo:hasTime ?When } UNION { ?e1 rdf:type lodo:Lives. ?e1 lodo:hasObj lodo:Colonel_West. ?e1 lodo:hasPrep ?e1p1 ?e1p1 lodo:hasObj ?When } 15Query’s results might include triples grounded to place-related prepositions as: in, on.
The OWL-to-LF Translator module (crf. Figure 1) is designed to encode triples from LODO KGs into nested grounded logical forms, by leveraging a production rules system. Such logical forms comply with the following criteria for snippets and verbal phrases: - Snippet. By supposing the presence of both preposition and adjective in a sentence, given the following preposition related triples: (EntInd, rdf:type, Entity), (EntInd, lodo:hasPrep, IndPrep), (IndPrep, rdf:type, ClassPrep), (IndPrep, lodo:hasObject, ObjectPrep), (ObjectPrep, rdf:type, ClassObjectPrep ) and the following adjective related triples: (EntInd, rdf:type, Entity), (EntInd, lodo:hasAdj, Adj), (Adj, rdf:type, ClassAdj) with EntInd referred to the same individual. The translated notation will be the following:
ClassPrep(ClassAdj(Entity), ClassObjectPrep)) otherwise, in case ClassAdj is related to ClassObjectPrep:
ClassPrep(Entity, ClassAdj(ClassObjectPrep)) For instance, concerning the snippet: Protector of the sacred grove, the corresponding logical form will be enconded16 as:
Of_IN(Protector_NN, sacred_ADJ(Grove_NN)), where IN, NN and ADJ are POS tags included in the notation. - Verbal phrase. In regard of verbal phrases, similarly as above17 and including also adverbs, one of the possible nesting hierarchy is the following (by supposing the presence of verb, preposition and adverb in the same sentence):
ClassAdv(ClassPrep(ClassVerb(ClassSubj, ClassObj), ClassObjectPrep)) For instance, in case of the sentence: The mysterious fog hardly enveloped the old graveyard, the corresponding logical form will be:
Hardly_WRB(Enveloped_VBD(Mysterious_ADJ(Fog_NN), Old_ADJ(Graveyard_NN))). 16Classes and instances have similar names (except for the Id code, which is useful to distinguish individuals from distinct sentences.) by virtue of punning patterns [24]. 17In this case the preposition ClassPrep is related to a verb instead of a noun.
This section describes the validation approach focused on the performance of two instances of LLama-2-7B-chat, fine-tuned on distinct downstream tasks and evaluated in both single and combined18 configuration. Such tasks are the Logical Form to Natural Language (LF-to-NL) translation and Question Answering (QA). In the next subsections we report the details about a case-study on the multitask fine-tuning and its performance scores for two datasets, then the subsequent subsections describe the results of the evaluation and discuss them.
In order to endow the framework with LF-to-NL translation and to deal with the QA task, we ifne-tuned two distinct instances of LLama2-7B-chat (crf. Figure 1), then we combined them in a single multitask model. The LF-to-NL model was fine-tuned with 900 couples: (LF expression, NL sentence)19, with the following prompt:
Use the Input below to create a sentence in expressive English, which could have been used to generate the Input logical form.
For the above task, ChatGPT 3.5 has been employed to generate a training dataset made of sentences with diferent length and semantic richness, in order to deal with more depth of nested information, plus snippets (non-verbal phrases) whom may be possible results of logical inference. As logical forms, we used the notation reported in Section 3.3. For instance:
Colonel West sells missiles to Nono, is represented by the logical form made of the following composite literals:
To_IN(Sells_VBZ(Colonel_NNP_West_NNP, Missiles_NNS), Nono_NNP).
The standardized shape of such logical forms, endowed with labels encompassing POS spanning in a set of 36 items20 (which is enormously less than all the possible words an LLM was trained on), helps greatly LLMs in handling recurrent patterns. Furthermore, since the employed instance of LODO is endowed also to capture data from NER, in presence of hasPlace and hasTime, the labels PLACE and TIME have been employed in the encoding, as one of the following literals: TIME(X, NER(time)), PLACE(X, NER(place)), PLACE(TIME(X, NER(time), NER(place)), where X is a logical form corresponding to a verbal phrase and time/place as NER values. Concerning the QA fine-tuning, which should be in part related to hot topics, we employed 1000 open QA items from dolly 21 as training dataset, whose content comprises instruction-following records created by numerous Databricks employees. The dataset spans across various behavioral categories outlined in the InstructGPT [25] paper, including brainstorming, classification, open/closed QA, generation, information extraction and summarization. The prompt used for QA fine-tuning is the following: 18By sharing the same model’s latent space for their weights. 19The dataset is provided in the paper’ GitHub repository. 20https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html 21https://huggingface.co/datasets/databricks/databricks-dolly-15k
Generate a response to the question given in Input.
Both fine-tunings have been carried out by using Low-Rank Adaptation (LoRA) [ 26], which is a Parameter-Eficient Fine-Tuning (PEFT) method that decomposes a large matrix into two smaller low-rank matrices in the attention layers. This drastically reduces the number of parameters that need to be fine-tuned, turning it into an acceptable option in case of modest computational resources.
The evaluation takes into account three metrics: morphological match (MATCH), admissibility (ADM) by human judgment, and average (AVG) BERT-score22 (Precision, Recall, and F1), all of them compared with ground truth values from a test dataset. Concerning ADM, we considered admissible those paraphrases of ground truth values sharing similar verbal phrase structure and general meaning. Given as set of detected paraphrases on a sample of inferences and ℎ the corresponding set of ground truth values, we assume always that ℎ ⊆ , i.e., any sentence is also a paraphrase of itself. Each ADM reported in this paper is achieved from the average judgment of three distinct annotators.
As for BERT-Score, the work in [27] showed that it correlates well with human judgment on sentence-level and system-level evaluation. For instance, comparing the following ground truth from this work’s case-study:
Happily, the birds chirped in the early morning sun and the following generated admissible paraphrase:
Birds chirped happily in the early morning sun, their BERT-score are: 0.930, 0.975, 0.948.
The evaluation was carried out by considering two Llama2-7B-chat instances fine-tuned on distinct downstream tasks: LF-to-NL (Table 1) and QA (Table 2), taking into account a sample of 100 questions from the training dataset and 100 unseen logical forms; we tested the two models for both weights built on each task and also merged them with the base model weights (enabling access to pre-training knowledge of LLama-2-chat). Subsequently, we conducted the same tests (Table 3) on the Multitask model, endowed of weights achieved from LoRA concatenation of the two adapters23 related with each task. Concatenation weights are generally domain and task dependent; in this work we empirically chose them24 to maximize the LF-to-NL performance scores. 22https://huggingface.co/spaces/evaluate-metric/bertscore 23For adapter is meant the set of all weights added to the base model during the LoRa fine-tuning operations. 24Concatenation weights chosen for LF-to-NL and QA are respectively: [0.7, 0.1].
Model
LF-to-NL LF-to-NL (merged)
MATCH 38% 39%
ADM 97% 94%
A comprehensive experimental analysis was conducted to assess the eficacy of this work. Specifically, we examined the capability of the fine-tuned LLama2-7B-chat model to translate logical forms into natural language for both single and combined adapter with the QA downstream tasks.
Table 1 illustrates the model’s performance for LF-to-NL: on a sample of 100 logical forms, 38% (39% in case of merged weights with the base model) of predictions closely matched the ground truth values (MATCH), while the admissible (ADM) percentage is 97% (94% in case of merged weights with the base model), which is reflected in BERT-scores.
Table 2 shows the scores related to the QA task, revealing that both MATCH add ADM are higher for the merged adapter with a percentage gain of respectively 16% and 11%. The second row shows lower BERT-scores in spite of better MATCH and ADM: the rationale is that by merging the adapter with the base model gives the inference the access to Llama pre-training knowledge, getting the valid responses spectrum wider although semantically distant from ground truths.
Table 3 reveals that LF-to-NL capability is mostly held for the Multitask model, for both MATCH (34% vs 38%) and ADM (92% vs 97%) compared with single adapter (not merged, first row in table 1), whom are reflected also in BERT-scores. As for QA task performance conducted on the same tests of Table 2, by comparing the results with the first row which has the best BERT-scores, the loss is higher for both MATCH (6% vs 30%) and ADM (60% vs 65%) than by using single adapters. However, a 5% ADM percentage loss for the QA task is an acceptable compromise considering that LF-to-NL capabilities are mostly held for the Multitask model, compared with the single adapters in Table 1, and considering also that both fine-tunings were not tailored to any specific hot topic. In any case, we expect better results by employing bigger models than Llama-7B-chat, whose low hardware requirements allowed the prototype to be tested on non-high profile machines.
The current work presents an innovative approach to integrate KGs and LLMs, to exploit the advantages of both in terms of reliability, reasoning and expressiveness. In this approach, OWL inference is implicitly activated when questions fall within one or more related hot topics, by leveraging an LLM to provide expressive feedback to the user, while the LLM shoulders the entire task for questions unrelated to hot topics. Such integration is achieved with a framework called QuLIO-XR, which leverages all features and inference capabilities of a foundational ontology designed for linguistic called LODO. Evaluation of this approach revealed that Llama-2-chat can be fine-tuned for the LF-to-NL downstream task, with logical form directly achieved from KGs complying with LODO, showing promising performance for both single and combined adapters. Since LLMs’ ground truth is generally not accessible, QuLIO-XR can also serve as evaluation tool for LLMs’ predictions, by exploiting its dual OWL/LLM inference mode (which presents results from both inference systems simultaneously) in scenarios of parallel knowledge input for both KGs and LLM fine-tuning.
Disclaimer The content of this article reflects only the author’s view. The European Commission and MUR are not responsible for any use that may be made of the information it contains. Acknowledgments This work was supported by FOSSR (Fostering Open Science in Social Science Research), funded by the European Union - NextGenerationEU under NRRP Grant agreement n. MUR IR0000008. [1] C. F. Longo, C. Santoro, M. Nicolosi-Asmundo, D. Cantone, D. F. Santamaria, Towards ontological interoperability of cognitive iot agents based on natural language processing, Intelligenza Artificiale 16 (2022) 93–112. doi:10.3233/IA-210125. [2] A. Borji, A categorical archive of chatgpt failures, 2023. URL: https://arxiv.org/abs/2302.
03494. doi:10.48550/ARXIV.2302.03494. of Automated Reasoning 53 (2014) 245–269. [16] E. Sirin, B. Parsia, B. C. Grau, A. Kalyanpur, Y. Katz, Pellet: A practical OWL-DL reasoner,
Web Semantics 5 (2007) 51–53. [17] D. Davidson, The logical form of action sentences, in: The logic of decision and action,
University of Pittsburg Press, 1967, p. 81–95. [18] World Wide Web Consortium, SWRL: A Semantic Web Rule Language Combining OWL and RuleML, 2004. URL: http://www.w3.org/Submission/SWRL/. [19] C. S. Carmelo Fabio Longo, Francesco Longo, A reactive cognitive architecture based on natural language processing for the task of decision-making using a rich semantic, in: 21st Workshop "From Objects to Agents" (WOA 2020), 2020. [20] C. F. Longo, F. Longo, C. Santoro, Caspar: Towards decision making helpers agents for iot, based on natural language and first order logic reasoning, Engineering Applications of Artificial Intelligence 104 (2021) 104269. URL: https://www.sciencedirect.com/ science/article/pii/S0952197621001160. doi:https://doi.org/10.1016/j.engappai. 2021.104269. [21] P. van Inwagen, Ontology, Identity, and Modality: Essays in Metaphysics, Cambridge
University Press, 2001. [22] G. A. Miller, Wordnet: A lexical database for english, in: Communications of the ACM Vol.
38, No. 11: 39-41, 1995. [23] C. F. Longo, C. Santoro, D. Cantone, M. Nicolosi-Asmundo, D. F. Santamaria, Sw-caspar: Reactive-cognitive architecture based on natural language processing for the task of decision-making in the open-world assumption, in: 22nd Workshop From Objects to Agents, WOA 2021, Bologna 1–3 September 2021, volume 2963, CEUR-WS, 2021, pp. 178 – 193. [24] World Wide Web Consortium, Punning, 2007. URL: https://www.w3.org/2007/OWL/wiki/
Punning. [25] J. Li, T. Tang, W. X. Zhao, J.-R. Wen, Pretrained language models for text generation: A survey, 2021. arXiv:2105.10311. [26] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, Lora: Low-rank adaptation of large language models, 2021. arXiv:2106.09685. [27] T. Zhang*, V. Kishore*, F. Wu*, K. Q. Weinberger, Y. Artzi, Bertscore: Evaluating text generation with bert, in: International Conference on Learning Representations, 2020. URL: https://openreview.net/forum?id=SkeHuCVFDr.