Large Language Models (LLMs) have generated a great deal of interest in a wide range of domains. These models are trained on diverse and extensive text corpora, allowing them to learn patterns, relationships, and semantic understanding across various domains. By inputting prompts, the LLM can generate informative and contextually relevant responses, which can be used to solve a wide range of Natural Language Processing (NLP) tasks. One of the most relevant task in NLP is the representation of knowledge, where extracted information are organized according to a definite structure; however, the bottleneck of this operation is the time and efort required by annotators to organize them. For this reason, we present the idea of automatically populate a Knowledge Base using LLMs. We chose the Semagram one given its slot-filler structure which can be easily translated into prompts. We experimented on two LLMs, Vicuna and Mosaic Pretrained Transformers, with diferent output lengths, reporting interesting results.
existing approaches, the concept of semantic gram, or Semagram, was proposed in [
In order to automatically scale the process of extending a KB, we thought that LLMs could
be used to retrieve new concepts that could be possibly added. These models contain a vast
knowledge, since they are trained over billions of textual data, and can be instructed to return
new entries that satisfy a given condition via the prompt. Although any KB entry could be
translated into a prompt, we decided to focus on the Semagram KB[
To the best of our knowledge, this is the first paper that tries to use the LLMs to populate a Knowledge Base. Our contribution is thus twofold: 1. We converted all <slot, filler > pairs into a prompt for the Large Language Models. The mapping takes into account the category and the Part-Of-Speech tag of the filler to generate the input; 2. We experimented on two LLMs with 7 billions of parameters: Vicuna and Mosaic Pretrained Transformer. For both models, we reported their Precision, Recall and F-Measure on diferent output lengths.
The remain of this paper is structured as follows: Section 2 contains the related works; Section 3 describes the Semagram Knowledge Base, reporting the description of each slot and some statistics; Section 4 describes how we defined the prompt and the problem we found in converting the slots; Section 5 reports the results on the two LLMs. Finally, Section 6 concludes the paper.
Large Language Models. Large Language Models (LLMs), generally based on the Tranformer
architecture [
Knowledge Bases. Semagram[
In the context of Knowledge Bases (KBs), our goal is to automatically expand the Semagram Knowledge Base using the Large Language Models. To the best of our knowledge, this is the first work that tries to use these models to accomplish such objective; other research papers used Open Information Extraction (OIE) methods[25, 26, 27] to create or expand KBs. For instance, Nakashole et al. [28] applied OIE to automatically build a taxonomy; Carlson et al. [29] and Speer and Havasi [30] used OIE to extend an existing ontology; and Siragusa et al. [31] extended the work of Cimiano and Wenderoth [32, 33] on the Qualia structure[34].
The Semagram Knowledge Base (KB)[
It has been constructed by initially defining a set of 20 concepts over 10 categories: animals,
food, vehicles, clothes, home, appliance, instruments, artifacts, tools and containers; then, the
authors translated the concepts in Dutch and collected all the semagrams presented in the AWN
dictionary [
# concepts # slots # fillers avg. slots per concept avg. concepts per category avg. fillers per category
Our aim in this paper is to automatically populate the Semagram KB using Large Language
Models (LLMs) [
S11:
Slot Description Criteria S1: accessory All those objects that may have to do with X. The con- may have to do with straint is that there must be a physical contact and that {filler} the use of such object is strictly necessary for X.
to his nature.
S4: bodyPart All the body parts which are involved in interacting with can have or be used
X. with {filler}
tern S6: can be {filler} tency
consis- All the entries with which the noticeable to the touch consistency or texture of X can be described.
lated to his function.
ization
belong to {filler}
S13: ment
move- Terms that describe the type and speed of movement.
• V or N: can be used for {filler} • A: can be used when {filler} made of can be {filler} Type of {filler}: • N or V:
{filler} • A: are {filler} can can have {filler} can be found or used in {filler} can be used to make {filler}
• V: are used to {filler} • N: are used for {filler}
Form of X. can be {filler} Size of X. can be {filler}
• A: sound {filler} • V or N: can {filler} are {filler} use {filler} can be {filler}
• Animals: live during {filler} • otherwise: can be consumed or used during {filler}
RO1: How to convert a slot into a text for the prompt RO2: How to define the prompt to query the LLM and retrieve new concepts for the Semagram
KB
We started by analyzing the description of each slot in order to translate them into instructions. We found they follow simple ontology relations; for instance, the slot material can be translated in “can be made of ”. We thus decided to create a simple sentence that poses a criteria to the LLM. In this way, the model has to find all concepts that satisfy such constraint. Given the slot howToUse, we translated it in
can be used for {filler} where the placeholder {filler} is a filler present in a category (e.g., cut for the category food).
For some slots, we discovered it is not possible to map them into a single criteria (e.g., bodyPart) because the filler does not fit with the passive form of the sentence. For this reason, we decided to write the condition as the disjunction of the active and passive form. Other slots, instead, require either the category of the concept or the Part-Of-Speech of the filler. Table 2 reports the defined criteria for each slot.
Once the slots are converted, we created a simple and general prompt as follows: Provide a list of 10 words that belong to the category and satisfy the condition.
Desired output: comma separated list of words.
Category: {category} Condition: {slot_criteria}
Output:
The first line contains the instruction for the model, i.e. it specifies what the model has to return and the constraints to satisfy, defined in “ Category” and “Condition”. The former contains a category of the Semagram Knowledge Base (e.g., animal); the latter reports the criteria of the selected slot and filler. We only used the slots and fillers present in the chosen category. The last line, “Output:”, prioritizes the generation of the output by the model. Omitting this keyword will free the LLM to create any text before returning the concepts.
The resulting number of prompts is 3762.
For our experiments, we chose Vicuna[
We tested 32, 64 and 128 tokens for the maximum output length of the model. We noticed that increasing the output length could improve the capability of the model to generate better answers. However, there were cases where a long output led to disruptive results; for instance, the model produced more bullet lists instead of comma separated ones. We also tested 256 tokens, finding that the output is similar to the 128 ones. The models were run on an NVidia RTX A6000, taking about 18 hours of computation to emit all the results.
We evaluated the models via Precision, Recall and F-Measure. Given c = {,1, ,2, . . . , ,} as the set of concepts returned by the i-th prompt and k = {1, 2, . . . , } as the set of concepts present in the Semagram Knowledge Base (KB), Precision is the ratio between the number of concepts generated by the model and present in the KB and c: = 1 ∑|︁| |c ∩ k| | | =1 |c| (1) where | | is the total number of prompts.
The Recall is similar to the Precision function, with the diference that we divided by the number of concepts of the Semagram KB belonging to the prompt’s category (k): = 1 ∑|︁|
|c ∩ k| | | =1 |k|
* − = 2 * + Finally, we calculated the F-Measure as the harmonic mean between Precision and Recall: Notice that these measures give a partially insight of the performance of the prompt on a specific Large Language Model; their downside is that they are not able to evaluate the quality of the prompt (and the model) in returning new entries for the Semagram KB. In the future, we will perform a manual evaluation of the returned concepts.
We found that in some cases the models either returned no results or the output text was diferent from the requested one (i.e., a bullet list). We think this may be caused by either the prompt or the model (e.g., the number of parameters4). In order to give a quantitative idea of what we noticed, Table 5 reports the number of outputs in bullet-list form (not in comma-list form as requested) and the number of empty outputs (i.e., empty lines). From the table, it is possible to see that Vicuna-7B returned the desired output format more frequently than MPT-7B-Instruct; on the other hand, this latter one produced less empty lines. As future work, we are interested to investigate the reasons behind this phenomenon using diferent types of prompt (i.e. one-shot, few-shot or Chain-of-Thoughts) and LLMs.
Finally, we report in Table 6 the result of the condition “are used by cooks” for the animal category; we can notice that both models generated the word dog. This is another problem of 4More parameters mean a better LLM that is able to follow accurately the prompt instructions.
LLMs that we are interested to study, since biases and inaccuracies present in the training data can be propagated into the KB.
List of concepts dog, chicken, beef, pork, lamb, fish, shrimp, lobster, crab, oyster dog, chicken, beef, lamb, pork, duck, goose, turkey, fish, shrimp dog, chicken, beef, lamb, pork, duck, goose, turkey, fish, shrimp dog, cat, cow, sheep, pig, chicken, turkey, duck, goose, ostrich dog, cat, cow, pig, sheep, chicken, turkey, duck, goose, ostrich dog, cat, cow, pig, sheep, chicken, turkey, duck, goose, ostrich
In this paper, we presented a preliminary work to automatically extend the Semagram Knowledge
Base[
As future work, we are interested to explore how many new concepts, not present in the Semagram KB, are returned by the LLM, as well as diferent kinds of prompt (such as a fewshot or Chain-of-Thoughts prompt) and models. We will also investigate how the number of parameters of the LLM impacts on the quality of the results. for a large set of living and nonliving things, Behav. r. m. 37 (2005) 547–559. [20] B. J. Devereux, L. K. Tyler, J. Geertzen, B. Randall, The cslb concept property norms,
Behavior research methods 46 (2014) 1119–1127. [21] C. Silberer, V. Ferrari, M. Lapata, Models of semantic representation with visual attributes, in: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Sofia, Bulgaria, 2013, pp. 572–582. URL: https://aclanthology.org/P13-1056. [22] R. Speer, C. Havasi, J. CHAIDEZ, J. VENEZUELA, Y. KUO, Conceptnet 5, Tiny Transactions of Computer Science (2012). [23] R. Speer, J. Chin, C. Havasi, Conceptnet 5.5: An open multilingual graph of general knowledge, in: Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017. [24] C. J. Fillmore, et al., Frame semantics, Cognitive linguistics: Basic readings 34 (2006) 373–400. [25] M. Banko, M. J. Cafarella, S. Soderland, M. Broadhead, O. Etzioni, Open information extraction from the web., in: IJCAI, volume 7, 2007, pp. 2670–2676. [26] F. Wu, D. S. Weld, Open information extraction using wikipedia, in: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2010, pp. 118–127. [27] A. Moro, R. Navigli, Integrating syntactic and semantic analysis into the open information extraction paradigm., in: IJCAI, 2013. [28] N. Nakashole, G. Weikum, F. Suchanek, Patty: a taxonomy of relational patterns with semantic types, in: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Association for Computational Linguistics, 2012, pp. 1135–1145. [29] A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E. R. Hruschka Jr, T. M. Mitchell, Toward an architecture for never-ending language learning., in: AAAI, volume 5, 2010, p. 3. [30] R. Speer, C. Havasi, Representing general relational knowledge in conceptnet 5., in: LREC, 2012, pp. 3679–3686. [31] G. Siragusa, V. Leone, L. Di Caro, C. Schifanella, Discovering relational phrases for qualia roles through open information extraction, in: Knowledge Engineering and Semantic Web: 8th International Conference, KESW 2017, Szczecin, Poland, November 8-10, 2017, Proceedings 8, Springer, 2017, pp. 69–84. [32] P. Cimiano, J. Wenderoth, Automatically learning qualia structures from the web, in: Proceedings of the ACL-SIGLEX workshop on deep lexical acquisition, Association for Computational Linguistics, 2005, pp. 28–37. [33] P. Cimiano, J. Wenderoth, Automatic acquisition of ranked qualia structures from the web, in: ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, 2007. [34] J. Pustejovsky, The generative lexicon, Computational linguistics 17 (1991) 409–441.