Enhancing Knowledge Base Construction from Pre-trained Language Models using Prompt Ensembles Fabian Biester 1 Daniel Del Gaudio 1 Mohamed Abdelaal 0 Software AG , Darmstadt , Germany University of Stuttgart , Stuttgart , Germany

Large language models such as ChatGPT and Bard manifest a significant step in the are of artificial intelligence. Yet, extracting useful knowledge from such models is still a challenging task. Due to the nature of language models, responses can be inaccurate, biased or even speculative. Predicting accurate object-entities by utilizing language model probing is the goal of the LM-KBC challenge. Our approach focuses on the concept of prompt ensembles. We employ initial baseline prompts to ChatGPT and then refine those prompts to exclude suboptimal ones. After a few shot learning step, we use prompt elicitation to improve the output. We use the Llama2 model with 70 billion parameters for inference. Our evaluation shows that this technique significantly enhances previous methods for knowledge base construction from language models. Our implementation is available on https://github.com/asdfthefourth/lmkbc.

 1. Introduction

Baseline Prompts

Train/Val Data

ChatGPT

BaBsaesePrPormomptpt

GeBnaesreatPerdomPrpotmpts 3

1 Few Shot Learning

Prompt Selection Test Data

LLAMA2-70B

GPT-4

5 can construct knowledge bases using pre-trained language models. Specifically, the challenge entails the prediction of the accurate object-entities through the utilization of Language Model probing, with the provided input subject-entity and relation. The LM-KBC challenge dataset comprises a diverse set of 21 relations, each covering a diferent set of subject-entities, and a complete list of ground truth object-entities per subject-relation-pair.

Our solution primarily relies on the concept of prompt ensemble1. Figure 1 depicts the architecture of our proposed solution, highlighting the various components. To craft the prompt ensemble, we initiate the process by employing the initial baseline prompts as inputs to the ChatGPT framework. Through this procedure, ten analogous prompts are generated, all of which pose the identical query. Subsequently, a step of prompt elicitation is executed, wherein supplementary contextual information is injected into the chosen prompts, furnishing them with additional insights about the anticipated nature of the predicted data.

Preceding the inference phase, a process of prompt refinement is undertaken to exclude suboptimal prompts, drawing from insights garnered from the train/validation dataset. The selected prompts of merit are then augmented with illustrative instances drawn from the training data, thereby facilitating the enhancement of few-shot learning capabilities. Furthermore, a validation process known as fact probing ensues, which assures the accuracy of the selected prompts. In this phase, a transformation of the relation is undertaken to assume the form of a Boolean question. For instance, for a relation like "BandHasMember," this transformation would manifest as the query, "Is member a constituent of band?".

The final stage involves the inference of object entities, using the Llama2 70B model, which engages its adeptness in disentangling complex data relationships. In the remainder of this paper, we will introduce these components in more detail, before discussing the obtained results.

1Our implementation is available on https://github.com/asdfthefourth/lmkbc.

 2. Prompt Ensemble

The main contribution of our work is using prompt ensembles for LM probing [ 3 ]. Instead of using a single prompt, a linguistic diverse set of prompts is used, i.e., an ensemble. We define an individual threshold per relation to reach consensus between the prompts. Additionally, the worst prompts based on the performance on the training data are removed. We take the number of possible answers from the dataset description and only use the number as the upper bound of possible answers for the model. We finetune the threshold on the validation dataset and use the resulting values for the test dataset. For choosing the optimal subset for the ensemble we calculated all subsets between 4 to all prompts on the training dataset and evaluated those on the validation dataset. The final prompt consists of the general context, the relation specific context, the few-shots and the prompt itself. The prompt ensemble approach is divided into five steps that are further explained in this section: prompt generation (Section 2.1), prompt elicitation (Section 2.2), few-shot example (Section 2.3), prompt selection (Section 2.4) and entity disambiguation (Section 2.5).

2.1. Prompt Generation

For generating additional prompts, we use the base prompt and ChatGPT to generate an additional prompts. Also, additional prompts were checked by humans for consistency and errors. The idea behind using LMs for prompt generation is to exploit model understanding to get more similar questions.

 2.2. Prompt Elicitation

In the prompt elicitation step, we add additional information to some of the relations [ 4 ]. This information is very specific for each relation.

For example, regarding the relation PersonPlayInstrument, we add the musician in front of the subject-entity. For the relation BandHasMember, we add the band in front of the subjectentity. One example for a prompt for the relation BandHasMember is "Who are the members of subject_entity?". After elicitation, the prompt looks like this: "Who are the members of the band subject_entity?". An example for the relation PersonPlaysInstrument is "What instrument does subject_entity play?", which looks like this after elicitation: "What instrument does the musician subject_entity play?".

Table 6, Table 7 and Table 8 in Appendix A show the worst and the best performing prompts resulting from our approach.

 2.3. Few-Shot Example

We choose the few-shot examples randomly for each relation by using the training data and the base prompt provided in the dataset repo [ 5 ]. We test which amount of few-shots produces the best F1-score. Table 1 shows the results for a given few-shot number .

Table 2 shows the similar results for only using the best prompt ensemble. The results show that the comparison of a higher amount of examples provides a better result. Yet, diminishing returns happen after = 10 and the computation efort increases too much for using = 20 or higher. Using only the best prompt, the F1-score reduces even when using ensemble. the few-shot examples are handpicked to include the maximum size of answers, the minimum size of answers, the empty set if it is allowed and a diverse selection of answers if only a few answer classes are available.

 2.4. Prompt Selection

Prompt selection is done by trying out a subset of ensembles on the training or the evaluation datasets and varying thresholds for finding a consensus. The best subset of prompts is chosen based on the F1-score.

We only search for thresholds in relations that allow more than one object and the empty set, because the consensus algorithm produces the same results regardless of the threshold. This is only the case for the two relations with numbers as solutions: PersonHasNumberOfChildren and SeriesHasNumberOfEpisodes.

 2.5. Entitiy Disambiguation

We use the baseline entity disambiguation provided with the dataset and modify the algorithm to return an empty string if no entity was found. We prompt the LM in the few shot example to return the objects in the following format: [Object1, Object2, ...]. Then we extract the objects and query the Wikidata2 API for IDs as it was done in the baseline.

2Wikidata: http://wikidata.org

 3. Fact Probing

Fact probing is used to check whether the result of the previous step is correct [ 6 ]. For each relation, the LM is asked whether it is true or not. Therefore, we invert the relationship of each relation and create a prompt as a boolean question. For example, for every band in a relation {band} BandHasMember {member}, we create a prompt "is {member} part of {band}". In this step, we also do few shot examples to improve the LMs response. For example, regarding the relations related to death, we first ask whether the subject is still alive and then confirm either location or cause.

4. Experimental Setup

The development of our approach was mostly done on the Llama2 model with 13 billion parameters to improve the turnaround time.

The following steps are taken to achieve the results. We run the validation dataset on the Llama2 model with 70 billion parameters with 10 few-shot examples. Then, we do fact checking on the results. Next, we generate the ensembles and the tresholds for each relation based on the validation dataset. We run test dataset on Llama2 and GPT-4 while using the generated ensembles from the previous step. Lastly, we combine the final dataset.

The last step contains the combination of the final dataset, we combine the solutions that were generated by gpt4 and llama respectivly and combine them to a single dataset.

4.1. Dataset Description

The dataset we use for the evaluation contains 21 m-to-n relations. Four of these relations contain the empty set as the solution. The dataset contains 1940 subjects for predictions and is divided into a training, validation and test dataset. The dataset contains relations in diferent domains, e.g., chemistry, geography and celebrities. The relations are in the triplemanner subject-predicate-object. An example for a relation in the chemistry dataset is: subject: "potassium, hydrogen, oxygen", relation: "CompoundHasParts", object: "Potassium Hydroxide". An example for an empty relation in the celebrity dataset is: subject: "Kobe Bryant", relation: "PersonHasNoblePrize", object": "" (empty set).

We use the training dataset for the few-shot training and the validation dataset for calculating the optimal ensemble.

 4.2. Model Selection

We use the Llama2 model in several editions: with 7 billion parameters, with 13 billion parameters and with 70 billion parameters. We run the Llama2 model with 70 billion parameters on two A100 GPUs with 40GB RAM each for inference. We furthermore use a non-fine-tuned version and a fine-tuned version for the chat completion. Table 3 shows the results of our comparison of the Llama2 model with diferent parameter sizes, each using the best prompt and the best ensemble. Our comparison shows that the model version with 70 billion parameters using the best ensemble achieves the best results.

 5. Results 6. Related Work

Alivanistos et al. [ 6 ] focus on prompting as probing which is a multi-step approach that combines a variety of prompting techniques to construct knowledge bases from LMs. We apply a similar approach in the step of fact probing. Yet, we further improve their approach be using prompt ensembles in a previous step.

Li et al. [ 4 ] focus on the task-specific to improve relation prediction using LMs. Therefore, they create a sentence from every subject-relation-object triple. They mask the tokens in the sentence that are relevant to the object entity and train the LM with the objective to predict the tokens. They furthermore apply prompt elicitation, similar to our approach.

Ning and Celebi [ 3 ] also apply a prompt ensemble step, similar to our approach. Yet, we furthermore improve this by applying additional steps like prompt elicitation and fact probing.

In their pre-print “Boosted Prompt Ensembles for Large Language Models”, Pitis et al. [ 7 ] use few shot prompts to create prompt ensembles. They adapt classical boosting algorithms to improve prompts in an iterative process. Their solution could be a promising future improvement for our work.

Jiang et al. [ 8 ] propose a mining- and paraphrasing-based method to estimate the knowledge contained in LMs more accurately. Similar to our approach, they use ensemble methods to combine the results of diferent prompts. They describe diferent methods to create the ensembles, which could be used to further improve our work. Yet, they do not involve ChatGPT for the prompts generation.

7. Conclusion and Outlook

Our evaluation shows that using prompt ensembles improves the overall performance of knowledge base construction using LMs. Our comparison described in Section 4.2 shows that using the non-fine-tuned Llama2 model with the most parameters achieves the best results. This leads to a trade-of between the best results and processing time.

In the future, we aim to evaluate our approach on more powerful hardware so we do not have to apply quantization to the model parameters. This should lead to an ever higher F1-score. We furthermore aim to test our approach of diferent LMs like GPT-4 and introduce the approaches of Ning and Celeb [ 7 ] and Jiang et al. [ 8 ] to our solution, as described in Section 6.

Acknowledgments

This research was funded by German Federal Ministry of Education and Research (BMBF) through grants 01IS17051 (Software Campus program), 02L19C155, 01IS21021A (ITEA project number 20219).

 A. Worst and Best Performing Prompts Provide a list of states that belong to subject_entity. What are the constituent states of subject_entity? Can you tell me in which position subject_entity participates in football? Tell me about the role subject_entity plays in football. What caused the death of subject_entity? Share details about what led to subject_entity’s passing. Mention the title of the book authored by subject_entity. I’m interested in knowing the title of subject_entity’s autobiography. Who is subject_entity’s employer? Can you tell me the name of subject_entity’s employer? In which field did subject_entity receive the Nobel Prize? What discipline was recognized when subject_entity was awarded the Nobel Prize? How many children does subject_entity have? What is the size of subject_entity’s family in terms of children? F1-Score 0.0

0.0 0.655 0.5566 0.7783 0.5366 0.4266 0.3166 0.3027 0.2142 0.9766 0.5966 0.52 0.39

 PersonHasPlaceOfDeath PersonHasPlaceOfDeath PersonHasProfession PersonHasProfession PersonHasSpouse PersonHasSpouse PersonPlaysInstrument PersonPlaysInstrument PersonSpeaksLanguage PersonSpeaksLanguage RiverBasinsCountry RiverBasinsCountry SeriesHasNumberOfEpisodes SeriesHasNumberOfEpisodes StateBordersState StateBordersState Prompt Where did subject_entity die? Identify the location of subject_entity’s passing. What does subject_entity do as their profession? State subject_entity’s area of expertise or job title. Who is the life partner or spouse of subject_entity? Provide information about the person to whom subject_entity is married. What is subject_entity’s chosen musical instrument? Tell me about the musical equipment with which subject_entity is proficient. What languages does subject_entity speak? What are the diferent languages known by subject_entity? State the country where the subject_entity river basin is situated. I’d like to know the geographical location of the subject_entity river basin. How many episodes have been produced for subject_entity series? Tell me about the total episode tally of subject_entity series. Tell me the names of the states that share borders with subject_entity. What states are connected to subject_entity’s state by borders?

F1-Score

 [1] F. Petroni , T. Rocktäschel , P. Lewis , A. Bakhtin , Y. Wu , A. H. Miller , S. Riedel , Language models as knowledge bases? , arXiv preprint arXiv: 1909 . 01066 ( 2019 ). [2] S. Singhania , J.-C. Kalo , S. Razniewski , J. Z. Pan , Lm-kbc: Knowledge base construction from pre-trained language models, semantic web challenge @ iswc, CEUR-WS ( 2023 ). URL: https://lm-kbc.github.io/challenge2023/. [3] X. Ning , R. Celebi , Knowledge base construction from pre-trained language models by prompt learning ( 2022 ). [4] T. Li , W. Huang , N. Papasarantopoulos , P. Vougiouklis , J. Z. Pan , Task-specific pre-training and prompt decomposition for knowledge graph population with language models , arXiv preprint arXiv:2208.12539 ( 2022 ). [5] Y. Wang , Q. Yao , J. T. Kwok , L. M. Ni , Generalizing from a few examples: A survey on few-shot learning , ACM Comput. Surv . 53 ( 2020 ). [6] D. Alivanistos , S. B. Santamaría , M. Cochez , J.-C. Kalo , E. van Krieken, T. Thanapalasingam , Prompting as probing: Using language models for knowledge base construction , arXiv preprint arXiv:2208.11057 ( 2022 ). [7] S. Pitis , M. R. Zhang , A. Wang , J. Ba , Boosted prompt ensembles for large language models , 2023 . arXiv: 2304 . 05970 . [8] Z. Jiang , F. F. Xu , J. Araki , G. Neubig, How Can We Know What Language Models Know?, Transactions of the Association for Computational Linguistics 8 ( 2020 ) 423 - 438 .