Language Models (LMs) have proven to be useful in various downstream applications, such as summarisation, translation, question answering and text classification. LMs are becoming increasingly important tools in Artificial Intelligence, because of the vast quantity of information they can store. In this work, we present ProP (Prompting as Probing), which utilizes GPT-3, a large Language Model originally proposed by OpenAI in 2020, to perform the task of Knowledge Base Construction (KBC). ProP implements a multi-step approach that combines a variety of prompting techniques to achieve this. Our results show that manual prompt curation is essential, that the LM must be encouraged to give answer sets of variable lengths, in particular including empty answer sets, that true/false questions are a useful device to increase precision on suggestions generated by the LM, that the size of the LM is a crucial factor, and that a dictionary of entity aliases improves the LM score. Our evaluation study indicates that these proposed techniques can substantially enhance the quality of the final predictions: ProP won track 2 of the LM-KBC competition, outperforming the baseline by 36.4 percentage points. Our implementation is available on https://github.com/HEmile/iswc-challenge.
Language Models (LMs) have been at the center of attention, presented as a recent success story
of Artificial Intelligence. LMs have shown great promise across a wide range of domains in a
variety of diferent tasks, such as Text Classification [
Natural Language Processing (NLP) researchers have recently investigated whether LMs
could potentially be used as Knowledge Bases, by querying for particular information. In Petroni
et al. [
In this paper, we describe ProP, the system we implemented for the “Knowledge Base
Construction from Pre-trained Language Models” challenge at ISWC 20221. The task is to predict
possible objects of a triple, where the subject and relation are given. For example, given the
string "Ronnie James Dio" as the subject and the relation PersonInstrument, an LM needs to
predict the answers "bass guitar" or "guitar", and "trumpet", as the objects of the triple. In
contrast to the LAMA dataset [
ProP uses the GPT-3 model, a large LM proposed by OpenAI in 2020[
The development of large pre-trained Language Models (LMs) has led to substantial
improvements in NLP research. It was shown that extensive pre-training on large text corpora encodes
large amounts of linguistic and factual knowledge into a language model that can help to
improve the performance on various downstream tasks (see [
LM as KG Petroni et al. [
While the original paper relied on manually designed prompts for probing the language
model, various follow-up works have shown the superiority of automatically learning prompts.
Methods can mine prompts from large text corpora and pick the best prompts by applying
them to a training dataset as demonstrated in [
Since the publication of the LAMA dataset, a large variety of other probing datasets for factual
knowledge in LMs have been created. LAMA-UHN is a more dificult version of LAMA [
Most existing approaches have in common that they only probe the language models for entities with a label consisting of a single token from the language model’s vocabulary. Thus, the prediction of long, complex entity names is mostly unexplored. Furthermore, most existing works have only asked language models to complete a triple with a single prediction, even though some triples might actually allow for multiple possible predictions. Both these aspects substantially change the challenge of knowledge graph construction.
LM for KG While probing language models has been heavily studied in the NLP community,
the idea of using language models to support knowledge graph curation is not suficiently
studied [
KG-BERT is a system that is most similar to what is required for the KBC Challenge [
Similar tasks This KBC Challenge task is similar to Factual Q&A with Language models, where the goal is to respond to questions that fall outside the scope of a knowledge base. This shared task difers in that the responses need to include 0 to k answers. Moreover, in the shared task, the factual questions are generated from triples, thus including variation in how a triple might be phrased as a question.
The LM-KBC dataset contains triples of the form (, , ), where is the textual representation of the subject, is one of 12 diferent predicates and the (possibly empty) set of textual representations of object entities for prediction. The subjects and objects are of various types. After learning from a training set of such triples, given a new subject and one of the known relations, the task is to predict the complete set of objects.
For each of the 12 relations, the number of unique subject-entities in the train, dev, and test sets are 100, 50, and 50 respectively. We include detailed distributions of the cardinality (the number of object-entities) for each relation type (Appendix, Figures 2 and 3). Table 4 in the Appendix shows the aggregated statistics about the number of object-entities, as well as the number of alternative labels per object-entities in the development set. Certain relation types have a much higher average cardinality (e.g. PersonProfession=7.42 or StateSharesBorderState=5.6) than others (e.g. CompanyParentOrganization=0.32, PersonPlaceOfDeath=0.50). We also note that only five of the relations allow for empty answer sets. For example, relations associated with a person’s death (PersonPlaceOfDeath and PersonCauseOfDeath) often contain empty answer sets, because many persons in the dataset are still alive. In these cases, a models needs to be able to predict empty answer sets correctly.
The baseline model is a masking-based approach that uses the popular BERT model [
Previous studies have shown that prompt formulation plays a critical role in the performance
of LMs on downstream applications [
For each relation in the dataset, we manually curate a set of prompt templates consisting of four representative examples3 selected from the training split. We use these relation-specific templates to generate prompts for every subject entity by replacing a placeholder token in the ifnal line of the template with the subject entity of interest. We task GPT-3 to complete the prompts, and evaluate the completions and compute the macro-precision, macro-accuracy and macro-F1 score for each relation.
2https://openai.com/api (temperature=0, max_tokens=100, top_p=1, frequency_penalty=0, presence_penalty=0, logprobs=1)
3This is an arbitrary number of training examples. Since few-shot learners are eficient at learning from a
handful of examples [
We ensured that the training examples for few-shot learning included all of the following: (i) questions with answer sets of variable lengths to inform the LM that it can generate multiple answers; (ii) questions with empty answer sets to ensure that the LM returns nothing when there is no correct answer to a given question; (iii) a fixed question-answer order, where we provide the question and then immediately the answer so that the LM learns the style of the format, and (iv) the answer set formatted as a list to ensure we can eficiently parse the answer set from the LM. We did not study the order of these examples, but hypothesize that this is not hugely important to GPT-3 as it can handle long-range dependencies well.
We formulate the questions either in natural language or in the form of a triple. We handdesigned the natural language questions and did not compare them in a structured manner to alternatives. However, we tried out several variations in OpenAI GPT-3 playground to get an intuition of what style of questions are efective. We found those are usually shorter and simpler. We include the prompt templates used in Section 7.2 of the Appendix. In our work, we investigate the use of both prompting styles and compare natural language prompts with triple-based prompts for the diferent relations.
There are some questions in the dataset for which the correct answer is an empty answer set. For instance, there are no countries that share borders with Fiji. The empty answer set can be represented as either an empty list [], or as a string within a list [‘None’]. We have observed (see Table 2) that the way that empty sets are represented afects the precision and recall of our approach. Allowing the explicit answer ‘None’ encourages the LM to exclude answers that it is uncertain about.
Our initial results indicated that the recall of our approach was high, but the precision for
certain relations was low. Therefore, we add a post-processing step called fact probing in ProP’s
pipeline. We use fact probing to ask the LM to probe whether each completion proposed by
the LM in the previous step is correct. Inspired by maieutic prompting [
In this Section, we analyse the ProP pipeline we built for generating prompts and evaluate the contribution of each component. We explain how we combine the best-performing components to yield a prediction that obtains a high macro F1-score on the test split.
Table 1 shows the quality of the predictions for the natural language prompts and triple-based prompts. We note that on F1, the performance between these two prompt styles is mixed, with F1 being higher on the triple-style prompts in only seven out of twelve cases. Unpacking the F1 into recall and precision shows that the triple-style prompts yield higher precision, while natural language prompts yield higher recall. Overall, the triple-style prompts do yield a higher F1 when averaged over each relation. Our intuition is that natural language prompts contain certain words that badly influence the precision of the predictions. It is, however, dificult to study this systematically as the enumeration of word combinations in the prompts is very large. Triple-based prompts circumvent this problem because they only contain the relevant terms in the prompts for the subject entities and relations that are required to predict the object entities.
As explained in Section 3.1, five out of twelve relations allow for empty answer sets. We experiment with the diferent ways to represent such empty answer sets, and Table 2 shows the results. Three relations get a performance boost when prompted with ‘NONE’ (CompanyParentOrganization, PersonCauseOfDeath, PersonInstrument), while the other two relations perform better when using empty lists (CountryBordersWithCountry, PersonPlaceOfDeath). For subsequent experiments, we modified the prompt of each relation to use the best performant representation.
Usually, the size of LMs is measured by the number of learnable parameters in the model. However, the OpenAI API does not quantify the number of total parameters but only the size of the embedding dimensions for the tokens 4. We assume there is a positive correlation between the token dimension size and the total number of GPT-3 parameters. Figure 1 shows our results, and we can see that as the language model size increases, the F1-score also increases. This shows that a larger LM gives better performance on KBC.
Up to this point, we have discussed how we generate the optimal prompts for the diferent relation types. Once the LM produces the completions using these optimal prompt techniques, we can employ two additional steps to enhance the precision and recall of our predictions. Table 3 shows the results of including fact probing and entity aliases in our system.
We found that fact probing has a diferent impact on diferent relation types. This diference could stem from the cardinalities of the relation types. For example, the relation PersonPlaceOfDeath, which has only one correct answer, should show a larger improvement than State Borders, which has a higher cardinality. We found that fact probing helped to boost the predictions of five relations ( CompanyParentOrganization, CountryOficialLanguage, PersonCauseOfDeath, PersonInstrument, PersonLanguage). We only apply fact probing to these relations. On the dev set, the precision of fact probing is 0.737, and 0.608 among the predictions removed by fact probing. That is, in 60.8% fact probing filtered a prediction, it correctly removed a prediction that was not in the ground truth set.
As we discussed in Section 3.1, the predictions from the language model are sometimes correct, but not according to what the gold standard expects. Whether this is problematic depends on the final use of the predictive model. In interactive use, this would not be an issue because the user will be able to disambiguate. For actual KBC, the system will have to disambiguate what exact entity it predicted. Here, however, we only check whether the text generated by the model corresponds to one of the gold standard alternatives.
While experimenting, we noticed that in the training and development datasets the names of entities often correspond with the labels of entities in Wikidata5. On Wikidata, these entities also have aliases, and we wanted to know whether we can improve our system by looking up the aliases on Wikidata. This lookup does use language models, so it is not included as part of the ProP pipeline, as this would violate the terms of the LM-KBC challenge. Instead, we perform it as an ablation study.
The alias-fetcher works as follows. First, we extract a set of types which could be relevant for the specific relation types. For example, country (Q6256) is relevant for RiverBasinsCountry. Then, for each relation type, we extract all correctly typed entities, their aliases, and claim count. Then, we take the prediction of the LM, and check whether there is an entity with that label for that relation. If so, we retain the prediction.
Otherwise, we check whether the prediction is equal to any alias. There could be multiple entities for which this is the case. Therefore, we pick the label of the entity with the most claims on Wikidata. The assumption is that it is more likely the answer if it is an ‘interesting’ entity and that these have more claims on Wikidata.
We observe that for four relation types, the changes in the scores are insignificant. For the eight other relations, we see that the F1 score goes up slightly. Overall, this results in an average improvement of the F1 score with 0.014 (Table 3) on the development set. On the test set, we notice a similar improvement. This experiment is not extensive enough to derive definitive conclusions, but it appears to be useful to use structured data to augment the predictions of an LM.
We found that questions regarding recent events, particularly those that occurred after 2020, did not yield good predictions by GPT-3 (see 7.3 in the Appendix). This is in line with related ifndings around LMs and was confirmed by OpenAI 6. Two examples of this include Facebook, Inc. changing its name to Meta Platforms, Inc. (in October 2021), and the country of Swaziland changing its name to Eswatini (in 2018). We also observed similar problems with several instances from the following relations: PersonProfession, PersonCauseOfDeath, and PersonPlaceOfDeath. It is worth noting that it is not important when the model was trained, but whether the training date contains up-to-date information.
A natural continuation of this work revolves around improving the individual steps in our
pipeline (e.g. fact checking) and their performance, which will directly reflect on the overall
macro-F1 score of our approach. Additionally, we could experiment with inverting our pipeline
and allow the LM to generate the best prompts by providing the ground truth as input. For
example, we could explore techniques that automatically learn prompts, similar to AutoPrompt [
In terms of additional components that make use of LMs, we considered developing
metaprompts as in [
Data augmentation difers from prompt tuning in the following ways: While prompt tuning is looking for the optimal prompt to increase the performance for a specific task, data augmentation acts as a diversification mechanism for our existing prompts. By employing a more diverse set of prompts we can increase our performance, especially the recall. We base our hypothesis on the fact that knowledge is expressed diversely in the training data (e.g. ambiguity), and we believe this should be considered when prompting an LM.
Furthermore, it would be interesting to further investigate if huge language models are required to perform knowledge graph construction and how to achieve the best prediction performance for the lowest costs.
We introduced ProP, our "Prompting as Probing" approach to performing knowledge-base completion using a high-capacity pre-trained LM. We showed how we developed diferent modular components that utilise both LM and the data provided by the organisers to improve ProP’s performance, such as the fact probing and the alias fetcher components. We also investigated well-known techniques around prompt engineering and optimisation and analysed the efect of diferent prompt formulations on the final performance. However, we conclude that the parameter count of the GPT-3 models is the most significant contributor to performance. Our ProP pipeline outperforms the baseline by 36.4 percentage points on the test split.
Our approach does not only obtain a high macro F1-score on the ground truth but its actual score is suspected to be higher because in several cases where the result was counted as incorrect, the ground truth was either incomplete or used aliases that refer to the same entity as our prediction. Overall, we conclude that language models can be used to augment Knowledge Bases, and we emphasise the dificulty of evaluating question-answering tasks where simple string matching does not sufice.
Supplemental Material Statement: Code and data are publicly available from https://github. com/HEmile/iswc-challenge.
We thank Frank van Harmelen for his insightful comments. This research was funded by the Vrije Universiteit Amsterdam and the Netherlands Organisation for Scientific Research (NWO) via the Spinoza grant (SPI 63-260) awarded to Piek Vossen, the Hybrid Intelligence Centre via the Zwaartekracht grant (024.004.022), Elsevier’s Discovery Lab, and Huawei’s DReaMS Lab.
Here, we provide statistics about the LM-KBC dataset for the training and development split. The statistics of the test split is unknown, because the test split is not public. We assume that the instances from the test are also sampled from a similar data distribution.
The LM-KBC challenge does not include entity linking. Instead, predicted entities are scored against a list of their aliases in the LM-KBC dataset. However, we noticed these lists are often incomplete. For example, for the "National Aeronautics and Space Administration", the extremely common and widely used abbreviation "NASA", is not included in the list of aliases. Another example occurs when the model predicts Aluminum (US and Canadian English) where the ground truth only has Aluminium (British English; a term globally adopted), a lower score gets obtained. Hence, if the model predicts Aluminum or NASA, the predictions are deemed incorrect.
Here, we show the templates we used to generate the prompts for the diferent relations. In our template, we use {subject_entity} to refer to the head entity for which we are predicting the tail entities for. The generated prompts were used for the following models: Ada, Babbage, Curie and Davinci.
Which countries neighbour Dominica? [’Venezuela’] Which countries neighbour North Korea? [’South Korea’, ’China’, ’Russia’] Which countries neighbour Serbia? [’Montenegro’, ’Kosovo’, ’Bosnia and Herzegovina’, ’Hungary’, ’Croatia’, ’Bulgaria’, ’Macedonia’, ’Albania’, ’Romania’] Which countries neighbour Fiji? [] Which countries neighbour {subject_entity}?
Suriname CountryOfficialLanguage: [’Dutch’] Canada CountryOfficialLanguage: [’English’, ’French’] Singapore CountryOfficialLanguage: [’English’, ’Malay’, ’Mandarin’, ’Tamil’] Sri Lanka CountryOfficialLanguage: [’Sinhala’, ’Tamil’] {subject_entity} CountryOfficialLanguage:
San Marino StateSharesBorderState: [’San Leo’, ’Acquaviva’, ’Borgo Maggiore’, ’Chiesanuova’, ’Fiorentino’] Whales StateSharesBorderState: [’England’] Liguria StateSharesBorderState: [’Tuscany’, ’Auvergne-Rhoone-Alpes’, ’Piedmont’, ’Emilia-Romagna’] Mecklenberg-Western Pomerania StateSharesBorderState: [’Brandenburg’, ’Pomeranian’, ’Schleswig-Holstein’, ’Lower Saxony’] {subject_entity} StateSharesBorderState:
Drava RiverBasinsCountry: [’Hungary’, ’Italy’, ’Austria’, ’Slovenia’, ’Croatia’] Huai river RiverBasinsCountry: [’China’] Paraná river RiverBasinsCountry: [’Bolivia’, ’Paraguay’, Argentina’, ’Brazil’] Oise RiverBasinsCountry: [’Belgium’, ’France’] {subject_entity} RiverBasinsCountry:
Water ChemicalCompoundElement: [’Hydrogen’, ’Oxygen’] Bismuth subsalicylate ChemicalCompoundElement: [’Bismuth’] Sodium Bicarbonate ChemicalCompoundElement: [’Hydrogen’, ’Oxygen’, ’Sodium’, ’Carbon’] Aspirin ChemicalCompoundElement: [’Oxygen’, ’Carbon’, ’Hydrogen’] {subject_entity} ChemicalCompoundElement:
Aamir Khan PersonLanguage: [’Hindi’, ’English’, ’Urdu’] Pharrell Williams PersonLanguage: [’English’] Xabi Alonso PersonLanguage: [’German’, ’Basque’, ’Spanish’, ’English’] Shakira PersonLanguage: [’Catalan’, ’English’, ’Portuguese’, ’Spanish’, ’Italian’, ’French’] {subject_entity} PersonLanguage:
What is Danny DeVito’s profession? [’Comedian’, ’Film Director’, ’Voice Actor’, ’Actor’, ’Film Producer’, ’Film Actor’, ’Dub Actor’, ’Activist’, ’Television Actor’] What is David Guetta’s profession? [’DJ’] What is Gary Lineker’s profession? [’Commentator’, ’Association Football Player’, ’Journalist’, ’Broadcaster’] What is Gwyneth Paltrow’s profession? [’Film Actor’,’Musician’] What is {subject_entity}’s profession?
Liam Gallagher PersonInstrument: [’Maraca’, ’Guitar’] Jay Park PersonInstrument: [’None’] Axl Rose PersonInstrument: [’Guitar’, ’Piano’, ’Pander’, ’Bass’] Neil Young PersonInstrument: [’Guitar’] {subject_entity} PersonInstrument:
Where is or was Susan Wojcicki employed? [’Google’] Where is or was Steve Wozniak employed? [’Apple Inc’, ’Hewlett-Packard’, ’University of Technology Sydney’, ’Atari’] Where is or was Yukio Hatoyama employed? [’Senshu University’,’Tokyo Institute of Technology’] Where is or was Yahtzee Croshaw employed? [’PC Gamer’, ’Hyper’, ’Escapist’] Where is or was {subject_entity} employed?
What is the place of death of Barack Obama? [] What is the place of death of Ennio Morricone? [’Rome’] What is the place of death of Elon Musk? [] What is the place of death of Prince? [’Chanhassen’] What is the place of death of {subject_entity}?
André Leon Talley PersonCauseOfDeath: [’Infarction’] Angela Merkel PersonCauseOfDeath: [’None’] Bob Saget PersonCauseOfDeath: [’Injury’, ’Blunt Trauma’] Jamal Khashoggi PersonCauseOfDeath: [’Murder’] {subject_entity} PersonCauseOfDeath:
Microsoft CompanyParentOrganization: [’None’] Sony CompanyParentOrganization: [’Sony Group’] Saab CompanyParentOrganization: [’Saab Group’, ’Saab-Scania’, ’Spyker N.V.’, ’National Electric Vehicle Sweden’’, ’General Motors’] Max Motors CompanyParentOrganization: [’None] {subject_entity} CompanyParentOrganization:
Here, we list three failure examples for each relation for the Davinci model. A comprehensive list of failure cases can be found under https://github.com/HEmile/iswc-challenge/tree/main/failure_cases.
SubjectEntity: Bahrain Ground Truth: [’iran’, ’saudi arabia’] GPT-3 Prediction: [’qatar’, ’saudi arabia’, ’united arab emirates’] SubjectEntity: Barbados Ground Truth: [] GPT-3 Prediction: [’trinidad and tobago’] SubjectEntity: Cuba Ground Truth: [’united states of america’, ’usa’] GPT-3 Prediction: [’bahamas’, ’haiti’, ’jamaica’, ’turks and caicos islands’, ’united states’]
SubjectEntity: Afghanistan Ground Truth: [’arabic’, ’baluchi’, ’dari’, ’nuristani’, ’pamir’, ’pashayi’, ’pashto’, ’turkmen’, ’uzbek’] GPT-3 Prediction: [’dari’, ’pashto’] SubjectEntity: Botswana Ground Truth: [’english’] GPT-3 Prediction: [’setswana’] SubjectEntity: Zimbabwe Ground Truth: [’barwe’, ’chewa’, ’english’, ’kalanga’, ’khoisan’, ’nambya’, ’ndau’, ’ndebele’, ’northern ndebele’, ’sesotho’, ’shona’, ’tonga’, ’tsonga’, ’tswana’, ’venda’, ’xhosa’] GPT-3 Prediction: [’chewa’, ’english’, ’ndebele’, ’shangaan’, ’shona’, ’sotho’, ’tonga’, ’venda’]
SubjectEntity: Andalusia Ground Truth: [’beja’, ’castile-la mancha’, ’extremadura’, ’faro’, ’gibraltar’, ’murcia’, ’region of murcia’] GPT-3 Prediction: [’castilla-la mancha’, ’ceuta’, ’extremadura’, ’melilla’, ’murcia’] SubjectEntity: Obwalden Ground Truth: [’canton of bern’, ’canton of lucerne’, ’lucerne’, ’nidwalden’, ’schwyz’, ’uri’] GPT-3 Prediction: [’berne’, ’uri’, ’valais’] SubjectEntity: Zagreb Ground Truth: [’krapina-zagorje’, ’krapina-zagorje county’, ’zagreb’, ’zagreb county’] GPT-3 Prediction: [’bosnia and herzegovina’, ’serbia’, ’slovenia’]
SubjectEntity: Didi Liakhvi river Ground Truth: [’georgia’] GPT-3 Prediction: [’georgia’, ’russia’] SubjectEntity: Dniester Ground Truth: [’moldova’, ’ukraine’] GPT-3 Prediction: [’moldova’, ’romania’, ’ukraine’] SubjectEntity: San Ground Truth: [’poland’, ’ukraine’] GPT-3 Prediction: [’mexico’]
SubjectEntity: Aluminum Iodide Ground Truth: [’aluminium’, ’iodine’] GPT-3 Prediction: [’aluminum’, ’iodine’] SubjectEntity: Atracurium Besylate Ground Truth: [’carbon’, ’hydrogen’, ’nitrogen’, ’oxygen’, ’sulfur’] GPT-3 Prediction: [’atracurium’, ’hydrogen’, ’oxygen’] SubjectEntity: Flavin Adenine Dinucleotide Ground Truth: [’carbon’, ’hydrogen’, ’nitrogen’, ’oxygen’, ’phosphorus’] GPT-3 Prediction: [’adenine’, ’dinucleotide’, ’flavin’]
SubjectEntity: A. R. Rahman Ground Truth: [’hindi’, ’tamil’, ’telugu’] GPT-3 Prediction: [’bengali’, ’hindi’, ’kannada’, ’malayalam’, ’marathi’, ’tamil’, ’telugu’] SubjectEntity: Akon Ground Truth: [’english’] GPT-3 Prediction: [’english’, ’french’, ’wolof’] SubjectEntity: Genelia D’Souza Ground Truth: [’hindi’, ’malayalam’, ’tamil’, ’telugu’] GPT-3 Prediction: [’bengali’, ’english’, ’hindi’, ’marathi’, ’telugu’]
SubjectEntity: Elon Musk Ground Truth: [’engineer’, ’entrepreneur’, ’inventor’, ’programmer’] GPT-3 Prediction: [’businessperson’, ’engineer’, ’inventor’, ’investor’, ’philanthropist’, ’product designer’] SubjectEntity: J. K. Rowling Ground Truth: [’author’, "children’s writer", ’executive producer’, ’film producer’, ’novelist’, ’producer’, ’producer’, ’screenwriter’, ’writer’, ’writer’] GPT-3 Prediction: [’author’, ’film producer’, ’philanthropist’, ’screenwriter’] SubjectEntity: Lady Gaga Ground Truth: [’singer’, ’songwriter’] GPT-3 Prediction: [’actress’, ’dancer’, ’philanthropist’, ’record producer’, ’singer’, ’songwriter’]
SubjectEntity: Ronan Keating Ground Truth: [] GPT-3 Prediction: [’piano’] SubjectEntity: Taeyang Ground Truth: [] GPT-3 Prediction: [’guitar’] SubjectEntity: Chester Bennington Ground Truth: [] GPT-3 Prediction: [’guitar’, ’piano’]
SubjectEntity: Kent Beck Ground Truth: [’meta platforms’] GPT-3 Prediction: [’facebook’, ’three rivers institute’] SubjectEntity: Serena Williams Ground Truth: [’unicef’] GPT-3 Prediction: [’tennis’] SubjectEntity: Guido van Rossum Ground Truth: [’microsoft’] GPT-3 Prediction: [’dropbox’, ’google’]
SubjectEntity: Avicii Ground Truth: [’muscat’] GPT-3 Prediction: [’muscat, oman’] SubjectEntity: John Coltrane Ground Truth: [’huntington’] GPT-3 Prediction: [’new york city’] SubjectEntity: Rachel Caine Ground Truth: [’texas’] GPT-3 Prediction: [’’]
SubjectEntity: Ahmed Zewail Ground Truth: [’lymphoma’, ’spinal cord lymphoma’] GPT-3 Prediction: [’cancer’] SubjectEntity: Avicii Ground Truth: [’exsanguination’] GPT-3 Prediction: [’suicide’] SubjectEntity: Ennio Morricone Ground Truth: [’femoral fracture’, ’fracture’] GPT-3 Prediction: [’’]
SubjectEntity: Aston Martin lagonda Ground Truth: [] GPT-3 Prediction: [’aston martin lagonda global holdings plc’] SubjectEntity: Austro-Daimler Ground Truth: [] GPT-3 Prediction: [’daimler ag’] SubjectEntity: Hyundai Motor Company Ground Truth: [’hyundai’] GPT-3 Prediction: [’hyundai motor group’]
Table 5 shows the values of the scaling experiments. These values were used to produce Figure 1. Figure 2: The number of answers per relation type for the training set provided by the organisers. Figure 3: The number of answers per relation type for the development set provided by the organisers.