Curated knowledge bases (KBs) contain billions of facts with millions of entities and thousands of predicates. Question answering (QA) systems are supposed to access this knowledge to answer users' factoid questions. Entity linking and relation linking are integral ingredients of many such QA systems, and aim to link mentions in the question to concepts in the KB. The quality of these linking modules is of high importance: a single error in linking can result in a failure for the whole QA system. The SMART 2022 Task poses challenges for entity and relation linking to evaluate the performance of diferent approaches. In this work, we adapt and extend our prior work CLOCQ. CLOCQ computes top- linkings for each mention to make up for potential errors, with set automatically based on an ambiguity score. As an extension, we design a module that prunes linkings for irrelevant mentions which helps to improve precision. We found that there is a trade-of between recall and precision: higher boosts recall (up to 0.87 for entity linking), while lower leads to high precision performance. The best choice for the linking modules may highly depend on the specific QA system, and whether it can make use of higher recall in the presence of noise.
Question answering (QA) systems provide natural interfaces for accessing human knowledge.
Such human knowledge can be stored in large-scale knowledge bases (KBs) like Wikidata [
Consider the following running example on the TV series House of the Dragon following the narratives of George R.R. Martin:
“Who plays Viserys in GRRM’s latest HBO series?”
Linking the mentions in the question to the KB (Wikidata for this example) is a non-trivial task that requires an understanding of the question as a whole. The entity mention “HBO” may refer to the H B O c o m p a n y , the H B O n e t w o r k , or to the H o l l y w o o d B o w l O r c h e s t r a . Understanding that the question is on a TV series helps to identify H B O n e t w o r k as the correct entity. Similarly, “plays” semantically or lexically matches with many relations in the KB, like p l a y s f o r t e a m , i n s t r u m e n t , n u m b e r o f p l a y s , c h a r a c t e r r o l e , or t i m e p l a y e d . The intended relation c h a r a c t e r r o l e only becomes clear from the question context.
“Viserys” is even harder to link, since there are diferent characters named Viserys in the Game of Thrones universe: V i s e r y s I I I T a r g a r y e n , the more well-known character from Game of Thrones, and V i s e r y s I T a r g a r y e n from the more recent House of the Dragon series. Thus, the mention “Viserys” is quite ambiguous, even if the general context of the question is clear. A deep understanding of the question is required to correctly link “Viserys” to V i s e r y s I T a r g a r y e n . Note that in case any of these disambiguations are incorrect, there is little hope to return the correct answer P a d d y C o n s i d i n e to the user.
To further investigate QA modules and pinpoint failure cases, the SMART Task 3.01
(colocated with ISWC 2022) poses tasks for entity linking (Task 3) and relation linking (Task 2).
There is also a task on answer type prediction (Task 1), which is not targeted in this work.
Approach and contribution. In this work we adapt our recently proposed CLOCQ
framework [
1 2 3 … 15 : “plays”
KB item play plays for team instrument
… character role coh 0.93 1 0.84 2 0.81 4 0.61 5
KB item character role
play instrument plays for team rel KB item 0.76 1 play 0.71 2 plays for team 0.65 3 instrument 0.55 15 character role
: “Viserys” KB item 1 Viserys III Targaryen 2 Viserys I Targaryen 3 Viserys II Targaryen 4 Vinery Stud Stakes … … …
: “HBO” KB item 1 HBO company 2 HBO network 3 Hollywood Bowl Orchestra 4 HBO Max (streaming) … …
… One obstacle with adapting CLOCQ on entity or relation linking tasks, is that it, by design, disambiguates all mentions in the question. It does not diferentiate between entities, relations, types or other concepts. This helps when retrieving a search space, but can hurt precision of linking results. For example, CLOCQ might link “latest” and “series” to the KB, even if these mentions are irrelevant. We therefore propose a simple pruning module, that identifies which mentions should be linked, and prunes linkings for other mentions. The module is implemented with a fine-tuned sequence generation model that is trained using distant supervision.
By evaluating CLOCQ on the entity and relation linking tasks of SMART 3.0 challenge, we essentially investigate its applicability to QA approaches generating an explicit query. We show that top- disambiguations can help boosting recall, at the cost of decreasing scores for precision and F1 score. Further, we find that the mention-pruning module helps to improve the precision and F1 score substantially on the entity linking task.
Consider our running example “Who plays Viserys in GRRM’s latest HBO series?”. Our goal is to link mentions in the question (“plays”, “Viserys”, “GRRM”, “HBO”, “series”) to items in the KB. Mentions in the question can be single question words or phrases. Named entity phrases can for example be detected using named entity recognition (NER).
We first collect candidates from the KB using a standard lexical matching score (like TFIDF or BM25) for each mention 1 … .
would be 5 in our example, and stopwords are
dropped. Here is analogous to a search query, while each item in the KB resembles
a document in a corpus. This “document” is created by concatenating the item label with
textual aliases and descriptions available in most KBs [
= ⟨ 1 , 2 , …⟩} of KB items , one list for each , scored by degree of match between the mentions and KB items.
A ranked lexical match list for “plays” could look like: 1 = ⟨1 : p l a y , 2 : p l a y s f o r t e a m , 3 : i n s t r u m e n t , 4 : n u m b e r o f p l a y s , 5 : t i m e p l a y e d , 6 : p l a y w r i g h t , 7 : g u i t a r i s t , 8 : P l a y s c o l l e c t i o n , …, 1 5 : c h a r a c t e r r o l e , …⟩ with the ideal disambiguation being shown in bold. The list for “HBO” could be: 4 = ⟨1 : H B O c o m p a n y , 2 : H B O n e t w o r k , 3 : H o l l y w o o d B o w l O r c h e s t r a , …⟩ Note that the correct KB item for can sometimes be very deep in individual lists . For example, c h a r a c t e r r o l e is at rank 15 in 1. combination for us would be:
Next, each list is traversed up to a depth to fetch the top- items per mention. The goal
is to find combinations of KB items ⟨ ⟩=1 that best match the question. For instance, an ideal
{ c h a r a c t e r r o l e , V i s e r y s I T a r g a r y e n , G e o r g e R . R . M a r t i n , H B O n e t w o r k , T V s e r i e s }
These combinations come from the Cartesian product of items in the lists, and would have
possibilities if each combination is explicitly enumerated and scored. This is cost-prohibitive:
since we are only interested in some top- combinations, as opposed to a full or even extended
partial ordering, a more eficient way of doing this would be to apply top- algorithms [
Thus, we propose an approach using top- algorithms to overcome this challenge. To go beyond shallow lexical matching, our proposal is to construct multiple lists per question token, each reflecting a diferent relevance signal . Specifically, we obtain one list for each mention and score . Then, we apply top- algorithms on these lists to obtain the disambiguation of each question token individually.
Note that considering the question as a whole is a key criterion for our scoring mechanism. Therefore, we integrate two global relevance signals. Specifically, a candidate KB item combination that fits well with the intent in the question is expected to have high semantic coherence and high graph connectivity within its constituents. These can be viewed as proximity in latent and symbolic spaces. Further, candidates should match well on question and mention levels. These motivate our four relevance signals for each item in list below.
Coherence. We consider global signals for semantic coherence and graph connectivity, which
are inherently defined for KB item pairs, on a global level, instead of single items. Therefore,
we need a technique to convert these signals into item-level scores. The idea is to use a max
operator over all candidate KB item pairs involving a candidate at hand. More precisely, the
coherence score of an item is defined as the maximum item-item similarity (averaged over
pairs of lists) this item can contribute to the combination. The pairwise similarity is obtained
by the cosine value between the embedding vectors of the two KB items, min-max normalized
from [−1, +1] to [
ℎ( ) =
1 − 1 ∑ max ( ⃗ ≠
, ⃗ ) ( ) =
1
− 1
∑ max (
≠
, )
Term match. This score is intended to take into account the degree of lexical term match (via
TF-IDF, BM25, or similar) for which was admitted into . However, such TF-IDF-like weights
are often unbounded and may have a disproportionate influence when aggregated with the
other signals, that are all in the closed interval [
The global connectivity score is then converted to an item-level score analogously to the coherence, using max aggregation over pairs. Formally, we define the connectivity of as: ( ) = avg ≠ ( ⃗ , ⃗ ) Note that (
) ∈ [
Question relatedness. We estimate semantic relatedness of the KB item to the whole
input question by averaging pairwise cosine similarities between the embeddings of the item
and each term . The same min-max normalization as for coherence is applied. To avoid
confounding this estimate with the question term for which was retrieved, we exclude this
from the average. We define semantic relatedness as:
2.3. Finding top- across sorted lists
We then sort each of these 4 ⋅ lists in descending score-order. Note that for each and each
score , all lists hold the same items (those in the original ). Top- algorithms operating
over such multiple score-ordered lists, where each list holds the same set of items, require
a monotonic aggregation function over the item scores in each list [
“plays” is highly ambiguous, and thus is set to a relatively high value. “Viserys” and “HBO” can also refer to diferent concepts. The word “GRRM” is relatively unambiguous.
The native CLOCQ method was primarily designed for retrieving a search space of relevant KB facts for a given user question. Therefore, the linking of entities and relations is more of a means to an end here, and is not optimized for the specific tasks. We identified two key obstacles when using the plain CLOCQ method for entity and relation linking.
The first obstacle is that CLOCQ links all mentions in the question. Not only entities and relations are linked, but also types (like “series”) and other mentions (e.g. “latest” in the running example). While linking such mentions is beneficial for coherence among linkings, and can improve initiating a search space, it often adds undesired noise to the outputs when evaluating entity or relation linking capabilities.
For example, CLOCQ would link “series” to the entities TV series and the relation “latest” to latest start date for the running example, which would both decrease precision.
The second obstacle is that CLOCQ does not diferentiate between entity and relation mentions. Any mention is disambiguated to the KB items that score best w.r.t. the specified scoring mechanism. For example, the relation mention “plays” could also be linked to the entities play or playwright, similarly as “director” could be linked to the relation director or to the type director. Again, this does not hurt when initiating the search space, but definitely restricts the relation linking capabilities of CLOCQ.
In the following, we will discuss how we optimized CLOCQ for the entity and relation linking tasks of the SMART 2022 challenge. The same intuitions apply to other entity or relation linking problems as well.
As discussed, linking all mentions jointly is beneficial for linking results, since it considers information on the whole question in the linking stage. This follows our intuition of understanding the question in its entirety. Also, we did not want to touch the main CLOCQ algorithm itself. Instead, our idea is to prune the linkings returned by CLOCQ. We propose a simple approach: the decision, whether an entity should be included in the linking results or not, should be made depending on the mention the entity was disambiguated for. If the mention should be disambiguated, we add the linking, if not it is dropped from the results. For example, the mentions “plays”, “latest” and “series” should not be disambiguated when solving an entity linking task.
Training. We aim to learn which mentions should be linked (and which not) using distant
supervision on the training data provided as part of the SMART task. Given a training instance,
we first obtain all ⟨mention, KB entity⟩ pairs (i.e. the linkings) using the native CLOCQ
method. From the training instance, we know the gold entities that should be linked. We
then consider all mentions, that are linked with a gold entity by CLOCQ, as mentions that
should be disambiguated. For the running example we would obtain the mentions “Viserys”,
“GRRM”, and “HBO”, assuming the gold entity set ⟨Viserys I Targaryen, George R.R. Martin, HBO
network⟩. With this information, we can create a training instance for learning the relevant
entity mentions. The input is the question, and the output is the concatenation of the mentions
linked to gold entities “Viserys|GRRM |HBO” separated by the special token “|”. We then simply
ifne-tune a pre-trained sequence generation model on this data. For this purpose we used
BART [
Inference. At inference time, the pruning module is applied in a post-hoc manner. We first run CLOCQ and our trained pruning module on the input question. We then keep a ⟨mention, KB entity⟩ pair only if the disambiguated mention matches with any mention generated by our pruning module. Here, matching is relaxed to substring matching. For example, if the pruning module generates “in GRRM” or “GRR”, linking pairs for “GRRM” are still kept. In addition, for the entity linking task, we remove all relations from the linkings (relation identifiers start with a “P” in Wikidata).
Note that the post-hoc pruning module is also capable of learning benchmark-specific properties. The SMART 2022 entity linking task can often (but not always) require linking types or concepts. For example, “airline” in “DC-3 is operated by which airline?” should be linked, but not “continents” in “How many continents are in Antarctica?”. Such benchmark characteristics are learned implicitly by our pruning module, which can help improve the performance. 3.2. Increasing for relation linking As mentioned earlier, relation mentions may also be linked to entities that are coherent with the other linkings. We found that this can often be the case for CLOCQ linkings, and that the appropriate relation can be deeper in the ranked linkings of a mention than the automatically set cut-of length . However, relations can easily be diferentiated from entities via the identifier (relation identifiers start with a “P”, entity identifiers with a “Q” in Wikidata). We therefore simply set =50 and =40 to increase the probability of obtaining relations, and prune all entities from the linkings. Finally, we explore the efect of keeping either the top-ranked relation per mention, or all relations per mention as the final result.
SMART 2022 tasks. Statistics on the entity linking and relation linking tasks of the SMART Task 2022 can be found in Table 1 and 2. For both tasks, the question and the corresponding gold entities or relations are given for the train set. For the test set, only the question is given. The datasets are made publicly available3.
Metrics. We use the standard metrics of the SMART 2022 Task for both tasks: i) precision, that measures what fraction of the predicted linkings are correct, ii) recall, that measures what fraction of the gold linkings are found, and iii) F1 score, the harmonic mean of precision and recall. The results on the test set were provided by the task organizers, after we submitted our system results (since the gold standard is not publicly available).
Initialization of CLOCQ. In our experiments, we use Wikidata [
All parameters are kept at default values ( =20, ℎℎ =0.1, ℎ =0.3, ℎ =0.2, ℎℎ =0.4) [
Initialization of pruning module. For implementing the pruning module, we use the pretrained BART model available on the Hugging Face library6. We make the code for the pruning module publicly available7. We choose =1 for CLOCQ during distant supervision (Sec. 3.1). The model is fine-tuned for 5 epochs, with 500 warm-up steps, a batch size of 10, and a weight decay of 0.01. We employ cross-entropy as the loss function. After each epoch, we run the model on the withheld dev set, and finally choose the model with the lowest loss there. CLOCQ variants. On the entity linking task, we compare the linking results of the native CLOCQ method with =1 or =AUTO, with the linking results after applying the post-hoc pruning module (again, =1 or =AUTO). On the relation linking task, we consider either the top-ranked relation per mention, or all relations per mention, as returned by CLOCQ.
The results on the entity linking task are shown in Table 3. When considering only the top-1 entity per mention, CLOCQ obtains a recall of 0.766. Setting =AUTO improves the recall by ≃ 0.1, indicating that potential errors can be overcome. Further, activating the pruning module 3https://github.com/smart-task/smart-2022-datasets 4https://clocq.mpi-inf.mpg.de 5https://github.com/PhilippChr/wikidata-core-for-QA 6https://huggingface.co/facebook/bart-base 7https://github.com/PhilippChr/CLOCQ-pruning-module can drastically improve the precision of CLOCQ, and thus also the F1 score. When adding the pruning module for =1, precision jumps from 0.281 to 0.714. Also, the results indicate that mostly noise is pruned, since the recall remains fairly stable. Again, recall can be substantially improved (≃ 0.1) by setting =AUTO, with the cost of a lower precision and F1 score.
The results indicate that the pruning module can successfully reduce noise in the entity linking
results. Further, we found that there is a trade-of between precision and recall, which makes it
impossible to determine a best variant for all scenarios. The best choice may highly depend on
the specific QA system. Some QA systems require precise linkings for each mention [
For example, a QA system optimized for eficiency may only issue exactly one explicit query
to the KB [
For example, the queries with incorrectly linked entities in top positions might not return
any result, while queries with lower ranked entities are able to identify the correct answer.
Re-ranking results after query execution might also be an option. When following a graph-based
approach without explicit queries, setting =AUTO was found to be beneficial [
An anecdotal example from the dev set for which an automatically increased helps is: “What was Toby Wright’s profession?”. There are diferent persons named “Toby Wright” in the KB, and the context does not help to resolve the ambiguity. With =1, only the incorrect Toby Wright (football player) is returned. When setting =AUTO, CLOCQ identifies the ambiguity of the mention and sets =2 for this mention. The correct entity Toby Wright (record producer) is then fetched at second rank of the results.
Another interesting question from the dev set is “which footballer was born in middlesbrough?”. The mention “footballer” indicates that the question is on the topic of football, and therefore CLOCQ provides Middlesbrough F.C. (football club) as the top-ranked linking for “middlesbrough”. However, in this question “middlesbrough” refers to the corresponding town. Again, in the auto- mode, CLOCQ chooses a higher (=3), and includes the correct entity in these top-3 linkings ⟨Middlesbrough F.C. (football club), Middlesbrough (borough), Middlesbrough (town)⟩.
The results on the relation linking task are shown in Table 4.
As for the entity linking task, considering more linkings per mention can help to boost recall, by 0.05 in this case. Again, precision drops substantially, leading to a decreased F1 score. Considering only the top-ranked relation per mention achieves the better F1 score. Interestingly, the average number of relations per question in the system results is quite close to the average number of gold relations (we assume that this property is similar on the train and test sets).
Overall, the results indicate that relation linking may require methods optimized specifically
for this purpose. Still, being a general linking method, CLOCQ can provide the correct relation
for a substantial part of the questions, often bridging the lexical gap between the relation
mention and the surface form of the relation in the KB. Note that there are very few existing
systems that can perform both entity and relation linking: this is one of the novelties in CLOCQ.
Another such system is [
For example, for the question “Which child of John Adams died on February 23, 1848?” from the training set, CLOCQ correctly links “child” to child, and “died on” to date of death. However, for questions like “What is the point in time that Nicolaus Cusanus was made cardinal by the Holy Roman Church?”, CLOCQ failed to link the correct relations start time and position held.
There has been extensive research on entity linking and we discuss some prominent works
here. TagMe [
Coming to more recent neural systems, REL [
Relation linking is particularly useful for QA systems constructing an explicit query. Early approaches used paraphrase-based dictionaries [28] or patterns [29] to link relation mentions. Following approaches often leveraged semantic parses [ 30] for relation linking, which has also been shown to be efective in combination with neural models [ 31]. There is also a line of work that approaches relation linking as a classification task [ 32, 26, 33]. While these methods often achieve high accuracy, a common bottleneck is that only a fraction of all KB relations that are provided in the benchmark can be recognized. Therefore, they are mostly applied in the context of information extraction (IE), rather than QA. Finally, for previous iterations of the SMART Task in 2020 and 2021, a range of relation linking methodologies has been proposed and evaluated [34, 35].
Entity and relation linking are complementary problems, where the results of one task can help
solving the other. Thus, either linking is often an intrinsic part of the QA pipeline itself, in
which entity and relation linking are implicitly solved in a joint manner [
We apply CLOCQ [
Question answering with knowledge base, in: ACL-IJCNLP, 2015. [27] Y. Yang, M.-W. Chang, S-mart: Novel tree-based structured learning algorithms applied to tweet entity linking, in: ACL-IJCNLP, 2015. [28] M. Yahya, K. Berberich, S. Elbassuoni, M. Ramanath, V. Tresp, G. Weikum, Natural language questions for the web of data, in: EMNLP, 2012. [29] C. Unger, L. Bühmann, J. Lehmann, A.-C. Ngonga Ngomo, D. Gerber, P. Cimiano, Templatebased question answering over RDF data, in: WWW, 2012. [30] W.-t. Yih, X. He, C. Meek, Semantic parsing for single-relation question answering, in:
ACL), 2014. [31] T. Naseem, S. Ravishankar, N. Mihindukulasooriya, I. Abdelaziz, Y.-S. Lee, P. Kapanipathi, S. Roukos, A. Gliozzo, A. Gray, A semantics-aware transformer model of relation linking for knowledge base question answering, in: ACL-IJCNLP, 2021. [32] D. Zeng, K. Liu, S. Lai, G. Zhou, J. Zhao, Relation classification via convolutional deep neural network, in: COLING, 2014. [33] J. Feng, M. Huang, L. Zhao, Y. Yang, X. Zhu, Reinforcement learning for relation classification from noisy data, in: AAAI, 2018. [34] N. Mihindukulasooriya, M. Dubey, A. Gliozzo, J. Lehmann, A.-C. N. Ngomo, R. Usbeck, Semantic answer type prediction task (smart) at iswc 2020 semantic web challenge, arXiv (2020). [35] N. Mihindukulasooriya, M. Dubey, A. Gliozzo, J. Lehmann, A.-C. N. Ngomo, R. Usbeck, G. Rossiello, U. Kumar, Semantic answer type and relation prediction task (smart 2021), arXiv (2021). [36] A. Abujabal, M. Yahya, M. Riedewald, G. Weikum, Automated template generation for question answering over knowledge graphs, in: WWW, 2017. [37] H. Joko, F. Hasibi, K. Balog, A. P. de Vries, Conversational entity linking: Problem definition and datasets, 2021.