To expand their boundary of knowledge, to solve a certain task, or merely to entertain themselves, people on the Web often hunt for surprising facts. While large repositories of structured knowledge like Wikidata hold the promise to provide surprising facts to their users and developers, currently no mechanism exists to identify surprising facts in Wikidata. In this paper, we study the ability of popular embedding models to estimate the surprise level of a fact in Wikidata. We formulate a novel task of Surprising Fact Identification , and we create two associated benchmarks: Trivia and Survey. We propose two scalable methods based on outlier detection and link prediction to estimate surprise scores for any statement in a graph like Wikidata. We evaluate our methods with various embedding models on the two benchmarks. We perform further analysis of the predictions for outlier and non-outlier facts to investigate to what extent link prediction models regress to the mean.
Surprise is a measure of how unexpected a certain statement is. For example, while it is expected
that Vladimir Putin speaks Russian, it is more surprising that he can speak German too. It
is expected that Putin is a politician, but it may be surprising that he has studied law. To
expand their boundary of knowledge, to solve a certain task, or merely to entertain themselves,
people on the Web often hunt for surprising facts [
The expansion of AI technologies and the vast amount of available knowledge provides an
opening for surprising facts to be identified automatically. Prior work by Tsurel et al. [
In this paper, we study the ability of popular embedding models, like TransE [
We define surprise as a measure of how unusual or unexpected a certain statement is. We formulate the task of Surprising Fact Identification as a ranking task. For a given set of facts , we want to assign a ranking to each of the facts ∈ that corresponds to the relative surprise a human would likely have at learning each is true. Each fact is of the form ∶ (, [( 1, 1), ..., ( , )]), consisting of an entity that is the subject of the fact and a non-empty list of predicate-object pairs ( , ). A fact indicates that each of the triples < , , > is true for ∈ [1, ] .
We collect two Surprising Fact Identification benchmarks: Wikidata-MC-Trivia-118 and WikidataSurvey-FunFacts-70.
Wikidata-MC-Trivia-118 This benchmark concretely defines surprise of a fact as the
probability that a human would not guess to be true. It consists of 24 multiple-choice questions
with a total of 118 candidate answers, each having an inferred human surprise score. Most
questions have five candidate answers, and questions can have more than one correct answer.
The questions are about 22 unique entities: 14 humans, 3 sovereign states, 3 paintings, 1
iflm, and 1 business. Of the 24 questions, 7 are single-answer (e.g., “What is the birthplace
of Boris Johnson”) and 6 ask about numeric values (e.g., “How many children does Arnold
Schwarzenegger have?”). Each question presents a single entity and property, and each candidate
answer is a potential object of that entity-property pair. Thus, all facts in this benchmark consist
of a single triple. We collected this benchmark with a quiz administered via Google Forms to
a group of 26 researchers who participated voluntarily. A screenshot of the quiz is shown in
2We expect that if a true statement is considered untrue by many participants, this is indicative of a surprising
fact. We expect that statements that are simply unknown would be guessed randomly by participants, leading to
an unsurprising score. We do acknowledge that a more principled annotation of this data may need to distinguish
between interesting, surprising, and unknown facts [
Wikidata-Survey-FunFacts-70 consists of 70 facts, each consisting of an entity and a
nonempty list of predicate-object pairs. Each fact is labeled with three scores indicating the degree
to which humans find the fact surprising, think the fact is a good trivia question, and would say
they knew the fact. The facts are about 47 diferent humans and only include entity-valued
properties. This benchmark was created by mapping a subset of the rows from the FunFacts
benchmark [
Prakash et al. [
Notably, Surprising Facts Identification is related to the popular task of Link Prediction
where the goal is to correctly predict the object of a subject-predicate pair. Previous work
has explored Link Prediction in Wikidata in various capacities. Safavi et al. [
We describe two novel methods that score the surprise for a Wikidata fact based on statistical outlier detection and on link prediction.
Our first method measures whether the entity is an atypical subject (outlier ) for the propertyobject pair. We estimate this by comparing to the set of entities for which < ′, , > is true ∀ ∈ [1, ] and ′ ∈ ∶ ′ ≠ . For example, to estimate whether < , , is an outlier, we would compare Putin to the entities which speak Russian according to Wikidata.
We compute the surprise score for a statement as a ratio between the distance between and the entities in , and the dispersion of entities within : ( ) = (, )/ ( We utilize embeddings to represent each entity in Wikidata. (, ) is a measure of distance from the entity to the entities in in the embedding space. We experimented with two such formulations: (, ) computes the cosine distance from to the centroid of and (, ) computes the average pairwise cosine distance between and each ′ ∈ . ( ) is a measure of how spread out is in the embedding space, where we again experiment with two analogous formulations: ( ) computes the average cosine distance from each ′ ∈ to the centroid of and ( ) computes the average pairwise distance between ′ and ″ for ′, ″ ∈ ∶ ′ ≠ ″.
Intuitively, the distance term causes an entity to be found more surprising for a fact if it is dissimilar from entities that have fact . The dispersion term normalizes this dissimilarity, decreasing our surprise for facts that many diverse entities have (e.g., “language written, spoken, or signed = English”) and increasing our surprise for facts that typically belong to very similar entities (e.g., “Occupation = Basketball Player”). For the edge case where | | < 2, there are too few entities in Wikidata that have all predicate-object pairs of for us to compute a dispersion score, so we assign the maximum surprise score (∞).
Another approach for identifying surprising facts is by using link prediction. Graph embedding models like TransE and ComplEx define ways of operating on their representations of entities and properties to yield predictions for where in the embedding space the corresponding subject or object entity resides. This gives us two potential methods to compute surprise scores: we define −ℎ ( ) as the aggregated distance from each ∈ to the predicted locations of the objects of < , >, and we define −ℎ ( ) as the aggregated distances from to each of the predicted locations of the subjects of < , >. There are several additional ). settings for these methods: the distances in both the lhs and rhs methods can be aggregated using either max or avg; the rhs method can be modified to instead compute the distance from to the centroid of the predicted object locations; and various distance functions can be used here, including cosine, L2, and negative dot-product.
Evaluation protocol We evaluate our methods’ ability to identify surprising facts by measuring correlation between their surprise scores with the human scores available in each benchmark using Spearman’s and Kendall’s . For Wikidata-MC-Trivia-118, we evaluate our method’s surprise scores by measuring their correlation with the inferred human surprise scores. To mimic the format that this benchmark was created in, we measure the average correlation of answers’ surprise scores within each question. We additionally report separate results for the subsets of questions that ask about entity-valued and numeric-valued properties. For Wikidata-Survey-FunFacts-70, we evaluate our method’s surprise scores by measuring their correlation with the crowdsourced scores for goodTrivia, surprise, and knew. A successful surprise identification method will give scores that correlate positively with goodTrivia and surprise, and negatively with knew. As this benchmark was created by presenting each fact to a human annotator independently, we measure the global correlation over all facts.
Models We experiment with eight embedding models: (1) BERT text embeddings that we
computed by automatically generating a node description based on seven properties and using
sentence-transformers to encode the resulting sentence; 3 (2, 3) TransE and ComplEx graph
embeddings computed directly on Wikidata; (4, 5) TransE and ComplEx graph embeddings that
have been computed over a graph derived from Wikidata by abstractive summarization into
‘profiles’ ( P-TransE and P-ComplEx) [
Method details We evaluate our outlier-based method with the eight unsupervised embedding models. We limit the size of for each fact to 10,000 by sampling. We evaluate our LP method with the embedding models TransE, ComplEx, P-TransE, P-ComplEx, and kNN-P-TransE, as these allow for unsupervised link prediction via translation and complex-diagonal operators. Vanilla TransE and ComplEx are not trained on numeric-valued edges and therefore cannot be used directly for LP on numeric-valued properties. Because the P- variants were trained on a graph that replaces numeric values with nodes corresponding to ranges, they can be used for LP on numeric-valued properties. Meanwhile, as the profile graph omits information from the original graph, the profile (P-) embeddings lack information for 19 of the facts in the WikidataSurvey-FunFacts-70 dataset, while the kNN model lacks information for 10 facts. For these missing facts, we use a frequency method as a fallback strategy. For the Wikidata-MC-Trivia118 benchmark, we report −ℎ for the vanilla and the supervised graph embeddings, and −ℎ for the profile-graph embeddings. For the Wikidata-Survey-FunFacts-70 benchmark, we use −ℎ for the unsupervised LP models, aggregating predictions by 3Properties used: P31 (instance of), P279 (subclass of), P106 (occupation), P39 (position held), P1382 (partially coincident with), P373 (Commons Category), and P452 (industry). measuring the distance from their centroid to the given fact’s subject entity. Analogously, we use −ℎ for the supervised LP model. We use cosine distance for all LP methods.
We define two baselines: random and frequency. The frequency-based method assigns the surprise score for a fact as the percent of Wikidata entities that it does not apply to.
The results on both benchmarks in tables 1 and 2 show that our SOD methods are able to identify surprising facts better than the baselines and our LP methods. On the Trivia benchmark, SOD yields the highest correlation scores with the BERT, TransE, and H embeddings, followed closely by the ComplEx embeddings. On the Survey benchmark, the H-RandomWalk embeddings perform the best on all three crowd-sourced scores, followed by ComplEx and TransE on the “goodTrivia” and “surprising” scores, and S-RandomWalk and BERT on “Knew”. The performance of the baselines is low on the Trivia benchmark, while the frequency baseline performs notably better on the Survey benchmark, giving scores that are competitive with most of our models. We think that this is due to the presence of compound facts in the Survey benchmark, whose combination reveals unlikely facts that align with human judgments of trivia-worthiness or surprise. Yet, several of the SOD methods improve over the frequency baseline on this benchmark as well, revealing that the subject entity embeddings provide key additional information that is not captured by the predicate-object frequencies.
Curiously, link prediction with the same embedding models performs consistently worse than the SOD methods overall and on the entity-valued facts, though this trend is reversed for the P-TransE and P-ComplEx on the numeric facts. This shows that, while the embedding models may contain valuable information, the method that uses them to identify surprising facts LP plays a key role in the final performance. While LP performs consistently better than random on both benchmarks, it has a relatively poor performance on the Survey benchmark, which may again be attributed to the presence of compound facts in this benchmark. Link prediction predicts a single subject vector at a time for each predicate-object pair and we account for the compound facts afterwards by computing their centroid. The centroid of these predicted vectors is an estimate of where in the embedding space an entity that has all of the predicate object pairs would be. This estimate may be poor when the predicted subject vectors are far apart (as in the case of facts that combine predicate-object pairs that rarely co-occur), and thus the LP method cannot as directly and efectively take into consideration interactions between such predicate-object pairs that are important to humans for determining if a fact is surprising.
Hypothesizing that LP methods regress to the mean of the entities with similar facts, we study the behavior of the embedding models further by analyzing the LP MRR of a fact in relation to the centroid of the cluster. The results are shown in table 3. The MRR for two of the three models is higher on the correct subset compared to the incorrect one, and on the non-outlier facts compared to the outliers. We observe that across any subset of the data, the supervised embedding model obtains highest MRR. Interestingly, while the supervised model performs the best on the link prediction task, its performance on the surprise task is the lowest out of the three models on this benchmark. The unsupervised version, P-TransE, assigns a comparably low MRR value on both the outliers and the non-outliers, yet, its performance on the surprise task is notably higher. This means that LP models optimize to predict a value that is typical rather than surprising. Based on these results, we hypothesize that more accurate link prediction models may not necessarily be better at predicting surprising facts; subsequent research on this topic is beneficial to further analyze their behavior.
In this paper, we studied the challenge of identifying surprising facts in Wikidata automatically. Inspired by earlier work on Wikipedia, we formulated a novel task of Surprising Fact Identification where an AI system has to mimic the surprise estimation of humans. We developed two novel benchmarks to evaluate representation learning models on this task. We proposed two generic methods to identify surprising facts based on statistical outlier detection and link prediction. Our experiments revealed that the best performance on both datasets was obtained with diferent variants of the outlier detection method. While link prediction methods performed relatively well on single facts, their performance on compound facts was much lower than the outlier detection methods and the frequency baseline. Further analysis revealed that link prediction models are optimized to predict typical values rather than surprising ones, as they tend to regress to the centroid of the entities with similar facts.
This paper represents a first investigation of identifying surprising facts in Wikidata by using automatic methods. However, the significance of our findings is limited by the small size of the benchmarks, which cannot be expected to be representative of the size of Wikidata. We have found it challenging to create a large evaluation dataset - a key future work task is to develop a more representative evaluation set. Future work should investigate the robustness of the obtained results by evaluating on such a larger dataset, and it should develop novel surprise methods that, for example, leverage entity embeddings in ways that focus on capturing surprise. We make our code and data publicly available to facilitate subsequent work on identifying surprising facts in Wikidata: https://github.com/usc-isi-i2/surprising-facts.
This material is based on research sponsored by Air Force Research Laboratory under agreement number FA8750-20-2-10002. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the oficial policies or endorsements, either expressed or implied, of Air Force Research Laboratory or the U.S. Government. [16] U. Joshi, J. Urbani, Searching for embeddings in a haystack: Link prediction on knowledge graphs with subgraph pruning, in: Proceedings of The Web Conference 2020, 2020, pp. 2817–2823.