Nowadays, information on any topic can be researched on the Internet. However, in addition to reputable news sources, there is also a great deal of fake news that is disseminated, e.g., via social media or in established newspapers. Thus, the veracity must be assessed for each piece of information. People, parties, and organizations want to push through their interests and sometimes do not hesitate to spread fake news. For some time now, one popular means has been to quote (supposed) experts in a field. For example, -due to his authority- Albert Einstein is often quoted by believers in God although he was primarily concerned with physics while his quotes on God are taken out of context. In this paper, we define a new task of expert suitability prediction and evaluate methods to assess the credibility of a person with reference to a statement and its context and compare it to state-of-the-art approaches applying transformer-based embeddings. In an R4 cycle in CBR this approach could be used for the ranking. In this pilot study, we restrict our experiments to researchers, which allows us to derive their expertise from their publications. Furthermore, we make a manually labeled dataset consisting of 1,700 (statement,expert) pairs where suitable experts were tediously searched out together with valuable context information (such as convincing text parts of the experts' contexts towards a statement) publicly available to stimulate further research in this very important, but up to now underrepresented area of fake news detection.
The Internet ofers countless opportunities to consume information, and social media such as
Facebook and Twitter increase the likelihood of contact with news [
Due to its efectiveness, a frequently used tactic to strengthen the message of own views is to
quote scientists on topics that are not the core area of their research. An infamous example
is Albert Einstein, who is often quoted by believers in God, although Einstein was primarily
concerned with physics and his quotes on God are taken out of context [
To tackle this issue, this paper presents a pilot study for validating the suitability of experts towards statements. To the best of our knowledge, there is no prior work addressing this. Since it is harsh and controversial in practice to generalize when a person is an expert in a field, we restrict our experiments to researchers only, since (1) we can derive their expertise from their publications and (2) the expertise of scientists with experience in a field is usually not disputed; the extension of the application to non-scientists is then part of the future work. Certainly, people who neither conduct research on a topic nor have published on it can also be experts on a number of subjects but they are usually not consulted as experts on controversial matters to increase trust such as energy supply or behavior in the event of military action. Thus, in this paper we pursue to answer the question whether somebody would believe a supposed expert if they made a certain statement. However, to find out whether the expert in question actually made the statement is not part of this work. In CBR, this approach could be used as ranking component in the R4 cycle.
We make the following contributions: (i) We define the new task of assessing researchers’ expertise regarding controversial statements. (ii) We provide a dataset containing 1,700 manually labeled (statement,expert) pairs along with important contextual information to push this research forward.1 (iii) We investigate the performance with state-of-the-art machine-learning approaches to make predictions about researchers’ expertise towards statements.
Next, we address related work in Section 2. In Section 3 we define the task of assessing researchers’ expertise regarding controversial statements and introduce the dataset. Then, we present our methods and report the results of our evaluation in Section 4. We conclude the paper and give an outlook to future work in Section 6. 1The dataset is available at the following link: https://doi.org/10.5281/zenodo.6586678
Due to the increasing prevalence of fake news, the research area of fact checking is becoming more and more prominent in NLP. However, to the best of our knowledge, no one has validated expert statements and particularly not by examining scientific publications of researchers. Hence, we briefly survey the state-of-the-art in (1) research in general, (2) evidence-based research studies, as our approach might be used to identify resources to fact check claims, (3) credibility prediction studies, and (4) data fusion and worker expertise in crowdsourcing, as our method has some similarities to them.
Fake News detection in general In their meta-analysis, Thorne et al. [
related to our paper to a certain degree are, to the best of our knowledge, evidence-based research studies and credibility prediction studies.
Those works consider social networks such as Twitter and measure, e.g., user influence [
Source Credibility in Data Fusion and Worker Expertise in Crowdsourcing Other
related fields are source credibility in data fusion and worker expertise in crowdsourcing. In the
ifeld of data fusion, the goal is to combine data from multiple sources to achieve a more accurate
overall picture than if only one data source is considered. For the calculation of the overall
picture, it is important to know the credibility of the individual sources and to include them in
the calculation [
of contexts min
focused min
all focused all
As indicated in Section 1, we aim to estimate a potential expert’s domain knowledge with respect to a statement using the context of both the statement and the potential expert. In this section, we first define the new task, then we discuss the dataset we created for this purpose. Note that this construction requires two components: the first consists of statements with their context. The second consists of potential experts and their expertise on these statements. Task Definition Given a statement with context (), as well as an expert with context (). The task is to determine whether is a credible expert given their context (), if making
Social innovation keep care affordable Social innovation prevents use of more expensive forms of care through better coordination between supply and demand.
Social innovation reduces care demands and costs by better
matching supply and demand.
Social innovation leads to fewer layers of management
enabling organizations to become cheaper.
Social innovation leads to lower absenteeism costs because satisfied employees are less sick.
Social innovation provides more fun at work Social innovation gives employees responsibility and they want to wear.
...
Finance
What are arguments for and against social innovation? Pro
Con Job satisfaction or being cited for the statement in context (). As we restrict our work to researchers, the context () will be represented by ’s publications as well as the research interests of . As in this study we aim to show that our approach is feasible and also to reduce complexity and to anticipate performance reasons, in this paper the expertise of each expert is represented by exactly one of their publications; an extension to all publications will be left for future work.2 Figure 1 depicts the type of context information we will use for the remainder of the paper. More precisely, we will investigate in all combinations of contexts for both experts and statements. The contexts are split into min, focused, and all. Here, min represents the minimum that could be required as context. The set focused expands the set min with further context that is available, at least for a longer period immutable, and realistic to be used in an application. The last set all contains even more context with the goal to examine its impact (1) either with information where the methods to determine are still being researched as in the case with the context of the statement and (2) with data that is rather related to the authority of a researcher as in the case of an expert’s publication.
al. [
We chose this dataset for several reasons: first, it covers various (Dutch) national as well as international political issues such as the environment or pensions, on which people might want to listen to expert opinions. Second, since the dataset consists of argument graphs, the root of each graph describes the core question, while each of the other nodes contains further 2We argue that this is not a drawback since in this case an expert would be classified as suitable as long as at least one of their papers fits. example from dataset Infection Prevention About this card: This Argumentation Card provides an overview of arguments for or against the implementation of infection prevention measures in all forms of residential care in the elderly. Healthcare professionals have laid down . . .
What arguments do nursing homes use for and against implementing infection prevention measures? In front of Feasibility Infection prevention prevents unnecessary work Caring for sick people takes extra time and is a drain on the daily routine. partial information such as the stance, the frame, or a cluster (with a short summary with the quintessence) of the statement. As we aim to incorporate diferent contexts to each statement in order to evaluate the estimation of expertise, this dataset allows us to do that conveniently. Starting with this dataset, we randomly picked 200 statements with their context information to proceed from there. To avoid bias, we constrained that no statement would be included in this sample if there was already another one with the same stance, the same frame, and consequently the same quintessence. An example of a statement with its context is shown in Table 1. Enriching the Dataset with Suitable Experts W.r.t. these 200 statements, three members of our team searched on Google Scholar for two ideal experts for each statement, i.e. researchers who had published papers that are very related to the content of the statements. We decided to use Google Scholar as our expert source as it is the largest scientific literature database of authors and their publications including diferent disciplines that we are aware of. It includes not only diferent disciplines, but also diferent source documents such as professional articles, theses, dissertations, books, abstracts, or court opinions.
The exact task was to find experts for these statements who would be believed if they were cited to substantiate them. In this context, we asked the annotators to find two experts from diferent papers for each statement, i.e., the expert of paper 1 was not allowed to be a co-author of paper 2 and vice versa.
One annotator proposed experts for statements, and the other two either approved them or replaced them with others that were in turn reviewed by the others. The annotators developed text snippets based on the statements to retrieve suitable experts when typing them into Google Scholar.
In finding the best variation of search snippets, they were allowed to be creative and enter words that do not appear in the sentence. The decision whether an expert was a good match or not depended on several characteristics, such as the title of their publications, their abstracts, and their research interests which are indicated by the scientist .3
From the 200 statements, a total of 340 experts could be determined for 170 statements. The remaining could not be associated with any experts at all, as they are not suitable for verification, e.g. because the statements are far too general, or even far too specific about regional problems.
W.r.t. the example in Table 1, the statement “Caring for sick people takes extra time and is a
drain on the daily routine.” was changed to the search snippet “daily efects of caring for sick
people” for which Google Scholar, finds, i.a., the paper titled “ Diferences in impact of long
term caregiving for mentally ill older adults on the daily life of informal caregivers: a qualitative
study” [
Rating of Random Scientists to the Statements Based on these 170 statements and the 340 experts, we randomly picked eight additional experts from this expert pool for each statement, yielding a total of 1,360 additional (statement,expert) pairs. Two annotators independently inspected the researchers’ Google Scholar profiles for these pairs and, based on the research interests, as well as their paper titles and abstracts, assigned the labels 0, 1, and 2, indicating whether the researcher is a suitable expert to make the statement. A third annotator also assigned a label if the ratings of the first two difered. This happened 532 times which shows the dificulty of this task caused by the subjectivity of the annotators.
The task was to assume that someone refers for statement to expert . Then, the annotators should indicate on the basis of the ’s research whether they would be convinced while neglecting the truth content of , i.e., they just concentrated on the topics of and the expertise of .
The label 0 was assigned if the expert did not match the statement at all. This could be, e.g., if the expert is a psychologist, but the statement deals with processors. In contrast to that, the label 2 was assigned if the expert is a perfect fit for the statement. The label 1 was assigned to experts who fit partially. This decision is slightly more dificult to assign because this is more subjective. The annotators were instructed to assign e.g. an economist the score 1, if the statement has something to do with economics. Here we made the assumption that they must have learned the same in their basic studies and are therefore reasonably familiar with each other’s subjects. However, obviously, this assumption may not always be correct.
Altogether, we received 3,252 (=2 · 1,360 + 532) assessments for the 1,360 statements. Label 0 was assigned a total of 2,192 times, label 1 a total of 909 times, and label 2 another 151 times.
We measured the robustness of the annotations using Krippendorf’s . Measuring the
assessments of the first two annotators resulted in an inter-annotator agreement (IAA) of 0.262
(interval metric). This IAA improved to 0.411 by adding the third annotator. According to
Landies and Koch [
This value is quite low and indicates the dificulty for this task from a completely diferent point of view, namely the subjectivity in evaluating expertise. We argue that the annotations 3Note that in addition to the expert itself, we include both the search snippets and the text snippets that convinced the annotator in our dataset. are nevertheless better than a first glance would suggest as the third annotator knew that the two annotators had previously given unequal annotations, and the latter therefore weighed particularly carefully what to assign.
Final dataset Finally, our dataset consists of a total of 1,700 labeled (statement,expert) pairs, where 340 were labeled with a score of 2 because the associated experts were manually picked for these statements. For the remaining 1,360 pairs, we used majority voting of the three annotators for the final label. Especially in light of the improved IAA, this seemed reasonable. W.r.t. the 1,360 pairs, 997 were assigned with label 0, another 309 were assigned with label 1, and 54 were assigned with label 2.
With the last section introducing a dataset suitable for learning and evaluating the prediction of researcher’s expertise, in this section we now present and evaluate methods for accomplishing this.
We examine three approaches: Our main approach bases on a transformer-based and finetuned cross-encoder model. In particular, as we want to measure the performance of diferent contexts (see Section 3), we consider all 9 (=3 · 3) combinations of the subsets {min, focused, all}× {min, focused, all} of contextual information as shown in Figure 1 and explained in Section 3.
The next approach serves as baseline and makes use of the state-of-the-art BERT where the resulting embeddings are classified by several, i.e., seven standard classifiers such as a support vector machine or gradient boosting.
The last approach serves as a comparison and makes use of a classical and very successful
IR approach, namely BM25F [
Classifiers based on Cross-Encoders Due to the great success of transformer-based
embedding methods, which have brought about great positive impact in the field of NLP, it is
appropriate to use a state-of-the-art model for predicting the expertise of researchers with
respect to statements. After weighing the advantages and disadvantages of the various
frameworks and approaches, our main approach consists of a cross-encoder [
In our experiments, we created 10 folds for this purpose and evaluated them via crossvalidation. We employed the python framework sentence-transformers and utilized the model “roberta-large”. More precisely, we fine-tuned this model for each combination and each (train,test) fold for 3 epochs always with a batch size of 16. We refer to this method using the cross-encoder with CERoBERTa.
standard classifiers. As input, the classifiers received embeddings created with the pre-trained
Sentence-BERT model [
BM25F This approach serves as comparison. The intuition behind this is the assumption that the experts’ knowledge represented by their textual publications could have textual overlap to a statement or its context. In order to implement the approach with BM25F, we first indexed all experts together with their context information (as shown in Figure 1) using the Java framework Apache Lucene. The goal here is to enter a statement and get a list of potentially matching experts. The main diference between BM25 and BM25F is that the latter allows us to add more ifelds than just one to the query. 4
We therefore added the fields from the statement’s context, which are shown in Figure 1, to the queries. For example, with the combination min × min, we have a total of four search fields, since we look for both the statement and the query in the research interests and the titles. When querying the statement, the result is a list of ten experts with scores that are above Apache Lucene’s internal and default threshold, but do not yet correspond to our labels 1 (partial expert) and 2 (full expert) and thus have to be converted. 5 In order to get the maximum out, we use wildcards for this prediction, i.e., the correct prediction 1 or 2 is automatically assigned to the expert if BM25F lists the researcher in its list. Using this oracle, we are able to detect the upper bound of this approach with more clarity. We coin this method OracleBM25F.
We measured the performance with precision, recall, F1, and accuracy. Therefore, we computed
these mean average values in a 10 fold cross validation for each context combination. More
precisely, the precision is calculated by the fraction of the experts that were predicted as 1
or 2 for which the prediction was correct. Further, the recall is computed by the fraction of
the experts labeled as 1 or 2 for which the prediction was correct.6 To better interpret these
values, we also included a method called Zero in the evaluation that always predicts 0 because
4In the Java framework Apache Lucene this is done by using the class BlendedQuery [
Table 2 shows the performance of the examined methods. Note, that we excluded all from the expert combinations in the method OracleBM25F as it does not make any sense for textual string matching, e.g., to search for the number of citations of a researcher. The values are sorted in descending order according to the F1 score, since the accuracy alone can be misleading. Going by the accuracy, we get the best performance by always predicting 0 (see method Zero with accuracy 0.764). However, in this case we also obtain 0 for the precision, the recall, and the F1-score.
The second best accuracy is 0.76 and is obtained with our classifier CERoBERTa with the contexts focused for statement and all for the expert. The accuracy value is almost equal to that of Zero but the other scores in the table show that CERoBERTa is much more useful as its F1 score (0.627) is also the highest, revealing that this approach also provides reasonable results for partially and full experts and not only for non-experts. We also see that while CERoBERTa seems to be a precision-oriented approach (0.783) OracleBM25F tends to be a recall-oriented method (0.721). W.r.t. the baseline we only show the classifiers with their best performing context combinations. Following the table, the method MLPRoBERTa produces the second best performance with the contexts min, respectively, when using the 1 score. While the recall of MLPRoBERTa (0.551) performs comparably well to the best method CERoBERTa(0.526), CERoBERTaachieves the better precision (0.783 instead of 0.625) and thus the better 1-score (0.627 instead of 0.584).
We can infer from the table that OracleBM25F performs better the more context we feed into
it. Note, that the good performance of OracleBM25F probably results from the fact that it works
partly as an oracle. Actually, it performs very poorly when it is not fed with all statement’s
context information. The rankings also suggest that the baseline is more sensitive to the context
of the statement. CERoBERTa performs best when providing focused context information of the
statement and all context information of the expert. However, these numbers should be treated
with caution, because the performance was boosted by adding publication-independent features
such as the ℎ-index. Considering only the textual content of the publication, all × focused
produces the best and most realistic result when applying cross-encoders. Almost all standard
classifiers perform best when the context focused is chosen for the statement. Too much
information probably has a negative efect here. Regarding the context of the expert, it is a bit
more mixed. In particular, it is noticeable that the second best method in the table, MLPRoBERTa,
performs best only with minimal input in each case. We suspect that the context in the statement
loses its impact due to long background information in focused. In the future, we consider using
automatic summarization methods like T5 [
Citing putative researchers to strengthen own viewpoints is a widely used means of Fake News. In this paper, we defined the task of assessing a person’s expertise towards a statement and showed successfully that we can predict whether a researcher is a partially or fully suitable expert to be cited to believe a statement by fine-tuning a state-of-the-art transformer model. We make the dataset consisting of 1,700 labeled (statement,expert) pairs together with valuable information to train search engines publicly available for further research towards this new task.
As this evaluation can be seen as kick-of, there is obviously room for improvement. For example, in our study we restricted the dataset to researchers that already published in the ifeld of the statement’s topic. Further research needs to expand this to other people who can be experts that have not published papers in the fields of the statements’ topics but deal with them such as journalists or politicians. In addition to that, we always measured a researcher’s expertise by representing them as the content of exactly one publication (and its context). Naturally, future work needs to incorporate multiple publications of a researcher, e.g. to examine whether other similar works are suficient to predict a researcher’s expertise.
This work has been funded by the Deutsche Forschungsgemeinschaft (DFG) within the projects ReCAP and ReCAP-II, Grant Number 375342983 - 2018-2024, as part of the Priority Program “Robust Argumentation Machines (RATIO)” (SPP-1999).