The task of fact-checking has been formalised as the assessment of the truthfulness of a claim. Be it a political proclamation or a technological development, veri cation of a new tidbit of information before its propagation to the general public is of utmost importance. Failing to do so leads to the spread of misinformation, which is a devious tool. Fact-checking is commonly performed by journalists, manually looking up information pertaining to the statement in question. This is a drawn out and tedious process with a chance of the concerned person not covering the domain exhaustively. Some of this e ort is reduced by the use of knowledge bases created over a period of time. In this work under Task 2 (Factuality) of the CLEF 2018 CheckThat! Lab, we detail a neural network based methodology which models the textual data of a claim based on various representations of its words and characters. An a xed attention mechanism allows us to encapsulate linguistic features common in false claims. We achieve an accuracy of 39.57% on the task dataset.
Fact-checking is the task of evaluating claims made in discourse by public gures like politicians, pundits etc. The process of creation and curation of an article often takes a journalist through the fact checking phase. This task is a part of the pipeline in the creation of knowledge bases for various domains. Apart from journalists, with the advent of the present political scenario, institutes and government organisations dedicated to fact-checking have risen to prominence. ?? First four authors have equal contribution.
Lawyers, as part of the judicial process, are required to meticulously vet facts and statements before presenting it to the concerned authorities.
Even with the plethora of information available on the Internet, just a click away, it is a tedious and long drawn-out process. Generally, it involves rst checking the validity of the claim from a curated list of trusted sources. Next, the consistency of the claim is veri ed by looking it up against various sources. For the preceding to happen, the fact-checker has to rst fully dissect and understand the claim being made since there can be multiple interpretations of the same, or it can, in turn, be based on a combination of factors or pre-requisite claims. Websites providing this service generally provide in-depth analysis, encompassing the di erent possibilities, of a claim rather than a numeric score for its validity.
Fact-checking requires more research and a more advanced style of writing than ordinary journalism. The di culty of fact-checking, exacerbated by a lack of resources for investigative journalism, leaves many harmful claims unchecked, particularly at the local level. Its e ectiveness is negatively impacted by a gap in time and availability (de ned as the necessity of a checker to have to look up a documented fact nevertheless).
The emergence of the eld of computational journalism has promoted the
conversation about the need and importance of automatic fact-checking systems.
[
With the advent and exponential rise of social media platforms, dissemination
of news might wholly skip the traditional channel, thereby not being veri ed by
journalists at all. Apart from this, an increase in citizen journalism [
Another problem with current fact-checking platforms is their outdated nature of publishing. Many of these rely on older content management systems built just for newspapers and blogs that are not designed for structured journalism. This limits how well they can be used in computational pipelines.
We propose a neural network based architecture for the task. We leverage
the distributional semantics of the claim and model its temporal and sequential
properties. The contribution of a word towards the validity of the claim is
calculated in a di erential manner since the output of the LSTM is passed into an
attention layer [
Most of the work done towards accomplishment of this task requires manual
e ort. [
We now describe our approach to developing a fact-checking system and the reasons behind devising such a model. We start with an explanation of the type of embeddings we have used and proceed to describe our proposed model, a bidirectional long short term memory network with an attention mechanism. An overview of the architecture can be seen in Figure 1. Finally, we cover how the parameters are learned.
Considering the e ectiveness of distributional semantics in modeling language
data, we use a pre-trained 300 dimensional Word2Vec [
We now move on to describing the crux of our proposed approach.
Recurrent Neural Network (RNN) is a class of arti cial neural networks which
utilizes sequential information and maintains history through its intermediate
layers. A standard RNN has an internal state whose output at every time-step
which can be expressed in terms of that of previous time-steps. However, it has
been seen that standard RNNs su er from a problem of vanishing gradients [
(!h 1; !h 2; : : : ; !h R) represent forward states of the LSTM and its state updates satisfy the following equations: !ft ; !it ; !ot =
W h t 1; !rt + !b ! ! !lt = tanh !V !h t 1; !rt + !d !ct = !ft
! !c t 1 + !it lt !ht = !ot tanh( !ct ) (1) (2) (3) (4) here is the logistic sigmoid function, !ft , !it , !ot represent the forget, input and ! output gates respectively. !rt denotes the input at time t and ht denotes the latent state, !bt and !dt represent the bias terms. The forget, input and output gates control the ow of information throughout the sequence. W! and !V are matrices which represent the weights associated with the connections.
( h 1; h 2; : : : ; h R) denote the backward states and its updates can be computed similarly.
The number of bidirectional LSTM units is set to a constant K, which is the
maximum length of all title lengths of records used in training. The forward and
backward states are then concatenated to obtain (h1; h2; : : : ; hK ), where
hi = !h i h i
Finally, we are left with the task of guring out the signi cance of each word in
the sequence i.e. how much a word shapes towards a writing style characteristic
of factual claims. The e ectiveness of attention mechanisms have been proven
for the task of neural machine translation [
cattention =
j hj K X j=1 (5) (6) where, h1,. . . ,hK represents the sequence of annotations to which the encoder maps the post title vector and each j represents the respective weight corresponding to each annotation hj .
We use binary cross-entropy as the loss optimization function for our model. The cross-entropy method [14] is an iterative procedure where each iteration can be divided into two stages:
(1) Generate a random data sample (vectors, trajectories etc.) according to a speci ed mechanism.
(2) Update the parameters of the random mechanism based on the data to produce a "better" sample in the next iteration. 4 4.1
[
The rst distinction between the types of sentences is that some of them contain claims, and need to be processed by the model, while the others do not, and can be ignored. Further, each of the sentences that has been tagged as containing a claim is assigned a label pertaining to the validity of the claim. Each claim can be one of three types : TRUE, HALF-TRUE and FALSE.
Each record is associated with a line number (position in discourse of the debate), the speaker (which can be either candidate, or the moderator, or SYSTEM in case of external noise), a tag establishing whether it is a claim or not, a chronological claim number, and a label for the validity of the claim. In case of the sentence not containing a claim, it is considered part of a separate 'N/A' class. Table 1 shows some basic statistics found on analysis of the training data les. We combine the three les and then randomly split the 4064 sentences into training and validation set in a 4:1 ratio. This ensures that the two sets do not overlap. The model hyperparameters are tuned over the validation set. We initialise the fully connected network weights with the uniform distribution in the range p6=(f anin + f anout) and p6=(f anin + f anout) [12]. We used a batch size of 256 and adadelta [13] as a gradient based optimizer for learning the parameters of the model. 4.3
The model was evaluated over test set of 7 les which contained a combined total
of 192 possible claims to be veri ed. The subsequent result les were submitted
to the CLEF Fact Checking Lab [
The use of separate knowledge bases and other information sources would have contributed to higher accuracy or better recall for other participants' models [11] in this task. 5
We see that LSTMs are able to model claims and gure out possible dependencies between its constituents. Furthermore, the use of an attention layer allows our approach to capture features unique to fake claims and leverage them in the veri cation of others.
As part of improving this approach, we would like to try the addition of a knowledge base for political claims, and augmenting the dataset with tweets from Politico's Twitter handle tagged with #PresidentialDebate. It is also possible to generate a similarity score between a claim to be veri ed and each fact stored in the knowledge base, and evaluate if they refer to the same core idea. In: Proceedings of the Ninth International Conference of the CLEF Association: Experimental IR Meets Multilinguality, Multimodality, and Interaction 11. Barron-Ceden~o, Alberto and Elsayed, Tamer and Suwaileh, Reem and Marquez, Llu s and Atanasova, Pepa and Zaghouani, Wajdi and Kyuchukov, Spas and Da San Martino, Giovanni and Nakov, Preslav: Overview of the CLEF-2018 Lab on Automatic Identi cation and Veri cation of Claims in Political Debates, Task 2: Factuality. In: CLEF 2018 Working Notes. Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum 12. Glorot, X. and Bengio, Y.: Understanding the di culty of training deep feedforward neural networks. In: Proceedings of the International Conference on Arti cial Intelligence and Statistics (2010) 13. Zeiler, M. D.: ADADELTA: an adaptive learning rate method. In: arXiv preprint arXiv:1212.5701 (2012) 14. de Boer, P. and Kroese, D.P. and Mannor, S. and Rubinstein, R.Y.:A Tutorial on the Cross-Entropy Method. In: Annals of Operations Research (2005)