This paper describes the work that we did at Indian Institute of Technology (ISM) towards CheckThat!: Automatic Identi cation and Veri cation of Claims for CLEF 2019. As per requirement of CLEF-2019 we submit the only one run in its Check Worthiness Task. Two-fold crossvalidation is used to select a model for submission to CheckThat Lab at CLEF 2019. For our run, we use SVM and LSTM method. Overall, our performance is not satisfactory. However, as new entrant to the eld, our scores are encouraging enough to work for better results in future.
Investigative journalists and volunteers have been working hard to get to the
core of a claim and present solid evidence in favor or against it. And in this day
and age of information abundant amount of info/data is readily available, thus
manual fact-checking is very time-consuming and therefore automatic methods
were proposed as a means of speeding up the process [
The rate at which a statement in an interview, a press release, or a tweet can
spread almost instantly across the globe has created an unprecedented situation
[
The organization of the rest of the paper is as follows. Section 2 describes about the dataset. We describe our methodology: eld categories and indexing, which document and topic elds we used for retrieval in Section 3. In Section 4 we describe our results. Finally, we conclude in Section 5 with directions for future work. 2
We participate in Task 1. The detail description of dataset can be found in
[
Data Preprocessing and Feature Extraction Data Processing is required to prepare data to be used by systems to classify and rank statements according to the check-worthiness score. Rich feature extraction is needed to perform ranking according to language constructs rather than relying on heuristic or encyclopedia Knowledge. Syntactic and semantic features were extracted from both speeches and debates to represent sentences consistently and converted every sentence into vector. These features are: Lexical Features, Sentence Embedding, Stylometric Features, Semantic Features, a ective Features and Metadata Features. First of all we perform speaker normalization by assigning each speaker an unique Id. The reason for this is that the same speaker is present in the training data with di erent names in di erent instances. The sentences in the training data are rst tokenized. We also remove stop-words and the remaining tokens are stemmed using a stemmer. A single le which contains sentences extracted from required le is made to be used as training data. Syntactic and semantic features are extracted from both speeches and debates to represent sentences consistently and converted every sentence into vector.
We use two approaches for designing the model for this task Support Vector Machine (SVM) and Recurrent Neural Network using Long Short Term Layers (LSTM). We use SVM because of its simplicity, ease of use and its ability to avoid over- tting using regularization parameters. However, since the datasets are more aptly described using a sequence of sentences, a model which considers this sequence-time nature of the data will obviously be a better choice. Since SVMs do not consider this, we also try to implement a recurrent neural network model having memory units, in the form OF LSTM layers.
For our model, we use a layer of LSTM cells having an input shape of (batch size, no. of time steps, feature dimension). There were 64 units of LSTM cells in each layer hence, the output shape of the layer will be (64). We then add a dense layer of 64 neurons after the LSTM layer with a Recti ed Linear Unit (ReLU) activation function. A dropout layer is also added to avoid over- tting of data. Finally, a softmax output layer is added producing output probabilities of the classes for each input instance. 4
The complete training data is divided into two parts i.e. training data and validation data. The validation data consists of a speech and a debate selected from the training part. The models are trained using the training part and evaluated using validation part. Many models with di erent parameters are generated and evaluated and the model giving the best results is selected from them. We are provided with some code that helped us to evaluate our models in various evaluation metrics. We get the best result using Recurrent neural network model rather than SVMs. The scores obtained by our six runs are given in Table 1. The o cial evaluation measure provided by CLEF'19 is M AP . We show the best score in the task demonstrated by run-id Copenhagen(*), for the sake of comparison. Copenhagen(*) 1 .1660 .4176 .1387
ISMD16title eld 10 .0835 .2238 .0714 This year we participate in Task 1 of the Check That!: Automatic Identi cation and Veri cation of Claim. A SVM and a Recurrent Neural Network model is used with supervised learning to classify and rank sentences according to their check-worthy score in political debates and speeches. A rich feature set is extracted to represent sentences in best possible way to tackle class imbalance and avoid heuristic or encyclopedia Knowledge. Two-fold cross-validation is used to select a model for submission to CheckThat Lab at CLEF 2019. While there can be no denial of the fact that our overall performance is average, initial results are suggestive as to what should be done next. Our work has shown us lot of interesting possibilities for future work. There is a lot of room for improvements in our task. Linguistic form of information is under-studied and can be explored in more depth for better result. We use shallow syntactic features in our project, this can be improved by using very deep syntactic feature to represent sentence. Further study is required for including more linguistic and non-linguistics features which may a ect check-worthy score of phrases. Furthermore, we can use more complex recurrent neural network models to achieve better result. We shall be exploring some of these tasks in the coming days.