This paper explains about the description of the task carried out by the team Amrita CEN CIQ: Classi cation of Insincere Questions for the shared task conducted by FIRE 2019.The main objective of the shared task taken is to classify the insincere questions into six ne grained classes - Rhetorical questions, Hate speech/ in ammatory questions, Hypothetical questions, Sexually explicit/objectionable content questions, Other and Sincere/ true Information Seeking questions. The proposed system predicts the test data with an accuracy of 48.51%. The classi cation model used in this task is the Decision Tree Classi er. The Word embedding algorithm used for the extraction of features is Fasttext algorithm. A balanced Decision Tree is used as a classi er and proved to get better results when compared to the Random Forest Classi er with 0.52 F1-score.
Community Question Answering (CQA) is forum which had been used by several thousands of users for seeking information and also to retrieve answers for their queries. These kind of community answer seeking websites gained popularity in the recent past and had been used by many people across the globe. However, such websites fail to provide appropriate information to the users in most of the cases because of the improper usage of forums. Quora is one such website which is facing these issues. Identi cation of the insincere questions posted by the users will help to resolve the above mentioned problems.
The questions posted in the forums are labelled as insincere questions by analysing the aspect of a question. This analysis and identi cation of the aspect of the questions are challenging and signi cant. Therefore, Forum for Information Retrieval Evaluation (FIRE-2019) organized a shared task on classi cation of Insincere Questions (CIQ). This shared task aims to categorize an insincere questions into six classes - Rhetorical questions, Hate speech/ in ammatory questions, Hypothetical questions, Sexually explicit/objectionable content questions, Other and Sincere/ true Information Seeking questions. This kind of classi cation not only helps the community moderators to make the website user friendly but also the users to navigate appropriate information. Thus, the CIQ had created a dataset for the classi cation of Insincere questions by selectively choosing Non-Information Seeking Questions (NISQ) data created by Kaggle.
We (Amrita CEN CIQ) developed a classi cation model using the Decision
Tree classi er for the identi cation of the various aspects of insincere questions.
Since, the distribution of training data is not even among all the classes, we used
a weighted decision tree classi er which gave more weights to minority classes
and less weight to majority classes. The sentence vectors were constructed with
the fastText [
Description of the Task The main objective of the shared task CIQ : Categorization of Insincere Questions was to classify the insincere questions into six di erent categories based on the nature of questions asked such as Rhetorical questions, Hate speech, Hypothetical classes, Sexually explicit questions, Sincere questions and other kind of questions which cannot be classi ed into other categories. Classi cation of Insincere questions(CIQ) has taken a set of questions from the Quora dataset as the training and testing data. The training set contains 900 questions across six ne grained categories of insincere questions and testing data contains 100 questions.
The number of questions in each classes are not uniform. Class "0" refers to the questions which are not sincere and contains the minimum number of questions of only 20. Whereas, Class "1" refers to the rhetorical questions which has the maximum number of questions among all other classes of classi cation. The unequal distribution of the data among the six di erent classes is one of the challenges faced during the training of the model. The statistics of the training data is given in Table 1 and a visualization of the class distribution is shown in Figure 1. The steps involved in the developing the system submitted by Amrita CEN CIQ is illustrated in Figure 3.1
The training and testing data are initially preprocessed to remove the noninformative content. The steps involved in the pre-processing are given as follows. { Removal of website links and usernames { Lower casing { Word tokenization { Removal of stop words { Removal of punctuation
Each of the sentence in the corpus are tokenized into words using word tokenize() function which are imported from the library Natural Language ToolKit(NLTK). The NLTK library also contains stopwords of 16 di erent languages. The stopwords in the data is also removed using NLTK library. 3.2
In this work, we tried fastText and Doc2vec for vectorizing the sentences with the dimension of 100. Doc2vec directly represent the sentences into a vector of dimension 100 whereas in fastText, the sentence vector was generated by taking the mean of the word vectors of constituent words. On comparison between the two algorithms used, fastText has provided a better result for the feature extraction than Doc2Vec. The parameters of the Fasttext used are tabulated in the Table 2 . 3.3
We used weighted Decision Tree algorithm for classifying the insincere questions. The performance of the model was evaluated using 5-fold cross validation. The fastText features gave the testing accuracy of 40.70 +/- 0.02% while Doc2vec features obtained 37.16 +/- 0.01%. Hence, we used word vectors generated using fastText for the submitted system. The same trend follows with the test data also. The classi cation accuracy of test data is 48.51% when the data were vectorized using fastText and Doc2vec gave 35.64% accuracy in identifying the di erent types of insincere questions. The confusion matrix for the test data is given in the following g 3.
The accuracy, precision and f1-score for the model is tabulated below in Table 3 with a comparison between fastText and Doc2Vec algorithms.The subset parameters which are used for building Decision Tree model and Random forest model are tabulated in the Table 4.The advantage of a simple decision tree model that it is easy to interpret and the accuracy keeps increasing with the number of splits made in the training data. Whereas, in Random forest, the accuracy keeps increasing with the number of trees but becomes constant at a certain point of time. Unlike decision tree, it won't create highly biased model and reduces the variance. Hence, we have chosen decision tree instead of random forest as a classi er in our proposed system. The number of cross validation splits used in both cases are same which is a ve-fold cross validation. The identi cation of insincere questions have gained importance due to the increased number of CQA forums. Classi cation of insincere questions is quite easy using the text classi cation algorithms in NLP. This classi cation of insincere questions into six di erent classes is quite a new strategy. In this paper, we tried to implement the above mentioned classi cation using the fastText and decision tree algorithm for feature extraction and classi cation respectively. The proposed model proved to give the accuracy of 48.51% and F1-score(weighted) of 0.52 with the given test data.