This paper presents details of the work done by the team of IIT (BHU) Varanasi for the “Classification of Insincere Questions” track organized in the FIRE 2019. We implement bidirectional long short term memory using pre trained GloVe word embedding. Further, we analysed the results with diferent other embeddings. 1 https://expandedramblings.com/index.php/quora-statistics/
In today’s era, all the major websites on the internet are facing the challenge of providing appropriate content on their space. They are relying on the user of the website for marking unsafe content so that they can remove those contents and hence make the content of their website safe. Quora is one of the widely used websites on the internet with the user base of 300 million monthly users1. We can ask questions on any topic that afect us or the world; moreover, if we want to have an opinion of experts about any real-world incidents or how others would have tackled any particular situation. We can get interesting answers about all the stuf that we care about. The people are frequently using web forums like StackOverflow, Quora, and many more for getting answers to their information-seeking questions. However, with the vast user base, some people tend to ask questions with objectionable content. Sometimes, some questions are posted to target some specific group or spread hate speech. It is challenging for the human moderators to filter out insincere questions manually due to the vast number of questions. The task is about the identification of non-information seeking questions with the varying characteristics into one of the six categories:– Rhetorical questions – Sexual Content – Hate Speech – Hypothetical – Other – Not an Insincere question 3
Task organizers provided participants with the question id and label assigned to each question for 900 training instances. Due to dataset sharing constraints of kaggle, they could not provide the question text; hence the participants were supposed to extract the question text from the competition organized on the kaggle2 website using question id. We extracted the question text from the data section of the competition on kaggle and formed the training set. Each instance of the training set consists of the question id, question text, and label assigned to it. A sample of the training set is given below in table 1.
Figure 1 shows the distribution of training instances among diferent categories. It can be seen from figure 1 that there is an imbalance of training instances among diferent categories.
Data Preprocessing: We perform preprocessing by removing punctuation, ‘#’,‘@’ and ‘https’ symbols. We keep the stop words and hashtag words to get a better understanding of the context during training. Also, we removed numerals as with diferent contexts they play diferent roles in question texts; hence, we feel it is better to remove them. We also lowercased all the texts of the question. 2 https://www.kaggle.com/c/quora-insincere-questions-classification Algorithm 1: System Description
Feature Representation: We use pre-trained GloVe [
Model Description: We determine the maximum length of the sentence from
question texts of all training instances. We perform padding of the question text
in each training instance whose sentence length is less than the maximum length
of the question texts. We use bidirectional Long Short Term Memory[
In this section, we will discuss the experimental settings, results obtained with the model, and further analysis of the results.
Experimental Settings: We use Keras4 neural network library for training our
model which uses Tensorflow as backend. The model is trained for ten epochs
with a batch size of 32. We use a validation split of 0.3 to analyze the overfitting
using validation loss that may occur during training. Table 2 list out the values
for parameters and hyperparameters used for training the model.
Analysis: The task organizers shared the true labels of the test instances hence
we used diferent other word embeddings for further analysis. The accuracy
obtained using diferent embeddings with Bidirectional LSTM model is listed in
the table 3. Model M2 was submitted for the evaluation. We trained vocabulary
of the train set using word2vec[
We used a bidirectional Long short term memory model for classification of insincere questions on Quora. The system can be used for the automatic classification of insincere questions. We obtained an accuracy of 64.35% on the test set. We can incorporate linguistic features to improve the system.