With the growth in our society, one of the most affected aspect of our routine life is language. We tend to mix our conversations in more than one language, often mixing up regional language with English language is a lot more common practice. This mixing of languages is referred as code mixing, where we mix different linguistic constituents such as phrases, proper nouns, morphemes etc. to come up code mixed script. With exponential growth of social media, we are using more and more code mixed cross script for our conversation on Facebook, WhatsApp, or Twitter. On the other hand, the language should be understood by the automated question answering system which is one of the most import application of AI. And now the trend is code mixed languages but current work is around a single language. At FIRE 2016, as a part of Shared Task1 CMCS (Code Mixed Cross Script Question Classification), we have worked on the problem of classify a code mixed question into 9 given classes. Shared Task is focused on Indian regional languages, wherein we worked on BengaliEnglish code mixed cross script questions classification. As scripting used in training data is English only, so all Bengali text was also written using English script only. We have used Machine Learning for question classification and used ensemble based Random Forest algorithm. As it's a code-mixed script, so traditional NLP components may not work well, so worked on a custom solution using own set of features for Classification.
According to Census of India of 2001, India has a total of 122
major languages and 1599 other languages [
Social media such as WhatsApp, Facebook[
But one of the next generation systems, automated
question answering systems, are still mostly catering one language
such as English or French. One of the first building blocks for any
question answering system is to understand and classify question.
A question could be of many types such as When, Where, What,
Why etc. So it’s important to classify question what kind of
answer is a question looking for. This shared task is about
classifying question in one of these classes. This paper describes
our approach for the subtask 1 in the shared task on Mixed Script
Information Retrieval [
A lot of similar work has been done in the past on mixed
languages and question classification on various sources. Khyati
and Manoj [
There were a total of 330 data points in the training set containing code mixed text of Bengali and English with 9 question classes to predict from. Each class describes type of answer it is expecting for subTask1. A detail distribution of classes and its count for CMCS task is mentioned in Table1. MNY
Count 26 61 26
.
PreProcess Tokenization bigrams
Replacements Each data point contains a code mixed question and a corresponding class in training data. Organizers have also provided a test dataset for evaluation purpose which had only code mixed questions for which our system have to predict class having a total of 181 questions.
As our solution is machine learning based, so we need to convert out text into some vectors to train our model. Before converting our text into vectors, we have applied some basic preprocessing on the raw code mixed text using NLP based custom pipeline.
We have used NLTK tokenizer on all questions to tokenize sentences into tokens.
Further, we have also generated bigrams of tokens, as bigrams are important to identify some important phrases to be used for feature generation such as koto run.
Also we analyzed that proper nouns entities such as person names, locations names were causing a lot of noise, so we have identified proper nouns and replaced them with XX.
We did not apply any stemming and stop words removal knowingly, as we analyzed and tested that stop words like koto, where, and who etc. are very important features for this classification task. So we kept all the terms in our vocabulary along with new set of bigram words.
A set of text features has been generated on word and bigrams level using text to vector conversions. Each feature is assigned a corresponding numeric value to train model. 4.2.1 Count Vectorizer Count vector simply counts the frequency of each token (unigrams and bigrams here) on Code Mixed questions corpus. This vector produces a sparse representation of the counts and number of features produced will be equal to the vocabulary size provided. As we did not filter out any word except replacement of proper nouns, so in total, 1030 features have been generated by this vector. Out of which as many as 667 terms had a frequency of 1, so we discarded those features and hence we are left with 363 features to be used. This is also called Term frequency and is denoted as below
tf(t,d) = ft,d
toiri koiti time bus xx te
run ache
r koto xx theke 0 50 100 150 4.2.2 TFIDF Vectorizer This vectorizer takes in all code mixed raw text and generate tf-idf of all the terms to get importance of all the words and bigrams. Tfidf is very useful feature generation model in text applications and is denoted as follows.
Where idf is inverse document frequency and is denoted as. We have only used terms having term frequency more than 1 and hence finally produced 163 features by this vectorizer having tfidf values for each term. A snapshot of features generated by tfidf vectorizer is mentioned in Figure 2. tuleche quarter
mela ghumiye
capital sobcheye masjid
sera prepaid
by distance
er koto 0 2 4 6 Overall we have generated ~500 features using both these vectors without removing any term as we found stop words relevant for this task.
Text vectors as generated in section 3.2 has been used as training data to train classifiers using Python Scikit. A number of different algorithms with different parameters have been tested before coming up with the best algorithm and its parameters. Support Vector Machines (SVM), Logistic Regression, Random Forest, Gradient Boosting have been tested using Grid Search to come up with best parameters and model. Finally, Random forest with 100 n_estimators, max_depth of 10, min_samples_leaf of 4 and min_samples_split of 4, were identified as the best set of parameters to be used with Random Forest. Overall training time for model is less than 1 second on quad-core Machine with 8GB of RAM.
We have used 10 fold cross validation to compute overall accuracy for the system. For the subtask 1 CMCS we have got overall accuracy of 0.8495 with F1 score of 0.84. Detailed performance matrix of the model is given as below in Table 3 for all the classes. 59 319 24 23 5 26 45 21 61 55 59 319 0.91 0.97 0.84 Recall 0.83 0.87
0 0.88 0.89 0.71 0.72 0.91 0.97 0.84 0.88 0.95 0.84 0.85 0.91
0 0.85 0.91 0.64 0.74 0.88 0.95 0.84
LOC
MNY NUM OBJ ORG PER TEMP avg total / / 0.85 0.93 0.83 0.87 0.95
0 0.82 0.93 0.58 0.76 0.85 0.93 0.83 24 23 5 26 45 21
In this paper, we have presented our approach for question
classification on code mixing. Question classification is a first
step towards building a question answering system. Moreover
work which has been done in code mixed languages is mostly in
Hindi-English [