Every language possesses plausible several interpretations. With the evolution of web, smart devices and social media it has become a challenging task to identify these syntactic or semantic ambiguities. In Natural Language Processing, two statements written using different words having same meaning is termed as paraphrasing. At FIRE 2016, we have worked upon the problem of detecting paraphrases for the given Shared Task DPIL (Detecting Paraphrases in Indian Languages) in Hindi Language specifically. This paper proposed a novel approach to identify if two statements are paraphrased or not using various machine learning algorithms like Random Forest, Support Vector Machine, Gradient Boosting and Gaussian Naïve Bayes on the given training data set of two subtasks. In cross validation experiments, Random Forest leads the other methods in terms of F1-score. The experimental results depicts that our algorithm gives better performance with the ensemble learning method than individual approaches for such classification problem. This can be used in various applications such as question-answering system, document clustering, machine translation, text summarization, plagiarism detection and many more.
With the plethora of information generated by the web these days,
it is challenging to understand the semantics of different
languages when each language has its own scripts and linguistic
rules. Hindi language is still in its early stage concerning to
natural language processing and applications [
The paper has been organized as follows. Section 2 gives the description of the training dataset provided by the organizing committee. Section 3 presents the proposed approach for paraphrase detection. The experimental evaluation has been carried out in Section 4. Section 5 concludes our research work followed by acknowledgment and references given in Section 6 and Section 7 respectively.
There were a total of 2500 data points in the training set with 2 classes P and NP to be used for classification in Subtask1, whereas Subtask 2 had 3 classes and a total of 3500 data points. A detail distribution of classes and its count for both the tasks is mentioned in Table1.
SubTask2 1500 1000 1000 Each data point contains a pID, a unique id for each data point and two Hindi sentences and their final tagged Class. The initial analysis of the data surfaced some noise, majorly in NP class. A few examples of false negatives and false positives have been identified and listed down in Table 2. Here the examples for class NP and class P denotes false negatives and false positives respectively. HIN2802 HIN2852 HIN3032 HIN0230 HIN2958 इस ककताब में
इस ककताब में काेंग्रस रविशंकर
ने कहा मुझे लगता है कक आईएसआईएस
The proposed approach includes three major steps which are text preprocessing, feature generation and classification model. Text preprocessing is done in various steps such as tokenization, stemming, soundex, stop word removal and handling synonyms. Feature generation involves the creation of five new features which will be used as an input to the classifier for classification of NP NP NP P paraphrases. Classification model includes the model training using the four machine learning algorithms. All the steps are described below.
For each data point, the preprocessing steps are as follows:
We have encoded the sentences using standard UTF-8 encoding that handles scripting of Hindi language.
We have tokenized the sentences into words using NLTK library.
We have applied custom set of rules for phonetics. The
normalization of phonetics has been done using soundex
algorithm [
ज
We have applied some more set of rules for stemming into its basic form. A set of Hindi suffixes characters were removed to get normalized Hindi word. For example, [ोो",ोे",ो",ोु",ोी",िो",ोा"][कर ",ोाओ",िोए",ोाई",ोाए",ने ",नी ",ना ", ते",ोीों",ती",ता",ोाो",ोाों",ोोो"ं,ोेों"] [ोाकर ",ोाइए",ोाईं ",ोाया",ोेगी",ोेगा",ोोग"ी,ोोग"े,ोाने ",ोाना ",ोात"े, ोाती",ोाता",ती"ं,ोाओ"ं,ोाए"ं,ोुओ"ं,ोुएं",ोुआं"]
We have removed irrelevant words from the sentences using a standard list of 164 Hindi Stop words. For example, [सारा , स,े सो, संग, ह , हुआ, हुई, हुए]
We have used Hindi WordNet, an extensive lexical dictionary of Hindi language having 40K~ synsets, to fetch synonyms for Hindi words. It was developed by researchers at the Center for Indian Language Technology, Computer Science and Engineering Department, IIT Bombay and we have downloaded it from the mentioned link in [5].
All the text preprocessing steps have been summarized with examples in the below Table 3.
After the preprocessing of sentences following features have been generated in the form of vectors to be passed as an input to the classifier.
The number of common tokens amongst two sentences is used as a feature. These tokens have been generated by comparing the preprocessed tokens after removing the stop words and then taking intersection of them symbolized as follows.
We have normalized the common tokens generated in the first feature to compute the proportion of commonality of tokens between two sentences. The value will be in the range of 0 and 1.It is 0 when there are no common tokens between two sentences and 1 if all tokens between the two sentences are exactly the same. Mathematically, It has been calculated by dividing the common tokens by number of unique tokens in both the sentences as shown below.
Sum of IDF scores of common tokens from two sentences is used as numeric similarity vector.
ΣIDF score (common tokens) IDF (Inverse Document Frequency) is defined as inverse of document frequency which is used to identify the importance of a token in a given corpus. Represented as: Here a document is an individual sentence. At first IDF has been calculated for all tokens and kept for reference. Then during feature generation process, IDF of common tokens has been calculated and is used as a feature.
Here the proportion of IDF score of common tokens between two sentences is computed for normalization and used as a vector to model. It gives us normalized common IDF score ranges between 0 to 1 of common tokens. It can be calculated as follows.
It denotes the count of number of tokens in sentence 1 and sentence 2 as separate columns.
Features generated in section 3.2 have been used as training data
to train the classifiers using Python Scikit library. Here, we have
used four popular machine learning algorithms which are random
forest, support vector machine and gradient boosting and
compared their performance. Since, random forest being an
ensemble learning method outperforms the other individual
methods, is implemented by growing many classification trees
and having them “vote” for a final decision according to a
majority role [
These are the number of trees that we want to build before having the vote for the final decision. More the number of trees better the performance however it also increases the time complexity.
It is the maximum depth of the tree which needs to be tuned.
These are the minimum number of samples or observations required in a terminal node of the tree.
These are the minimum number of samples or observations
needed in a node to be considered for splitting.[
We have used 10 fold cross validation to compute overall accuracy for the system. In this work three evaluation metrics have been considered which are precision, recall and f1-score. We have calculated the values of the evaluation parameters for all of the four respective algorithms. For the subtask 1 we have got the overall accuracy of 0.92 with F1 score of 0.94 maximum for Random Forest algorithm. Detailed performance matrix of the model is given as below in Table 4 in which we have to predict for 2 classes. Subtask 2 which had same problem with 3 classes to predict from. We have used similar approach and similar features set for training our model. With 3 classes and larger training set of 3500 data points we have got overall accuracy of 0.85 with 10 cross folds and a F1 score of 0.91 which is again maximum for Random forest algorithm. Detailed summary of performance matrix of subtask2 is given in Table 5. Following figures gives the summarized view of the performance of the various machine learning algorithms for both subtasks
In this paper we have proposed our novel approach for the
detection of Hindi paraphrases which is a very important building
block of semantic text analysis. Building a question answering
system, document clustering, knowledge extraction, plagiarism
detection, building ontologies etc. are the potential applications
for paraphrase identification in NLP [
We would like to thank organizers for conducting this shared task and also building the training data. We also would like to thank Sapient Corporation for giving us an opportunity to work and explore the world of text analytics.
[4] https://www.aclweb.org/aclwiki/index.php?title=Paraphras e_Identification_(State_of_the_art) [5] http://www.cfilt.iitb.ac.in/wordnet/webhwn/downloaderInf o.php