In this work, we describe a system that detects paraphrases in Indian Languages as part of our participation in the shared Task on detecting paraphrases in Indian Languages (DPIL) organized by Forum for Information Retrieval Evaluation (FIRE) in 2016. Our paraphrase detection method uses a multinomial logistic regression model trained with a variety of features which are basically lexical and semantic level similarities between two sentences in a pair. The performance of the system has been evaluated against the test set released for the FIRE 2016 shared task on DPIL. Our systemachieves the highest f-measure of 0.95on task1 in Punjabi language.The performance of our system ontask1 in Hindi language is f-measure of 0.90. Out of 11 teams participated in the shared task, only four teams participated in all four languages, Hindi, Punjabi, Malayalam and Tamil, but the remaining 7 teams participated in one of the four languages. We also participated in task1 and task2 both for all four Indian Languages. The overall average performance of our system including task1 and task2 overall four languages is F1-score of 0.81 which is the second highest score among the four systems thatparticipated in all four languages.
The concept of paraphrasing is defined in [
The task of paraphrasing can be of two types based on its
applications: paraphrase generation and paraphrase recognition. In
broader context, paraphrase generation has various applications.
One of the most common applications of paraphrasingis the
automatic generation of query variants for submission to
information retrieval systems Culicover(1968)[
Ravichandran and Hovy (2002)[
Like paraphrase generation, paraphrase recognition is also an
important task which is to assign a quantitativemeasurement to the
semantic similarity of two phrases [
Detecting redundancy is a very important issuefor a
multidocument summarization system because two sentences from
different documents may convey the same semanticcontent and to
make summary more informative, the redundant sentences should
not be selected in the summary. Barzilay and McKeown
(2005)[
Another recently proposed natural language processing task is that
of recognizingtextual entailment: A piece of text T is said to
entail a hypothesis H if humans readingT will infer that H is most
likely true [
One of the important requirements for initiating research in paraphrase detection is creation of annotated corpus. The most commonly used corpora for paraphrase detection is the MSRP corpus1 which contains 5,801 English sentence pairs from news articles manually labelled with 67% paraphrases and 33% nonparaphrases. The shared task on Semantic Textual Similarity conducted as a part of SemEval-20122 was targeted to create benchmark datasets for the similar kind of task, but its main focus was to develop systems that can examine the degree of semantic equivalence between two sentences unlike paraphrase detectionwhich determines yes/no decision for given pair of sentences.
Howeverthere are at present no annotated corpora or automated semantic interpretation systems available for Indian languages. So creating benchmark data for paraphrases is necessary. With this motivation, creating annotated corpora for paraphrase detection and utilizing that data in open shared task competitions is a commendable effort which will motivate the research community for further research in Indian languages.On this note, the shared task on detecting Paraphrases in Indian Languages (DPIL)@FIRE 2016 is a good effort towards creating benchmark data for paraphrases in Indian Languages. In this shared task, there were two sub-tasks: task1 is to classify a given pair of sentences in Punjabi language as paraphrases (P) or not paraphrases (NP) and task2 is to identify whether a given pair of sentences are completely equivalent (E) or roughly equivalent (RE) or not equivalent (NE). Four Indian Languages –Hindi, Punjabi, Malayalam and Tamil were considered in this shared task. We describe in the subsequent sections our proposed methodology used to implement our system participated in the shared task and we also present performance comparisons of our system with other systems participated in the competition.
We view the paraphrase detection problem as classification problem. Given a pair of sentences, the task1 is to classify whether the pair of sentences is a paraphrase (P) or not – paraphrase (NP). When task1 is a two class problem, task2 is a three class problem. The task2 is to classify a given pair of sentences into one of three categories: completely equivalent (E) or roughly equivalent (RE) or not equivalent (NE).
Since the problems are basically a classification problem, we have used a traditional classifier for implementing our system. We have used multinomial logistic regression classifier with ridge estimator for both task1 and task2.Each pair of sentences is considered as a training instance. Features are extracted from the training pairs. We consider a number of features for representing sentence pairs. The features which we have used for implementing our system are described in the subsequent subsections:
We have used various similarity measures as the features. 1https://www.microsoft.com/en
us/download/confirmation.aspx?id=52398 2https://www.cs.york.ac.uk/semeval-2012/task6/index.html , , To compute cosine similarity, we represent each sentence in a pair using a bag-of-words model. Then cosine similarity is computed between two vectors where each vector corresponds to a sentence in a pair. Basically we consider the set of distinct words in the pair as the vector of features based on which the cosine similarity between two sentences is computed. The size of the vector is n where n is |S1US2| , S1 is the set of words in the sentence 1 and S2 is the set of words in sentence 2.. Each sentence in a pair is mapped to vector of length n. If the vector for sentence 1 is <v1, v2 …..vn> and the vector for sentence 2 is <u1, u2 …un>, where vi and ui are the values of i-th word feature in sentence 1 and sentence 2 respectively, the cosine similarity between two vectors is computed as follows: = , = +
+⋯ √ + +...
√ + +...
(1) Here the vector component vi in vector V corresponds to value of the i-th word feature which is basically the TF*IDF weight of the corresponding word. Similarly vector U is also constructed for the sentence 2.
We also used the word overlap measure as a feature for paraphrase detection. If two sentences in the pair are S1 and S2, the similarity based on word overlap is computed as follows: | ∩ | = | |+| | (2) Where | ∩ S | is the number of words common between two sentences.and|S| is the length of sentence S in terms of words.
Since the most Indian languages are highly inflectional, stemming is an essential step while comparing words. Accurate stemmers are also not available for Indian languages. So, we applied a lightweight approach to stemming. In this approach, when we match two words, we find the unmatched portions of two words. If we find that the matched portion of two words is greater than or equal to a threshold T1 and the minimum of unmatched portions of word1 and word2 is less than or equal to a threshold T2, we assume that there exists a match between word1 and word2. Stemmed Word overlap is computed using equation (2) with the only difference in word matching criteria. We set T1 to 3 and T2 to 2. We indicate such similarity between two sentences S1 and S2as , .
The similarity measures mentioned above compares sentences based on individual word matching. But bag-of-words model does not take into account the context of occurrences of words. We consider n-gram based sentence similarity as one of the features for paraphrase detection.
We compute n-gram based similarity as follows: Where , = + (3) = # = # − ℎ a=# − We have only considered bigrams(n=2) for implementing our present system.
We have used semantic similarity between two sentences as one
of the features for paraphrase detection. To compute semantic
similarity between sentences, we calculate whether words in the
sentences are semantically similar or not. To determine whether
two words are semantically similar or not, we have cosine
similarity between word vectors for the words. The vector
representations of words learned by word2vec models[
Such representation of words into vectors positions the word in the vector space such that words that share common contexts are positioned in close proximity to one another in the space We have used word2vec model available in Python for computing word vectors for the words. We have used gensim word2vec model under Python platform with dimension set to 50, min_countto 5(ignore all words with total frequency lower than this). The training algorithm used for developing word2vec model is CBOW (Continuous Bag of words). The other parameters of word2vec model are set to default values. If the cosine similarity between the word vectors for the two words is greater than a threshold value, we consider these two words are semantically similar. We set the threshold value to 0.8. We combine a small amount of additional news data with the training data for each language to create the corpus used for computing word vectors. Size of the corpora used to compute word vectors for the different languagesis as follows: For Hindi, 1.93 MB(8752 sentences), for Punjabi, 1.5 MB(5848 sentences), for Tamil, 2.20 MB (7847 sentences) and for Malayalam, 2.12 MB (7448 sentences) We compute semantic similarity between two sentences as follows: (4) where
= # f=# = # , = + ℎ
We have used multinomial logistic regression as the classifier for
paraphrase detection task. We view the paraphrase detection
problem as a pattern classification problem where each pair of
sentences under consideration of paraphrase checking is mapped
to a pattern vector based on the features discussed in section 2.1.
We have chosen multinomial logistic regression classifier from
WEKA. This is present in WEKA with the name “logistic”.
WEKA is machine learning workbench consists of many machine
learning algorithms for data mining tasks [
We set the “ridge” parameter to 0.4 for all our experiments. The other parameters of the classifiers are set to default values.
We have obtained the datasets from the organizers of the shared task on detecting paraphrases in Indian Languages (DPIL) held in conjunction with FIRE 2016 @ ISI – Kolkata. The datasets released for four Indian languages-(1) Hindi, (2) Punjabi, (3) Tamil and (4) Malayalam. For each language, two paraphrase detection tasks were defined: Task1- to classify a given pair of sentences in Punjabi language as paraphrases (P) or not paraphrases (NP) and Task2- to identify whether a given pair of sentences are completely equivalent (E) or roughly equivalent (RE) or not equivalent (NE). The training data set for task1 contains a collection of sentence pairs labelled as P (paraphrase) or NP (not a paraphrase) and the training dataset for task2 contains a collection of sentence pairs labelled as completely equivalent (E) or roughly equivalent (RE) or not equivalent (NE). The description of the datasets is shown in Table 1 and Table 2.
For system development, we have used training data [
Malayala m Punjabi Tamil
Malayala m Punjabi Tamil
Hindi As we can see from the tables, only 4 teams out of 11 participated teams submitted their systems for all four languages- Hindi, Punjabi, Malayalam and Tamil and the remaining 7 teams participated in only one of the four languages.
We have shown in the tables in bold font the performance scores highest in a particular task for a particular language. It is also evident from the tables that most systems perform well on Punjabi and Hindi languages, but they show relatively poor performance in Tamil and Malayalam languages. We think that the main reason for achieving the better performances inPunjabi and Hindi language domain is the nature of training and testing data sets supplied for those languages. Most likely, that is why most systems perform almost equally well on the Punjabi and Hindi languages. Another reason for having poor performance on Tamil and Malayalam may be the complex morphology of these languages.
We have computed the relative rank order of the participating teams based on overall average performance on task1 and task2 in all four languages (simple average of F1-scores obtained by a team on task1 and task2 over all four languages). Since only four teams have participated in all four languages, we have only shown rank order of these four teams in Table 5.As we can see from Table 5, our system (Team code: KS_JU) obtains second best accuracy among the four systems which participated in all four languages. KS_JU
Overall Average F1-Score 0.84 0.81 0.78 0.53
In this work, we implement a paraphrase detection system that can detect paraphrases in four Indian Languages-Hindi, Punjabi, Tamil and Malayalam. We use various lexical and semantic level similarity measures for computing features for paraphrase detection task. We view paraphrase detection problem as a classification problem and use multinomial logistic regression model as a classifier. Our model performs relatively better on task1 than on task2.
Our system has the scope for further improvement in the following ways:
Word2Vec models requires large corpus for proper representation of word meaning, but for our present implementation, we have used a relatively small size corpus for computing word vectors. Use of large corpus for computing word vectors may improve semantic similarity measure leading to improving system performance.
Since we have only used multinomial logistic regression model as the classifier, there is also the scope to improve the system performance using other classifiers or combination of classifiers.
Most Indian languages are highly inflectional. So, use of morphological analyzer/stemmer/lemmatizer may improve the system performance. 5. ACKNOWLEDGMENTS This research work has received support from the project entitled ‘‘Design and Development of a System for Querying, Clustering and Summarization for Bengali’’ funded by the Department of Science and Technology, Government of India under the SERB scheme.