In this paper we describe the detailed information of NLP-NITMZ system on the participation of DPIL1 shared task at Forum for Information Retrieval Evaluation (FIRE 2016). The main aim of DPIL shared task is to detect paraphrases in Indian Languages. Paraphrase detection is an important part in the field of Information Retrieval, Document Summarization, Question Answering, Plagiarism Detection etc. In our approach, we used language independent feature-set to detect paraphrases in Indian languages. Features are mainly based on lexical based similarity. Our system's three features are: Jaccard Similarity, length normalized Edit Distance and Cosine Similarity. Finally, these feature-set are trained using Probabilistic Neural Network (PNN) to detect the paraphrases. With our feature-set, we achieved 88.13% average accuracy in Sub-Task 1 and 71.98% average accuracy in Sub-Task 2.
Ambiguity is one of major difficulties in Natural Language
Processing (NLP). In ambiguity, one text can be represented using
many forms like lexical and semantic. This is known as
paraphrasing. Here we consider only lexical level similarity for
paraphrase detection. Paraphrase detection is a very important and
challenging task in Information Retrieval, Question Answering,
Text Simplification, Plagiarism Detection, Text summarization
and even paraphrase detection on SMS [
This paper describes the NLP-NITMZ system which participated
in DPIL shared task [
Pair of Sentences പിുഞ്ചകുഞ്ുളങ്െ ളകൊന് യുവതി
വിഷം ളകൊ ടുുത് ആഹത്മതയ ളെ യ ുത. രുണ്ട മക്ളെ ശ്നേഷം യുവതി വിഷം ളകൊടുുത് ആത്മഹതയ
ളകൊ ളെ യ ുത. மும்பை குண்டுவெடிப்பு வழக்கில் மேலும் ஒருவர் கைது. பிரசெல்ஸ் குண்டுவெடிப்பு முக்கிய கு ற்றவாளி நஜீம் லாஷ்ராவி ஐ.எஸ் அ மை ப்பில் ஜெயிலராக இருந்தார். ਹੁਣ ਵਿਭਾਗ ਨੂੰ ਬਣਦਾ ਕਿਰਾਇਆ ਅਦਾ ਕਰਨ ਲਈ ਕੇਸ ਬਣਾ ਕੇ ਭੇਜ ਦਿੱਤਾ ਹੈ ਤੇ ਜਲਦ ਹੀ ਕਿਰਾਇਆ ਅਦਾ ਕਰਦਿੱਤਾ ਜਾਵੇਗਾ। ਹੁਣ ਵਿਭਾਗ ਨੂੰ ਬਣਦਾ ਕਿਰਾਇਆ ਅਦਾ ਕਰਨ ਲਈ ਕੇਸ ਬਣਾ ਕੇ ਭੇਜ ਦਿੱਤਾ ਹੈ| क्रिकेट के भगवान सचिन को जन्मदिन मुबारक हो, दीजिए बधाई| बधाई| के हुए सचिन तेंदुलकर जन्मदिन मुबारक हो, दीजिए Tag
P NP SP P Similarly in Sub-Task 2, the participants have to classify sentences into a three point scale i.e., three categories: Completely Equivalent (E), Roughly Equivalent (RE) and Not Equivalent (NE) i.e. (Paraphrase, Non-paraphrase, and Semi-paraphrase). Table 1 describes the examples of DPIL training dataset. probability of two sentences to be paraphrases is high when the edit distance of those two sentences is small.
( , ) = − In Section 2 we provide the detailed architecture of our system like feature-set and machine-learning technique. Section 3 describes the detailed statistics of test and training data which are used by our system. The result on test data is described in Section 4. Section 5 describes the conclusion and future work.
In this section, we elaborate our proposed architecture. As shown in Figure 1, our system NLP-NITMZ is based on three languageindependent features: Jaccard Similarity, Levenshtein Ratio and Cosine Similarity. To find the Jaccard Similarity, first we calculate the number of similar unigram between two texts. After that, similarity score is obtained by dividing the count by the total unigram of those two sentences. Next one is Levenshtein Ratio which calculates total number of operations required to change one string to another form. Final feature is Cosine Similarity where each word of sentences is represented using Vector Space model.
For machine learning portion we have used Probabilistic Neural Network to predict the class. Probabilistic Neural Network (PNN) is derived from Bayesian network. PNN is normally used in classification problem and it has 4 layers. Those layers are namely Input layer, Pattern layer, Summation layer and Output layer. The advantage of PNN is that, that are much faster than feed forward Neural Network.
Our system NLP-NITMZ used three types of features which are
Language Independent. We used lexical based features which are
mainly used to find the similarity between sentences for all
Languages [
The similarity and difference of two sets is calculated using Jaccard Similarity coefficient. For our task, Jaccard similarity coefficient between two sentences is the ratio between the numbers of unigram match to the total number of unique words in those two sentences. If S1 and S2 are two sets, then the Jaccard similarity is defined using following equation.
( , ) = ∩ ∪
The most common feature to compare two strings is the
Levenshtein Distance which is obtained by minimum number
of operations required (i.e. replacements, insertions, and
deletions) to convert one string to another [
Sentence 1 Sentence 2
Cosine similarity is another widely used feature to measure the similarity between two sentences. In this feature, each sentence is represented using word vectors. Here word vectors are mainly the frequency of words in the sentences. After that cosine similarity is calculated using the dot product of those two word vectors divided by the product of their lengths.
( , ) =
.
| || |
probabilistic neural network is illustrated in Figure 2. The PNN
[
A simple probabilistic density function (pdf) for class k is as follows where X = unknown (input), Xk = “Kth” sample, σ = smoothing parameter and p = length of vectors ( ) = ) . −|| − || The accuracy of PNN classification depends mainly on probability density function. The probability density function for single population is described using the following equation where n = no of samples in the population.
( ) = ( ) . = ∑ −|| − || If there are two classes i, j then classification criteria is decided using the following comparison: gi (X) > gj(X) for all j ≠ i ( The advantage of PNN networks is that the training process is easy and quick. They can be used in real time. For our experiment we used existing MATLAB toolkit to classify test data2.
DPIL shared task includes sentence pairs of four languages: Tamil, Malayalam, Hindi, and Punjabi. This shared task is divided into two sub-tasks. In Sub-Task 1, the main aim was to classify those four sentences as paraphrases (P) or not paraphrases (NP). Similarly Sub-Task 2 is to assign those sentences into three categories completely equivalent (E) or roughly equivalent (RE) or not equivalent (NE). Table 5 describes the details statistics of training and test dataset. 2 http://in.mathworks.com/help/nnet/ref/newpnn.html
Hindi
Hindi Malayalam Malayalam Punjabi Punjabi Tamil Tamil
Task 1 Task 2 Task 1 Task 2 Task 1 Task 2 Task 1 Task 2
In this paper, we presented our NLP-NITMZ system used for DPIL shared task. Overall, our approach looks promising, but needs some improvement. There are some disadvantages of PNN like: require large memory, slow execution. In future we want to overcome those problems using better machine learning approach and also want to implement semantic features for all languages to increase performance. We can also identify stop words of all four languages so that we can omit them from the corpus. Since our approach is based on language independent feature set so our methodology can be extended to various languages.
This work presented here is under the Research Project Grant No. YSS/2015/000988 and supported is by the Department of Science & Technology (DST) and Science and Engineering Research Board (SERB), Govt. of India. Authors are also acknowledges the Department of Computer Science & Engineering of National Institute of Technology Mizoram, India for providing infrastructural facilities and support.