The paper discusses about our work submitted to the track IECSIL [3] organized by ARNEKT in conjunction with Forum for Information Retrieval and Evaluation 2018. The track primarily focuses on developing language independent system for information extraction on Indian languages. We focused on the identi cation and categorization of entities in the text. We used word embedding for the feature representation. We proposed Bidirectional LSTM recurrent neural network model for entity extraction from the provided text for the ve Indian languages such as Hindi, Kannada, Malayalam, Tamil and Telugu. The proposed technique is evaluated in terms of two metrics, accuracy and F1-score.
With the growing information and huge availability of structured and unstructured data, it is important to extract relevant information from the available text. The relevant information is further identi ed and useful keywords are extracted within the digital texts. The term which de nes the actual meaning of the sentence and acts as important nouns in the sentence are extracted for information. These nouns are further categorized to determine datenum, event, location, name, number, organization, occupation, things and other information. In the subsequent sections, we discuss objective of the task, statistics about corpus, present an overview of the framework, and analyze performance of the developed system. We conducted literature review for one of the fundamental task in the domain of Natural Language Processing. Named Entity Recognition can be performed broadly using three approaches namely rule-based, machine learning and deep learning based approaches. However, there is requirement of language expertise to develop rule-based techniques for entity extraction.
Asif et al. [
A survey about Named Entity Recognition systems for Indian and Non-Indian
Languages is done by Nita et al. [
There are two tasks scheduled in ARNEKT-IECSIL track for information extraction. First task is to identify and categorize entity within the conversational systems. Second task is to determine relation between the entities extracted by the rst task. We built the system for entity extraction only.
The corpus comprises ve Indian languages (Hindi, Kannada, Malayalam, Tamil
and Telugu) and is provided by the task organizers [
Word embedding is developed to capture information about the words in the
text corpus using Word2Vec [
Word representation discussed earlier is used as input to BiLSTM layer. Our proposed model uses BiLSTM to learn the contextual relationship between words from past and future context. Word representation obtained during feature representation stage is given as input to the model. BiLSTM layer uses 2 LSTM layers having number of units as 100 in each layer. One of the LSTM layer is connected in the forward direction which takes input in the same sequence while the other is connected in the backward direction which takes reverse copy of the input sequence. Output obtained by both the layers is concatenated to produce embedding for the input token. Further, we explore various activation functions and varying recurrent dropout at BiLSTM layer.
Dropout [
We used Keras1 neural network library to implement bidirectional Long ShortTerm Memory recurrent neural network which uses either Tensor ow2 or Theano3 as backend. The model is trained for 10 epochs with batch size of 32. Hyperparameters used in the model is shown in Table 2. 1 https://keras.io/ 2 https://www.tensor ow.org/ 3 http://deeplearning.net/software/theano/ Algorithm 1: Framework for Entity Extraction
Data: Train and Test data
Result: Predict type of entity for tokens in Test data 1 Read Train dataset and Test dataset 2 Generate nested list of words and tags of each sentence in Train dataset 3 Generate nested list of words of each sentence in Test dataset 4 Determine maximum length of sentence in Train and Test dataset 5 Generate list of unique tags and unique words 6 Represent unique tags using one hot encoding 7 Develop word embedding using nested list of words of Train dataset 8 Create dictionary with key as word if it exists and value as vector
representation of that word 9 Perform padding on Train and Test dataset based on maximum length obtained in Step 4 10 Train model comprises BiLSTM, Dropout and Dense layer 11 Predict tags for Test tokens
The evaluation of the proposed technique is carried on the Test-1 and Test-2 corpora provided by the task organizers. The evaluation is divided into two stages to help the participants to test the system built in real time. Pre-evaluation was performed on Test-1 corpora and nal-evaluation was performed on Test-2 corpora. Ranking is predicted based on the accuracy measure by taking average of accuracy obtained for all Indian languages.
Baseline system built on Naive Bayes Classi er was released by the task organizers during pre-evaluation stage. We achieved 5.45% better accuracy as compared to the baseline system for Test-1 corpora. Accuracy obtained during the pre-evaluation stage for di erent languages is shown in Table 3. Accuracy is calculated by comparing the result obtained by the proposed system with the labelled data provided by the organizers.
For analyzing the performance of the system on Test-2 corpora, task organizers evaluated the submissions on two metrics, accuracy and F1-score. We submitted three submissions for nal-evaluation on Test-2 corpora. Detailed statistics about the accuracy obtained in di erent submissions for di erent Indian languages is listed in Table 4. It is observed that Submission 3 performs better than other two submissions. Also, F1-score is calculated by the task organizers for Test-2 corpora. Performances of our 3 submissions are shown graphically for F1 metric in Figure 2, 3 and 4. We proposed a fully automatic system for entity extraction in Indian Languages namely Hindi, Kannada, Malayalam, Tamil and Telugu. We obtained 91.18% accuracy during pre-evaluation stage on Test-1 corpora and 90.94% accuracy during nal-evaluation stage on Test-2 corpora. The system can be improved either by incorporating language speci c features or probably by representing tokens at character level. 8
We would like to thank the organizers for giving us an opportunity to work on the challenging task, providing us the guidelines and support. 8