This overview paper describes the rst shared task on Information Extractor for Conversational Systems in Indian Languages (IECSIL) which has been organized by FIRE 2018. Motivated by the need of Information Extractor, corpora has been developed to perform the Named Entity Recognition (Task A) and Relation Extraction (Task B) for ve Indian languages (Hindi, Tamil, Malayalam, Telugu and Kannada). Task A is to identify and classify the named entities to one of the many classes and Task B is to extract the relation among the entities present in the sentences. Altogether, nearly 100 submission of 10 di erent teams were evaluated. In this paper, we have given an overview of the approaches and also discussed the results that the participated teams have attained.
Applications of conversational systems and social media platforms have seen
increased adoption by Indian language users on account of local language enabled
keyboards and smart phones [
Understanding the above scenarios, Arnekt in collaboration with FIRE has come up with a track Arnekt-IECSIL - Information Extractor for Conversational Systems in Indian Languages (IECSIL). FIRE started of with the aim of building a South Asian counterpart for TREC, CLEF and NTCIR. FIRE has since evolved continuously to meet the new challenges in multilingual information access.4. Arnekt aims to power the world's smartest business solutions by providing state-of-the-art AI based Cognitive Intelligence as a Service (CIaaS)5. IECSIL basically involves ve Indian languages (Hindi, Kannada, Malayalam, Tamil and Telugu) to start with and is likely to be further extended to cover the major languages spoken in India (near future).
Resources for developing this prototype was collected using an automated and language independent framework which has been developed by Arnekt, that creates corpus for Named Entity Recognition (NER) and Relation Extraction (RE) (tasks in IECSIL) from DBpedia. Corpora contains tags of Named Entities and Relations for ve Indian languages (Kannada (kn), Malayalam (ml), Hindi (hi), Tamil (ta) and Telugu (te)) which are not just restricted towards creating a single application. An elaborated portion on steps taken for data creation and its statistics could be seen below in the coming sections.
Motivated by the need of Information Extractor described above, we have the following two tasks:
Corpora for ve Indian languages (Hindi, Tamil, Malayalam, Telugu and
Kannada) has been provided. Task A is to identify and classify the named entities
to one of the many classes [
NER Corpus Creation: The abstract and info-box property les from DBpedia are the resources for corpus creation. In preprocessing stage, info-box properties are extracted as a meta tags and the long abstract les are cleaned to remove the texts in foreign language, URL links, and other special symbols. The meta tags which are in non-English language has been translated into English through Google Translator. The meta tags that occurs more than 100 times across all the languages has been considered to create the nal entity and its corresponding text pairs. With this entity-text pair, the text in the cleaned abstract le has been tagged. There are totally nine tags (Date, Event, Location, Name, Number, Occupation, Organization, Other and Things) which are considered for the NER corpus creation.
Creation of meta tag to the entity list is the only manual processing involved in this framework and it takes very less time compared to the general manual annotation process. This corpus has been made available on-line6 to the research community through the Information Extractor for Conversational Systems for Indian Languages (IECSIL)7. The detailed NER corpus statistics has been given in Table 1:NER Corpus Statistics.
Continuation to Task A, corpora without named entities for ve Indian languages
(Hindi, Tamil, Malayalam, Telugu and Kannada) has been provided. Task B is
to extract the relation among-st the entities present in the sentences [
Relation Extraction Corpus Creation: Similar to NER, here also
relation tags are annotated through semi-automated methodology. Initially sentence
which has minimum NER tags count two has been taken and POS tagging is
applied on it. The tagger from the [
For evaluation, the classic Accuracy measure has been taken into consideration. It could simply be briefed as a predictive model that re ects the proportionate
number of times that the model is correct when applied to data. Evaluation has been computed in two stages,
Team participating in the shared tasks were encouraged to test their modules in real time8. They could feel free in submitting as many submissions as they prefer. The leader board is evaluated with approximately 20% of the data (Test-1 corpora). Test-1 corpora statistics are given in Table 3 and 4.
The nal ranking is based on another 20% (Test-2 corpora) of the data. Unlike the Pre-Evaluation, here the participants are requested to submit their models or code or submission le to task organizers. Test-2 corpora statistics are given in Table 3 and 4.
For each sub-task and language, submissions are evaluated by calculating the accuracy with the corresponding Gold labels. The accuracy scores across all the ve languages will be averaged to determine the nal ranking for both the sub-tasks.
(1) Acc = # terms correctly assigned to entity
total # terms A server similar to Kaggle/Coda Lab was hosted9 to check the developed system in real time, where participants submitted their test results for pre-evaluation corpora. Five days before the nal deadline Test 2 corpora for nal evaluation has been released. Participants were allowed to make at most 3 submissions against the Test 2 corpora. The nal ranking was then computed based on the participants system performance on Test 2 corpora. The results are described in Table 5, 6, 7 and 8.
The CUSAT TEAM have made use of deep learning in extracting the relation between entities. They have used Convolutional Neural Network (CNN), which has been modelled to address processing in sentence level for Malayalam language. Due to the absence of pre-trained word embedding for other languages
like Hindi, Kannada, Tamil and Telugu that ts in to their machine memory, they
have restricted their Relation extraction model development with Malayalam
language for which they have their own corpus to simulate word vectors. The
same team have used a statistical model in nding the entities from a given
sentence. CRF based sequence labelling model with features that are speci c to
Indian languages has been utilized in tagging the words with entities provided
[
SSN NLP have used Neural Machine Translation architecture to identify
and classify named entities for all the ve Indian languages that are in focus.
The deep neural network was built using multi-layer Recurrent Neural Network
(RNN) and Long Short Term Memory (LSTM). About four di erent models
were developed for each of the languages. It was found that bi-directional LSTM
with attention having eight layers of depth worked well for all languages other
than Malayalam [
SSN NLP have made use of the deep learning approach that they have
utilized for Named Entity Recognition (NER) for Relation Extraction as well.
While two models use the deep learning framework that use SeqtoSeq model,
three others were developed using statistical Machine Learning approach [
HiLT have used two-layer Convolutional Neural Network (CNN) for
character level (word-matrix) and word level encoding (sentence-matrix), along with a
Bidirectional Long Short Term Memory (Bi-LSTM) as a tag decoder for Named
Entity Recognition. This non-linear model has been developed as a language
independent framework with the aim of extending it to other Indian languages
other than the ve languages in focus. It is an added advantage that their model
does not seem to be biased for a particular language [
IIT(BHU) generated vector representation of words and their
corresponding tags, that were fed to the Bidirectional Long Short Term Memory (Bi-LSTM)
for identi cation and categorization of entities in the text. Word representation
has been done for all possible words in the corpus and a set of unique words were
represented using one-hot encoding. The BiLSTM layer here learns the
contextual relationship between words from past and future context. This team has
come up with a language independent framework for Named Entity Recognition
(NER) and has proven the same for the ve languages provided [
Khushleen has made use of character level information in order to include
word representation for rare words or out of vocabulary words from the given
corpora. The team has performed word embedding using fastText without changing
the parameters for each language for building a uni ed model. This is then fed
to a two-layer Bidirectional Long Short Term Memory (BiLSTM) for training
and prediction of entities for words in sentences [
Semantic relation among-st words were captured using word embedding as
done in Khushleen work using fastText by the Raiden11 team. As a next step
they have experimented this work using linear models like Naive Bayes and
Support Vector Machine. Apart from this they were able to prove that a simple
Arti cial Neural Network (ANN) model worked better than the former linear
classi ers, as it could capture the composite relation between words [
idrbt-team-a used a two stage LSTM based network with character based
emebeddings, word2vec embeddings and sequence based bi-LSTM embeddings
together to carry all the requisite features necessary for the NER prediction
problem [
In Relation Extraction, the team idrbt-team-a used features like POS tags,
NER tags along with the words in input text sentence to classify the given input
into one of the prede ned relationship class. By performing the initial experiment
with other statistical classi ers, Logistics Regression is chosen as the classi er
[
By using word embedding from fastText as a representation method, team
raiden11 have experimented the linear models like Naive Bayes and SVM, and
also a simple Neural Network to develop the NER system. The best results are
achieved for neural network for all languages combined [
Participants were mostly used deep learning based algorithms for both the Relation Extraction and Named Entity Recognition tasks. CNN, Bi-LSTM and CNN with Bi-LSTM are commonly used architectures. Participants yields 90 5 % as the accuracy for NER task. Even though the accuracy is high, it has to be noted that the accuracy obtained by selecting all entity as the class "other" is 80 5 %. This can be observed by measuring the performance of the team through f1 score.
Unlike NER, participated systems could not able to attain the best results. The above points shows the need of research in Indian Language based NER and Relation Extraction systems. The detailed results including the precision, recall and f1 score for target class and language is made publicly available 10. 4
Arnekt in collaboration with FIRE has come up with its rst track on Information Extraction for Conversational Systems in Indian Languages (IECSIL), which has utilized ve Indian languages (Hindi, Kannada, Malayalam, Tamil and Telugu) for identifying the entities (Task A : Named Entity Recognition) and also extracting relation from the same (Task B : Relation Extraction). IECSIL has developed its own corpora for both the tasks. While this corpus is not 10 https://github.com/BarathiGanesh-HB/ARNEKT-IECSIL/blob/master/IECSIL2018-Final-Evaluation-Results.xlsx restricted for a single application, it has been made available on-line11 to the research community through the Information Extractor for Conversational Systems for Indian Languages (IECSIL)12. The teams who have participated have come up with feasible solutions and most of them have utilized Deep learning methods to build their models. With the increase in need of Indian language usage, we are likely to extend the number of Indian languages used in the near future. 5
Arnekt thanks all the participants for showing their interest towards IECSIL. We would also like to show are gratitude to the FIRE 2018 organizers for their endless e orts and support. 11 https://github.com/BarathiGanesh-HB/ARNEKT-IECSIL 12 http://iecsil.arnekt.com