This paper presents the working methodology and results on Code Mix Entity Extraction in Indian Languages (CMEE-IL) shared the task of FIRE-2016. The aim of the task is to identify various entities such as a person, organization, movie and location names in a given code-mixed tweets. The tweets in code mix are written in English mixed with Hindi or Tamil. In this work, Entity Extraction system is implemented for both Hindi-English and Tamil-English code-mix tweets. The system employs context based character embedding features to train Support Vector Machine (SVM) classifier. The training data was tokenized such that each line containing a single word. These words were further split into characters. Embedding vectors of these characters are appended with the I-O-B tags and used for training the system. During the testing phase, we use context embedding features to predict the entity tags for characters in test data. We observed that the cross-validation accuracy using character embedding gave better results for Hindi-English twitter dataset compare to TamilEnglish twitter dataset.
• Information Retrieval ➝ Retrieval tasks and goals ➝
The explosion of social networking site like Facebook, Twitter and Instagram in a linguistic diverse country like India impacted many users using multiple languages in tweets, personal blogs etc. The scarcity of proper input tools for Indian Languages forced the user to use the roman script mixed with the native script that led to the code mixed language. This code-mixing further complicates the application of Natural Language Processing (NLP) methods. One such application is Named Entity Recognition (NER), which identifies and classifies entities into categories such as a person, organization, date-time etc. The NER is used in text mining application, information retrieval, and machine translation etc. NER task for Indian languages is a complex task. The use of code-mixing further complicates this task.
The shared task on Code Mix Entity Extraction in Indian Languages (CMEE-IL)1 held at Forum for Information Retrieval (2016) focuses on extracting entities from the code-mix twitter data.
Many works have done on Entity Recognition using word
embedding as a feature. Few research works are done on Entity
extraction using Character Embedding technique such as Entity
recognition using character embedding by Cicero dos Santos et al.
[
We have submitted a system based on the character embedding a
feature representation. To retrieve character embedding features,
the word2vec model is used [
An outline of the task is given in Section 2. Implemented system is described in Section 3. Section 4 describes Experiment and result. The conclusion is discussed in Section 5.
Hindi-English and Tamil-English code-mixed language is given as training data, to implement the shared task on CMEE-IL. The task is to extract the named entities along with their named entity tags. The training and testing dataset had raw tweets and annotations in different files for code-mixed Hindi-English and Tamil-English language. There were 18 types of entities present in the training dataset in which entities like a person, location, and entertainment were more in number compare to other entity types. The number of tweets and number of tokens in training, as well as testing corpus for both code-mixed languages, are shown in Table 1. @_madhumitha_ next time im in chennaidef going over like \"hello ennakuongapethitheriyum\" @thoatta நான் ச ால் ல வரது ஸடீர் ரிடரலர் நல் லா இருக்னகு ச ான் ன படங் கள எல் லாம் சவற்றி அரடயல.over expectations nala flop airumnu STD ச ால் லுத @aixo_anjum sister me aesy bkwas nhi krta na muje shoq he I liked your pinned tweet at that time too just wait I shall give you proof Named Entity (NE) type, Named Entity String, Start Index and Length of the string. The participants were asked to submit a similar annotation file after extraction all the entity related information from test data file.
In the proposed system, feature extraction plays a vital role as the accuracy of the system relies majorly on the extracted features. Before extracting any features, the dataset should be in an appropriate format for the learning algorithm. The I-O-B tagging format was used for this format conversion. The first step in format conversion was to tokenize the given data and arrange each word (token) per line. To identify start and end of a tweet we added a tag <S> to make it easy for further processing. The proposed system is based on character based context embedding. The basic intuition behind the character based context embedding is shown in Figure 1. In the figure, the Word(W0) (LIKE2) is split into corresponding characters. Each character is represented by a 50 − dimensional vector (vector length is user defined). These character embeddings are further concatenated to form a word vector. This unique vector is appended with the corresponding word. This 50-dimensional representation is converted to 150-dimenstion by appending the neighboring context words. W-1 represents previous context word of the word W0 and W+1 represent the next context word of the word W0. Appending the vectors of W-1, W0 and W+1 are called context appending.
The overall methodology is split into two modules; each module is described in separate figures. Figure 2 shows the method 2 The capitalization of character is just for representation purpose. The given raw text file contains Tweet-ID, User-ID and corresponding tweet. The annotation file was arranged in columns, with each column representing Tweet-ID, User-ID, Further, we analyzed the corpus for the counts of major entities. The count of major entities with the tags in the training corpus can be seen in Table 3. It can be observed from Table 3 that entities related to Entertainment tags were the most occurring entities. followed for feature representation. Figure 3 shows steps involved in the classification task. extracted from the output file are converted to required annotated format.
The first module depicted in Figure 2 explains step involved in
feature representation. We collected additional datasets from the
different data source to improve the accuracy of the system. The
additional dataset for Tamil-English was collected from Sentiment
Analysis in Indian Languages (SAIL-2015) [
This huge file comprising of training as well as additional dataset collected was then subjected for tokenization. These tokenized words are further split into corresponding characters. These characters are fed to vector embedding module. The output of the module will have a matrix of length 1 × 50 for every character in our corpus. These vectors are concatenated back to words using an appending module. The embedding features for each word are further subjected to context appending, resulting in a vector of length 1 × 150. These vectors contain the information about the word as well as neighboring word contexts. Format conversion involves raw tweets from the given training data set and annotated file is given to the I-O-B module. I-O-B converts given training data into the I-O-B format. Here I-O-B formatted training set consists of training data along with corresponding I-O-B tags. This formatted training set is appended with the feature vector to form training data that has to be fed to the classifier.
Figure 3 describes procedure involved in classification. During the training phase, the classification model is built based on the feature and label pair. During the testing phase, test data along with its embedding features are fed to predict entity tags for testing data. In Figure 3, Context vectors represent features and label pairs that are fed to SVM module. Here we used SVM Light tool to build anSVM-based classifier model. The classifier takes features and label as an input and trains itself finally builds a classification model. From the learned classification model, the system predicts the entity tags for testing data. Predicted tags
This paper describes work of entity extraction system for codemix twitter data using character embedding technique. In character embedding approach words in raw code-mix training data is tokenized into one word per line fashion and further split into characters. This character is embedded with the n-dimension vectors to produce feature vectors. Feature vectors appended with the I-O-B tags to represent feature - label sets. This set is fed to Support Vector Machine (SVM) classifier to train a classification model. During testing phase test data undergoes character embedding to represent the test data as a feature vector. These feature vectors are fed to classification model that we already trained. Classification model predicts the entity tags for the test data set. The output of the test data set is converted to a suitable form to represent the Annotated format. Annotated format contains Tweet ID, User- ID; word its corresponding predicted tags, Index and Length. Table 3 shows Cross Validation result for both Hindi-English and Tamil-English datasets. The overall accuracy of character embedding for the Hindi-English system is 95.996%. For Tamil-English datasets overall accuracy of character embedding is 94.3451%. 2 3
32.83
20.94 Table 5 and 6 displays result participated in a shared task. Implemented system ranked eighth in Hindi-English twitter dataset with Precision 48.17 and F-measure 32.83. For TamilEnglish our system was ranked fifth with Precision 47.62 and Fmeasure 20.94.
The proposed system for code-mix entity extraction is submitted as a part shared task on code-mix entity extraction system in Indian languages (CMEE-IL) conducted by FIRE 2016. Task involves extracting entity from the code-mix tweets. Given dataset consist of Hindi-English and Tamil-English code-mix tweets. In our work, we implemented character embedding system. We conclude that overall accuracy of Character embedding for HindiEnglish is better compared to the Tamil-English.
Few errors that resulted in decrease performance of the system are
We would like to give thanks to the task organizer - Forum for Information Retrieval Evaluation. We also thank organizers of CMEE-IL task.