Social media text holds information regarding various important aspects. Extraction of such information serves as the basis for the most preliminary task in Natural Language Processing called Entity extraction. The work is submitted as a part of Shared task on Code Mix Entity Extraction for Indian Languages(CMEE-IL) at Forum for Information Retrieval Evaluation (FIRE) 2016. Three different methodology is proposed in this paper for the task of entity extraction for code-mix data. Proposed systems include approaches based on the Embedding models and feature based model. Creation of trigram embedding and BIO tag formatting were done during feature extraction. Evaluation of the system is carried out using machine learning based classifier, SVM-Light. Overall accuracy through cross validation has proven that the proposed system is efficient in classifying unknown tokens too.
Entity extraction has always been the most primary task in Natural language processing. It is defined as a task of extracting the named entities from any text. Generally, entities fall under the categories of name, person, and organization. It extends to date, time, period, month etc. Entity extraction in social media text is viewed as an information extraction task. Social media text is generally unstructured, yet it is informative. Extracting such informative content from an unorganized text format is the most challenging task. In our task we deal with social media text, specifically code-mix twitter dataset. In a society using multilingual languages, conversation in code mixed language is prevalent. Code-mix language is the combination of English language with any other language. An example for code mixed language from the Hindi-English training data is given below.
shaq hai ki humari notice ke bagair tere ghar ke secret route ki help se you met
In the above example, Hindi words are in italics and English words are in bold. Communication through social networking sites likes Facebook and Twitter are in code-mix language. Dataset given for this task includes two subsets, where Indian languages like Hindi and Tamil were mixed with Roman script. The major task is to develop a method that is applicable to process and extract the entities of the data in code-mix language.
In recent years, a significant number of researches were
carried out in the field of data processing using code-mix
data. A language identification task was carried out for
code-mix social media data [
Our submission includes three systems. First system is
using word embedding features obtained from wang2vec tool
[
An overview of the task description is given in Section 2. Details regarding the dataset used is given in Section 3. Section 4 discusses on the proposed system we used for the task. Experiments carried out and their corresponding results are discussed in Section 5. The conclusion of the paper is stated in Section 6. 2.
The task organizers provided us with dataset obtained from Twitter and other few microblogs. Given training data contains two set of dataset with code mixed tweets - HindiEnglish and Tamil-English. The task is to extract the entities from these two dataset. Named entities in the dataset include Person, Location, Organization, Entertainment and so on. With the increasing number of social media platforms and the use of code-mix language in them, this task holds a significant relevance in today’s world.
The task contains two code-mix dataset, Hindi-English
and Tamil-English. The training data contains three
fieldsTweet ID, User ID and the tweets. Each training file holds
a corresponding annotation file. Annotation file contains
Tweet ID, User ID, Length, Index and Entity tag of the
entities present in the train data. Hindi-English training data
includes tweets entirely in code-mix language but
TamilEnglish dataset includes some tweets in pure Tamil
language. Since we proposed embedding based methodology,
we were in need of additional dataset to train our word
embedding model. The additional dataset for Hindi-English
were collected from Mixed Script Information Retrieval 2016
(MSIR) [
Our submission for the task of entity extraction in codemix language includes three systems.
• System 1: Wang2vec based embedding features • System 2: Word2vec based embedding features • System 3: Stylometric features Social media text is generally subjected to various preprocessing tasks. Given dataset includes Twitter data which is subjected to tokenization. Each token from this tokenized dataset is converted to conventional BIO format. This results in BIO tag information for each word in the training data. BIO tag is defined as Beginning, Inside, Outside tag of entities. For example, consider the sentence, “Pranab Mukherjee is the President of India”. In general, the entity ‘Pranab Mukherjee’ indicates PERSON and ‘India’ indicates LOCATION. Since Pranab Mukherjee has two parts, it is tagged as beginning and inside. Words other than entities are tagged as O i.e. Outside. So using BIO tag, the proposed system labels Pranab as B-PERSON, Mukherjee as I-PERSON and India as LOCATION. This BIO tag information is utilized in the three systems proposed in the paper.
Illustration of proposed feature based method is shown in Figure 1 and word embedding based models are shown in Figure 2. 4.1
Wang2vec model is the modified version of word2vec with an improvement in the structure of skip gram model. This modification made wang2vec better than word2vec. The major difference in these two embedding models is that the skip gram model in word2vec becomes Structured skip gram model in wang2vec. The significant modification in this model is the fact that the word order information is taken into consideration. Wang2vec features are the word vectors obtained using wang2vec model. The size of the vector n is fixed during the training of wang2vec. Thus each word in the training data holds a vector of size n, which is set as 50. The resultant vectors are the word embedding features of the given dataset. From these vectors, the left context and the right context features were extracted and appended to the original embedding features. This resulted in a feature set of size 150. Integrating the context features to the original features forms Trigram embedding features. Thus embedding features along with BIO tag information is integrated and given to train the SVM classifier. Hence a SVM model corresponding to system 1 is obtained. Similar procedure is followed for extraction of wang2vec embedding features for test data. These features are used to form the trigram embedding feature set, which is given to the SVM classifier for testing. 4.2
Word2vec model provides the vector representation for each word. Input for word2vec is sentences, as the major advantage of this model is that it provides vectors for each word based on the context. Vector representation for each word in the training data is obtained through Skip gram model in word2vec. These vectors are used to develop a entity extraction system. Similar to system 1, this system also includes trigram embedding feature set of word2vec embedding vectors.
Each word from the training data is combined with its corresponding BIO tag information and the trigram embedding feature set. This combined feature set is given for training machine learning based classifier, SVM-Light. After training using SVM model, the test data is appended with the trigram embedding features and is given for testing. SVM classifier uses the knowledge acquired from training data and performs recognition of entities in testing data. 4.3
Our third system is implemented using stylometric feature extraction. Features such as length, position, numbers, hash tag, punctuation are considered to be the stylometric features. The list of features used in our system is tabulated in Table 2. Stylometric features of each word in the training data is extracted. This feature set is integrated with BIO tag information and used for developing a SVM model. These features are also extracted for test data and given for testing the system.
The proposed methodology results with three SVM models for the three systems using wang2vec features, word2vec features and stylometric features.
Experiments for system 1 and 2 are similar in case of extracting word embedding features. Major difference beIrshad-IIT-Hyd Deepak-IIT-Patna
Amrita CEN NLP CEN Amrita Rupal-BITS Pilani
TEAM Deepak-IIT-Patna
Amrita CEN
NLP CEN Amrita Rupal-BITS Pilani-R2
CEN@Amrita tween these two features lies in the fact that system 1 will use wang2vec embedding features which is retrieved using structured skip gram model that takes the word order into consideration. System 2 will use word2vec features retrieved using skip gram model without the word order consideration. In order to train word embedding model, additional dataset is required. Input data for word embedding models i.e., word2vec and wang2vec will be the combination of training data and additional dataset. The size of vector to be generated is set to 50. From these 50 vectors, trigram embedding feature set of size 150 has been extracted. The training dataset is tokenized based on whitespace and is converted to BIO-formatted data. For each word in the training data, its BIO-tag along with the 150 embedding features are given as input to the SVM classifier. Trigram embedding feature set of test data is also extracted in the same manner. After tokenization of test data, the trigram embedding feature vectors of size 150 are given to classifier for testing.
System 3 implements stylometric feature extraction for code-mix data. Training data of Hindi-English and TamilEnglish are subjected to the preprocessing task, tokenization. For these tokenized words, features listed in Table 2 are extracted from training data. BIO tag information of these words is combined to the extracted features and thus forms stylometric feature set of training data. As far as testing is concerned, the tokenized words and its corresponding feature set is integrated and given to the classifier.
Cross validation results for Hindi-English and Tamil-English dataset using system 1, 2, 3 are tabulated in Table 3. The System 1 which uses wang2vec based features have shown better results in case of unknown tokens.
According to the results provided by CMEE-IL organizers, for Hindi-English we have acquired third place and for Tamil-English we have acquired second place. The Precision, Recall, F-measure of the top five teams for HindiEnglish and Tamil English is tabulated in Table 4 and Table 5 respectively.
The work is submitted as a part of Shared task on Code Mix Entity Extraction for Indian Languages in FIRE 2016. The use of native languages using Roman script in social media platforms is commonly seen today and extracting entities like person, location or organization from them is a challenging task. The task organizers provided us with data from Twitter and other few microblogs. Three systems were submitted for the task. The first two systems uses the word embedding features of word2vec and wang2vec for entity extraction task. The training data along with some additionally collected dataset were used for training the word embedding models. The third system uses only stylometric features for classification. The three systems were trained and tested using machine learning based classifier, Support Vector Machine. As future work, instead of SVM based classifier we are planning to use regression based methods. 7.
We would like to thank organizers of Forum for Information Retrieval Evaluation 2016 for organizing the task. We would also like to thank the organizers of the CMEE-IL task. 8.