INTRODUCTION

Social media is vast source of information from which we can extract lots of important data as per the specific requirement. This paper presents a technique for named entity recognition from English and Hindi text data. Our main task is to extract name entity from social media tweets in Indian language (Hindi and English) and classify these tweets in named entity tags as people, location etc., which is around 22 classes to be tagged. We used machine learning algorithm CRF (Conditional Random Field) [5] to identify Named Entities in corpus. CRF algorithm is implemented using CRFSuite [5] tool. CRFsuite [5] is an implementation of Conditional Random Fields for labeling sequential data which provides Fast training and tagging, Linearchain CRF, etc. Supervised learning is used for training dataset. We have used this training dataset to train out system for tagging named entities. CRFsuite [5] generate model based on the supervised learning provided.

CONDITIONAL RANDOM FIELDS (CRFs)

Given Conditional Random Field is a type of discriminative probabilistic model used for the labeling sequential data such as natural language text. Conditional Random Fields (CRFs) is mainly used as a class of statistical modeling method which is applied in pattern recognition and machine learning. CRFs are undirected graphical models, a special case of which correspond to conditionally-trained finite state machines. In the special case in which the output nodes of the graphical model are linked by edges in a linear chain, CRFs [5] make first order markov assumption and can viewed as a conditionally trained probabilistic finite automata. CRFs model consists of F=<f1,…,fk>, a vector of feature functions, θ = <θ1,…,θk> a vector of weights for each feature function. Let O=<o1,…,ot> be an observed sentence. e e

METHODOLOGY

We use two different methods for identifying Named-Entity form given text. In one method we use Handcrafted or automatically generated rules for NER. In second method or approach we use machine learning technique for modeling. Also we have different machine learning technique i.e. supervise learning, semisupervised learning, unsupervised learning for modeling. Supervised learning gives best performance but it requires large amount of good quality annotated data. Unsupervised and semisupervised learning is used when there is scarcity of annotated data in training.

We have used Machine learning based approach to perform NER task for given data, because it is more efficient than rule-based approach and it is more frequently used.

Pre Processing

The given task requires prediction of named entities from social media, so first task is to tag the word from the whole sentence. Therefore we have to split into word by doing these we get 'The' 'brown' 'cat' for both English and Hindi. Next step is to give part of speech(POS) [2] to text here we have used RDR POS Tagger for both the languages which identifies noun, verb, adverb from the given text. We used genia tagger for chunking in English. Genia tagger tag words with relevant IOB chunking tag. For example:

"The brown cat" will get chunk tag as the: B-NP, brown: I-NP, cat: I-NP.

We were provided with NER tagged data for training by FIRE-2015. We prepared a file with tag word and its pos tag, chunk tag and NER tag for training purpose. For example: Location India NNP B-NP

Training

We have used the open-source tool, CRFsuite [5] which is one of the popular implementations of CRF (Conditional Random Fields) for training data and also for tagging test data. CRFsuite [5] internally generates features from attributes in a data set. In general, this is the most important process for machine-learning approaches because a feature design greatly affects the labeling accuracy.

Testing

The untagged test data are given for testing with its POS tag [2] and Chunk tag. POS tagging and chunk tagging is done with help of RDR POS [2] tagger and genia tagger. After that this untagged test data with its POS tag and chunk tag are given as input to our model to get test result.

Feature Set

Feature set which is used for CRF [5] based NER System which includes Prefix or Suffix of word, length of word, Capitalization, POS tag, Chunking etc. we created two different model for both Hindi and English using different feature sets.

Post Processing

CRFsuite [5] gives only NE tag as output. So we combined output with its named entity. Then we prepared output as given format in training file by adding relevant information like tweet_id, user_id, Index, length of word. For example:

Tweet ID:618698235092152320 User ID:2922444438 NETAG:LOCATION NE:india Index:122 Length:5

RESULTS

Evaluation

There are two standard measures used for evaluation of NE tagger. (I) Precision(P) is the measure of the number of entities correctly identified over the number of entities identified. (II) Recall(R) is the measure of number of entities identified correctly over actual number of entities. Both precision and recall are therefore based on an understanding and measure of relevance. Harmonic mean of precision and recall which is F measure is calculated.

Test Result

CONCLUSION

Conditional random field(CRF) [5] are better for Indian languages than other models like HMM, MEMM etc. NER learned using CRFs takes more time for training. As part of Speech (POS) and Chunking is part of training, incorrect tagging also reduce the accuracy of the Recognized Named Entity. For achieving high performance and accuracy of NER system more study and deeper understanding of linguistic features are required.