-
Named Entity Recognition (NER) is the process of identifying the proper names
in text and classifying them as one of a set of prede ned categories of interest.
There are three universally accepted categories which are the names of locations,
people and organisations. There are other common categories such as
recognition of time/date expressions, measures (money, percent, weight etc.), email
addresses etc. In addition, there can be domain-speci c categories such as the
names of medical conditions, drugs, bibliographic references, names of ships, etc.
NER is useful for applications such as question answering, information retrieval,
information extraction, automatic summarization, machine translation and text
mining [
1
].
Arabic is one of the ve o cial languages used by the United Nations.
Approximately 360 million people speak Arabic in more than 25 countries and
Copyright © 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
Arabic script represents 8.9% of the world's languages [
2
]. Although there is
existing work on Arabic NER, it still in the primary stage compared with
English NER [
2
]. Certain characteristics of the Arabic language o er challenges for
the task of NER. Unlike English and other European languages, capitalization
does not exist in Arabic script. Thus, employing capitalization as a feature in
Arabic NER is not an option. However, translation to English is one way to
solve this problem [
3
]. The Arabic language is morphologically complicated, a
word may consist of pre xes, lemma and su xes in di erent combinations [
4
].
That can a ect the performance in Arabic NER as typically features derived
from the su x and a x of the words are used. Also, spelling alternates can be a
challenge in Arabic NER. In the Arabic language, words (including named
entities) may be spelt in di erent ways but have the same exact meaning generating
a many-to-one ambiguity [
2
]. The lack of resources in Arabic presents another
challenge for Arabic NER. There is a lack of the freely available Arabic datasets
and gazetteers as many of the available ones are not appropriate for Arabic NER
tasks because of the absence of NEs annotations.
In this paper we explore approaches for NER on Arabic text to determine
how the state of the art approaches to NER work on the Arabic language. We
investigate the impact of using di erent classi ers and sets of features including
both language-independent and language-speci c features, testing them on ve
di erent datasets. We have taken English as the second source language in our
work because English NER is the most developed among other NER models.
Recently, research on English NER have achieved the best performance in the
eld and represents the state of the art. We also compare against the more recent
deep neural network approaches. The neural network approaches were found to
perform better than the traditional machine learning approaches for both Arabic
and English NER. However the SVM classi er outperformed the neural network
based model on one dataset (AQMAR). Our proposed models for the Arabic
NER outperformed other's proposed models on two Arabic datasets out of three.
The rest of this paper is organized as follows. Related work is discussed in
section 2; the datasets and proposed models are presented in the methodology
section 3; experimental results and analysis in section 4 and nally the
conclusions are discussed in section 5.
2
2.1
Related Work
General NER
There are three main approaches for the NER task: rule-based, machine-learning
and hybrid approaches. Early NER approaches were rule-based using
handcrafted rules. In rule based approaches, the rules are designed as regular
expressions for pattern matching generally with a list of lookup gazetteers [
4
].
Rule based approaches require expert linguists to design rules for the NER task
and usually target a single language. Therefore, few researchers use rule-based
systems to develop NER systems [
5
]. Although the knowledge-based approach
can achieve good results, it requires a very exhaustive lexicon in order to work
well. That resulting in ine ciency as entities that don't exist in the lexicon
cannot be recognised [
6
].
There are common classi ers used for NER task such as Conditional
Random Fields (CRF), Support Vector Machines (SVM), Maximum Entropy (ME),
Decision Trees and Hidden Markov Models (HMM).An important factor in the
machine learning based approach is the features that are used. There are some
features that have been often used in NER systems such as the case of the word,
upper or lower, whether the entity is a digit or contains a digit, and the part
of speech associated with a word. The digit feature is useful in NER as it can
be used to recognize dates, percentages, money, etc., [
7
]. The morphology of a
word can be captured by including pre xes or su xes as features. For example,
a word can be recognized as an organization if it ended with "tech", "ex" or
"soft" [
8
]. To extract features a window is typically passed over the text. An
example of using window feature was proposed by [
9
] where the part-of-speech
of two words before the current word and two words after was used to recognize
the named entities. Word length (number of characters) has also found to be an
e cient feature for NER task [
10
].
The third approach to NER, the hybrid approach, which combines both
rulebased and machine learning to optimize the system performance [
11
], In this
approach, the output of the rule based system as tagged text is used as input to
the machine learning system).
Most of the more recent proposed NER systems are based on recurrent neural
networks (RNN) architecture over characters or word embeddings [
12
]. Those
features (word embeddings) are representations of words in n-dimensional space
using unsupervised learning over large collections of unlabeled data. The rst
neural network based approach for NER was proposed by [
13
]. The system used
feature vectors created from orthographic features (e.g., capitalization of the
rst character), lexicons and dictionaries. Later they replaced these manually
created feature vectors with word embeddings. Since then, and starting with [
14
],
implementing neural networks for NER systems have become popular. These
kind of models are attractive because they do not require feature engineering
e orts, and are thus more domain independent. Current research has shown
using pre-trained word embeddings is important for neural network based NER
because they are more e ective and less time and resource consuming [
15
]. Also,
pre-trained character embeddings is essential for character-based languages such
as Chinese (one Chinese character may represent a word meaning) [
16
].
2.2
Arabic NER
A number of research studies have focused on Arabic Named Entity
Recognition ANER. An early attempt for Arabic NER was proposed by [
7
] where they
used a rule-based approach. Their approach consists of a whitelist
representing a dictionary of names, and grammar in the form of regular expressions to
recognize the named entities. A machine learning-based approach was proposed
by [
18
] where they developed an Arabic NER system named ANERsys 1.0.
Linguistic resources have been built by the authors for their experiments including
ANERCorp, the rst freely available manually annotated Arabic NER dataset
and ANERgazet, an Arabic gazetteer. Contextual and gazetteer features were
used in the rst version and then part-of-speech features were added in the
second version which improved the system performance. A hybrid approach which
combines rule-based and machine learning for Arabic NER was proposed by [
7
].
They used the GATE toolkit 1 for the rule-based approach. The ML-based
component used a Decision Tree algorithm. The system used NE tags produced by
the rule-based approach besides other language independent features and Arabic
speci c features.
The missing capitalization feature in the Arabic language is compensated for
in some Arabic NER work by using an Arabic morphological analyzer named
Buckwalter [
33
]. Among those features provided by Buckwalter is a feature
named English-gloss which provides the English translation for each word in the
input Arabic text. Later a tool named MADA was built on Buckwalter and
upgraded to be named MADAMIRA [
38
]. It provides up to 19 orthogonal features.
We used some of those features in our designated models which were proven to
be e cient in Arabic NER models [
38
]. More details of the implemented features
produced by MADAMIRA are in the features section.
Similar to English, recent work in Arabic NER focuses on developing neural
network based approaches. A neural network based approach for Arabic NER
employing Bi-LSTM and CRF to predict the named entities has been used [
17
].
However, their model is missing some techniques such as character
representations and hyper parameters tuning. Another approach proposed by [
40
] used
an LSTM neural network model combined with a CNN for character-level
features representation. Their model is well designed but is also missing the
hyperparameter tuning technique to boost the performance. Also, a new e cient
multi-attention technique has been used [
41
] which uses a combination of word
embeddings and character embeddings via an embedding-level attention
mechanism. The output is fed into an encoder unit with Bi-LSTM, followed by
another self-attention layer to boost the performance.They evaluated their model
on ACE and ANERCorp and Twitter datasets. Their model achieved relatively
better performance on the ACE dataset which has a di erent tagging style (not
CoNLL-2003 tagging style) and relatively lower performance on Twitter dataset
and that is probably due to the noisy text. Their model evaluation is very
similar to our neural network based model with a slight inprovement in our results
where we are using di erent hyperparameter values.
Model learning as well as evaluation requires high quality annotated datasets.
Initial benchmark datasets were generally created by labeling news articles with a
small number of entity types, e.g. CoNLL-2003 [
39
] and ANERCorp dataset [
23
].
Later, more datasets were created on numerous kinds of text sources including
conversation, Wikipedia articles, and social media such as WNUT-2017 [
19
].
Arabic datasets are relatively few compared with English datasets and other
languages. This represents one of the Arabic NER challenges. Some of widely
1 https://gate.ac.uk/sale/tao/split.html
used Arabic datasets are ANERCorp created by Benajiba [
23
] and ACE2
(commercial dataset).
3
Methodology
In the proposed models, we implemented both traditional machine learning and
deep learning for running our experiments and evaluated their performance on
di erent datasets (English and Arabic).
3.1
Datasets
There are ve datasets used in the experimental comparison, two English datasets
and three Arabic datasets. To cover di erent datasets aspects (attributes), we
adopted diversity in the datasets represented in the di erent text source of each
dataset such as newspapers, Wikipedia and social media. Each dataset is split
into training, development and testing sets as indicated in the speci ed Table 1
for the English datasets and Table 2 for the Arabic datasets. The development
set was used for hyperparameter tuning to avoid over tting.
English Datasets CoNLL-2003: This is a benchmark dataset which was
introduced in the Conference on Natural Language Learning (a shared task for named
entity recognition) [
39
] and it has been extensively used in the NER task. The
CoNLL-2003 datasets cover several languages and we will focus on the English
dataset. The English data was taken from the Reuters Corpus which consists
of Reuter's news stories between August 1996 and August 1999. There are four
types of named entities in the dataset which are persons (PER), organizations
(ORG), locations (LOC) and miscellaneous names (MISC).
WNUT-2017: This high variance dataset was introduced in the Shared Task
on Novel and Emerging Entity Recognition 2017 [
29
]. The named entity tags
in this dataset have a wider range including Person, Location (including GPE
(Geo Political Entity)), Facility (center, station, etc.), Group (including
music band, sports team, and non-corporate organizations), Creative work (song,
movie, book, and so on), Corporation and Product (tangible goods, or
wellde ned services). The source of this dataset is comments taken from social
media websites including YouTube comments, Stack Over ow responses, Twitter
text for major events in 2016-2017, un ltered Twitter text 2010, and Reddit
comments.
Arabic Datasets ANERCorp: This is a widely used Arabic corpus that was
developed by [
23
] and has the same format as the ConLL-2003 dataset. ANERcorp
consists of 316 articles chosen from di erent newspapers for the sake of
generalization. The named entities in this corpus are persons (PER), organizations
(ORG), locations (LOC) and miscellaneous names (MISC).
2 https://www.ldc.upenn.edu/collaborations/past-projects/ace
AQMAR: This dataset contains 28 hand-annotated Arabic articles collected
from Wikipedia with 74,000 tokens [
31
]. The format of this dataset is similar to
CoNLL-2003.
WikiFANEGold: This dataset which is part of dataset named \gold-standard
ne-grained NE corpora" was manually created by [
32
]. The dataset contains
Wikipedia articles which were selected by choosing the articles that discuss
named entities and considering a fair level of distribution among the classes.
In addition, the textual data extracted from the Wikipedia articles was cleaned
by removing elements such as headings, lists, and captions on images and tables
etc. This dataset consists of 8 coarse-grained classes and 50 ne-grained classes.
The coarse grained named entities in this corpus are PER: Person, ORG:
Organisation, LOC: Location, GPE: Geo-Political, FAC: Facility, VEH: Vehicle,
WEA: Weapon, PRO: Product. We are using coarse grained named entities in
our experiments with a total size of 246,303 tokens.
The gazetteers we used in our experiments are ANERgazet [
23
] for Arabic
NER and the English gazetteers used by [
30
] for English NER which contain
lists of persons, locations and organizations names.
In the traditional machine learning models we implemented supervised machine
learning approaches for the NER task as supervised learning approaches
outperform the unsupervised learning approaches [
20
]- [
23
]. A variety of classi ers
have been used for NER, however, in our experiments we used Conditional
Random Fields (CRF), Support Vector Machine (SVM) and Random Forest (RF)
algorithms which have been proven to perform well [
10, 21, 22
].
3.3
Features
Features to be used in the supervised machine learning approaches were selected
based on their performance in other NER research, we used both
languageindependent and language-speci c features. In our proposed models, and in line
with previous research, we used the following features which have been proven
to be e ective:
First, the language-independent features:
{ The 3-character-su x of the word [
20
]: word su x information is helpful
to identify NEs. This is based on the observation that the NEs share some
common su xes.
{ The 3-character-pre x of the word [
20
].
{ Character length of a word [
21
]: this is a logical valued feature used to check
whether the character length of the current word is less than three characters
or not. This is based on the observation that the very short words are rarely
NEs. If the length of the corresponding word is less than or equal to 3 then
the feature values are de ned and denoted by False.
{ Whether the word contains any digit (0-9) [
22
]: This feature is helpful in
recognizing miscellaneous NEs, such as time expressions, measurement
expressions and numerical numbers etc.
{ Whether the word contains any punctuation [
18
].
{ Previous NE tag: the previous predicted named entity tag of the current
token [
22
].
Second, the language-speci c features:
{ Whether the word starts with a capital letter (English only).
{ List lookup features (gazetteers) [
20
]: a set of binary features which capture
whether the word is present as a speci c entity type in the gazeteer (English
and Arabic).
{ Part of Speech tags [
20
]: this feature represents the part of speech tag of
the current word and its surrounding words (two previous and two after)
(English and Arabic).
The following are the morphological features generated by MADAMIRA
tool [
38
] for the Arabic NER only:
{ Aspect: describes the aspect of an Arabic verb. It has four possible values:
Command, Imperfective, Perfective and Not applicable.
{ Gender: the nominal Gender. This feature has three values: Feminine,
Masculine, Not applicable.
{ Person: indicates the person information. The possible values are: 1st, 2nd,
3rd, Not applicable.
{ Voice: the verb voice. The values for this feature are: Active, Passive, Not
applicable, Unde ned, etc.
The baseline model (for both English and Arabic NER) we used included the
following features: the 3-character-su x of the word; the 3-character-pre x of the
word; character length of the word; whether the word contains any digit; whether
the word contains any punctuation, the Part of Speech tags and capitalization
(for English NER only).
3.4
Deep Learning Based Models
In this section, we describe the architecture of the neural network used which is
adopted from [
24
]. The model uses an end-to-end approach that does not require
language-speci c feature engineering or data pre-processing beyond
implementing pre-trained word embeddings. The recognition accuracy can be improved in
sequence labeling tasks such as named entity recognition by using the sequence
around the word under prediction. Thus, using a Bidirectional Long Short-Term
Memory (Bi-LSTM) model can give good performance [
24
]. The Bi-LSTM model
can learn from futuristic and past input features at a particular period of time
(e.g. a window approach). The Bi-LSTM model can learn from the past input
features using the forward pass technique and the futuristic input features using
the backward pass. According to the state-of-the art literature, the Bi-LSTM
model can be combined with a Conditional Random Field (CRF) layer to
enhance the model performance [
24
]. The approach of this kind of model is to
inherit the ability of learning futuristic and past input features from the
BiLSTM model and then implement a sentence-level tag to predict the probable
tags with the aid of the CRF layer.
In our experiments, we used both word embeddings and character
representation as features for our neural network based model (see Fig. 1). To encode
character-level information of a word, we used convolutional neural networks
(CNNs) for character-level representation. CNNs have been shown to be able
to extract morphological information from characters of words [
25
] such as the
word pre x or su x and encode this information into neural representations.
We also choose to use a CNN because we are dealing with the Arabic language.
Since Arabic is rich morphological language, using such a technique for Arabic
NER will identify di erent character-level features (word pre xes and su xes)
through the CNN (see Fig. 2). A combination of character- and word-level
representations then was fed into a Bi-LSTM. We used a sequential CRF on top of
the Bi-LSTM to cooperatively decode labels for the entire sentence.
Employing word embeddings bene ts NLP especially when we are dealing
with languages that have many rare words and large vocabularies [
26
], such as
Arabic. The nature of the Arabic language, speci cally word in ections,
generates several lexical variations which lead to sparseness in the Arabic corpus. In
our work, for the English NER we used Glove embeddings with 100 dimensions
which is publicly available by Stanford, trained on Wikipedia and web text,
and contains 6 billion words [
27
]. For Arabic NER, we used AraVec which is a
pre-trained distributed word embedding [
28
]. AraVec is an open source project
which provides free to use Arabic word embeddings trained on more than 3
billion words from web pages and Wikipedia.
3.5
Hyperparameter tuning
In our experiments, we used random search for hyperparameters tuning because
it was proven to be more e cient than other tuning approach such as grid
search [
34
]. The hyper parameters identi ed for the traditional machine learning
and the neural network based models are stated in Table 3.
Experimental Results and Discussion
To evaluate our NER models, we used the above mentioned datasets. The
CoNLL2003 dataset splits as training, development and test sets were those o ered by
the benchmark datasets. The rest of datasets were split into 80% for training,
10% for the development set and 10% for testing. The traditional F-score
measure was used to measure the performance. Results for the traditional machine
learning approaches are displayed in Table 3. The baseline model for each
classier included the implementation of the classi er with the language-independent
features listed in the features section. Since we are focusing on Arabic NER
in this paper, we separated the results into two rows for each Arabic dataset.
We compared the models performance using two sets of features, the baseline
language-independent features labeled as Sub and larger feature set, labelled
Full which included the language-speci c features listed in the features section.
Baseline results for the three classi ers are displayed in the columns labelled
SVM, RF and CRF respectively.
Columns labelled with -P show the results of adding in the previous tag
because it has been shown that this feature has a big e ect on model
performance. We excluded this feature in the CRF model as it is, in e ect, already
included as CRF is a sequential classi er. We also experimented to see the
benet of adding in the use of gazetteers, labelled -G in the table. Results show that
adding gazetteers also boosted the model performance. The columns labelled as
-PG include both the previous tag and gazetteers G.
From Table 4 we can reveal some information related to the impact of using
previous NE tag and gazetteers features on the performance of each model. On
the CoNLL-2003 dataset, the performance of the models improved dramatically
by using the previous NE tag as well as gazetteers particularly in the SVM
model. The best performance on CoNLL-2003 dataset was achieved by using
CRF model.
The performance on the WNUT-2017 dataset as we can notice from Table
4 is relatively low due mainly to the noise in the text as it was collected from
social networks. Using the previous NE tag or gazetteers on this dataset didn't
improve the performance. Instead the performance was decreased probably due
to the fact that the previously predicted NE tag is more likely to be wrong
which impacts the overall model performance negatively. The same applies to
gazetteers - it is di cult to provide support in a gazetteer for such noisy text.
However, the CRF model again proved to be relatively successful on this dataset.
In the Arabic datasets, using both language-independent and language-speci c
features and including the previous predicted NE tag and gazetteers as Full
features enhanced the general performance. The best performance was achieved
using the SVM model on ANERCorp and AQMAR datasets. The models
performance on AQMAR dataset was almost similar to ANERCorp but the
performance on WikiFANEGold dataset was lower which is possibly due to the higher
number of classes in this dataset. The best performance on this dataset was
achieved by using CRF model.
The performance of the Full feature was better than that of the Sub features
on all Arabic datasets. However, the e ect of using the additional previous NE
tag -P and gazetteer -G is bigger as we notice from Table 4
Table 5 shows the performance of the deep learning based models. The rst
column in the table labelled Bi-LSTM gives performance using only the
Bidirectional Long Short-Term Memory algorithm. The second column labelled
BiLSTM-CNN gives the performance of the combination of convolution neural
network CNN for character representation and Bi-LSTM. The third column
labelled Bi-LSTM-CNN-CRF gives performance including the addition of
Conditional Random Fields CRF algorithm.
In general, the performance of the deep learning based models is higher than
the traditional machine learning based models. Again, the performance on the
English datasets is higher than the performance on the Arabic datasets and that
is probably due to the challenges with Arabic NER already discussed. Unlike the
traditional machine learning based models the di erences in performance across
the deep learning based models are small.
The current best performance on the WikiFANEGold dataset [
35
] used
Buckwalter transliteration, English gloss, POS and NE tag in their model besides
window-based and dependency-based representation. They created an approach
to capture a global information in the corpus instead of focusing inside the
sentence using a CRF classi er in their model. Both the SVM classi er and the
deep learning approach used in this paper outperform this approach.
For the ANERCorp dataset the current best performance [
36
] created a
neural network based model which they named Arti cial Neural Network (ANN).
Their approach included three stages: the rst stage was preprocessing the data,
in the second they converted Arabic letters to Roman alphabets and in the nal
stage they applied a neural network to classify their data. They split the dataset
into 90% for the training set and 10% for testing set. However, compared to our
models, they achieved better performance most probably because of the data
pre-processing and converting the Arabic letters to Roman alphabets.
For the AQMAR dataset [
37
], the authors proposed a model that integrates
various custom-made techniques together, including representation learning (a
model using word embeddings and Bi-LSTM), feature engineering, sequence
labeling, and ensemble learning. They train multiple LSTM-CRF models to
construct the mapping from representations to predictions and then concatenate
their outputs as ensemble learning. Both the SVM-PG and the deep learning
BiLSTM-CNN-CRF approach used in this paper outperformed the state of the art
on this dataset. Our model SVM-PG gave the best performance on this dataset
and this is possibly because in our model, the performance was boosted by
using comprehensive language-speci c features in addition to the previous NE tag
and gazetteers. Also, in our approaches the hyper parameters were tuned using
random search technique while it was neglected in both compared models on
WikiFANEGold and AQMAR.
In this paper, we have explored a variety of di erent approaches to NER on
Arabic text with reference to how these approaches perform also on English
text. The exploration involves evaluating di erent classi ers and features on
a number of datasets. The selected datasets are diverse in terms of contents
source (e.g. news articles, twitter, etc.). We evaluated both language speci c
and language independent features. We found that adopting the language speci c
features and using gazetteers and the previous predicted named entity tag can
achieve higher performance in traditional machine learning based models. Also,
the deep learning based models have higher performance evaluations on the
most of datasets. Our proposed models outperformed the related work on two
Arabic datasets out of three. However, the performance on the English datasets
are higher than the Arabic datasets because of the characteristic of the Arabic
language represented in the morphological ambiguity.