In Machine Learning(ML) in general, more data is better data. Besides, more data defeats better algorithms for performance improvement in most cases. In practice, access to such data may not be directly forthcoming but become available with time. Existing ML solutions for Sinhala Named Entity Recognition (NER) and Part of Speech (POS) tagging, typically retrain their models from scratch when new data becomes available. The training time required for this purpose increases proportionally to the overall dataset size. This research proposed two online ML models that do not require retraining their models from scratch when data is obtained in batches, namely an Online Conditional Random Fields (CRF) Model, and a Bidirectional Long Short Term Memory-Conditional Random Fields (LSTM-CRF) Model. A Sinhala NER experiment using the Online CRF model improved on previous attempts by an F1-measure of 31.5% to reach 76% while the Bidirectional LSTM-CRF model improved on a previous attempt by an F1-measure of 51.6% to reach 80%. In the Sinhala POS tagging experiment using the Online CRF model improved the accuracy from approximately 71 % to 76% while the Bidirectional LSTM-CRF model improved the accuracy from 70% to 76%. The training time consumed by the proposed online learning model remains constant in each incremental training step since the model is not built from scratch. Further, both the Online CRF model and the Bidirectional LSTM-CRF model achieved improvements over the state-of-the-art Sinhala POS tagging accuracy of 4%. Based on the extrapolation of these results it can be seen that the existing Sinhala NER dataset needs to approximately double in order to obtain state-of-the-art performance reported for English.
Recent approaches to Named Entity Recognition (NER) and Part of Speech (POS)
tagging, use Machine Learning (ML) techniques [
This process indicates that the batch learning techniques retrain the ML models
using the same dataset multiple times. This retraining process results in a higher
training time. The main motivation to carry out this research is to avoid this overhead of
retraining, without losing the accuracy of the model. The Natural Language
Processing (NLP) models should need the capability to understand the present context of the
natural language. The proposed ML models should also adapt to the current natural
language context using the most recent data. The state-of-the-art methods for Sinhala
NER obtained nearly 92% precision [
2.1
The main approaches to NER and POS tagging can be stated as: Rule-based, Datadriven, and Hybrid. In the rule-based approach, linguistic knowledge is used to create a set of rules to identify named entities or part of speech tags. The data-driven approach contrastingly depends on three main ML methods, namely, Supervised, Semisupervised, and Unsupervised learning. The supervised learning methods use models such as Conditional Random Fields (CRF), Maximum Entropy (MaxEnt), and Hidden Markov Model (HMM) to build models from labeled data.
CRF is an undirected graphical model and matches up with the conditionally trained
probabilistic finite-state automata. CRF is capable of including arbitrary features
easily because it trained conditionally. CRF model has been used over the HMM model
because CRF solves the label bias problem [
Online learning algorithms are those which execute the training process on the data as it
becomes available and not all at once. Incremental learning is an online learning strategy
that works with limited memory resources and relies on the compact representation of
the already observed data [
Carreras et. al. [
Recurrent Neural Network (RNN) architecture has been designed to learn in an
online manner [
Athavale et. al. [
The implementation of the research followed the common machine learning pipeline as below: 1. Data Preprocessing 2. Apply Online Learning Algorithm 3. Post Processing
This research followed the supervised learning approaches because supervised
learning methods obtained superior performance compared to other learning methods [
Algorithm 1 Train and Predict 1: procedure TRAIN AND PREDICT(data, tokens) 2: train data,test data,validation data SPLIT(data) 3: train tokens,test tokens,validation tokens SPLIT(tokens) 4: X1; X2; X3; X4 CREATE MINIBATCHES(train data) 5: Y1; Y2; Y3; Y4 CREATE MINIBATCHES(train tokens) 6: ML INITIALIZE(params) 7: for i in (1, 2, 3, 4) do 8: TRAIN(ML, Xi, Yi) 9: VALIDATE(ML, validation data, validation tokens) 10: results PREDICT(ML, test data) 11: end for 12: ANALYZE(results, test tokens) 13: end procedure
Algorithm 1, first splits the data using a SPLIT function and divides the data set into train, test, and validation sets. Subsequently, the CREATE MINIBATCHES function creates four (in this case) mini-batches to train the model using the training dataset. The ML models are then initialized. The f or loop in the algorithm iterates through each mini-batch. The training, validating, and testing procedures are applied to each minibatch in that loop. At the end of the for loop, predicted results are stored in the variable results. The results are used to analyze the performance of the proposed ML models.
The research proposes two online ML models: an Online Conditional Random Fields (CRF) Model and a Bidirectional Long Short Term Memory-Conditional Random Fields (LSTM-CRF) Model.
The architectures of the two proposed models are shown in Figure 1. Model A
depicts the Online CRF model while Model B depicts the bidirectional LSTM-CRF model
with a dropout layer. Both models start with an embedding layer which converts words
into vector representations. The previously unseen words will not be represented in the
BoW vector model because it generates a sparse vector representation. This problem
of unseen data does not occur in the embedding technique because it creates a dense
vector space [
The proposed two models have a final CRF layer that can use sentence-level
features in the predictions. Model B also uses a bidirectional LSTM layer which can use
past and future contextual features for making predictions. The architecture of Model B
is influenced by a previously proposed solution [
Two tasks were carried out in this research: 1). Sinhala NER and 2). Sinhala POS tagging. For these experiments we used Sinhala NER and POS tagging datasets from the Language Technology Research Laboratory (LTRL) of the University of Colombo School of Computing (UCSC) under the Lesser General Public License (LGPL) for Linguistic Resources. The datasets applied to the proposed models adhere to the CoNLL2003 format.
The research proposed two online learning models, namely, an Online CRF and a Bidirectional LSTM-CRF. We carried out two experiments for each task mentioned above. Hence the overall research consisted of four experiments using the models. For each experiment in the research, we simulate the batch learning technique and the online learning technique to compare and contrast the performance of the learning techniques. Each experiment used four mini-batches of data. Suppose the four mini-batches are denoted as m1; m2; m3; and m4. A separate dataset T is used to test the performance of the models. In each training step, the batch learning model retrains from scratch while the online learning model has been saved and used it for further training. The training phase of the batch learning approach and the online learning approach consisted of four steps. These four steps of each batch learning and online learning experiment can be described as follows:
– Batch Learning Experiment Step 1: Initially trained using m1.
Step 2: When m2 mini-batch becomes available, we train the model from scratch using whole aggregated (m1 + m2) dataset.
Step 3: When m3 mini-batch becomes available, we train the model from scratch using whole aggregated (m1 + m2 + m3) dataset.
Step 4: When m4 mini-batch becomes available, we train the model from scratch using whole aggregated (m1 + m2 + m3 + m4) dataset.
– Online Learning Experiment Step 1: Initially trained using m1 mini-batch and test on dataset T .
Step 2: When m2 becomes available, the model trained only using m2. Step 3: When m3 becomes available, the model trained only using m3. Step 4: When m4 becomes available, the model trained only using m4. 5 5.1
Sinhala NER Experiment For the Sinhala NER Experiment, Figure 2 shows the variation of precision, recall, and F1-measure of the online and batch learning techniques in each step of the experiment. The precision values of the batch learning were higher in the first three steps, but in the fourth step, the online learning method performed better. The Recall values of batch learning techniques have higher values in all the steps, except for the third step of the experiment. The F1-measure values of batch learning were higher throughout the experiment. However, the online learning F1-measure value was closely related to the batch learning value in the last mini-batch.
Note that batch learning and online learning gives distinct accuracies in the initial step of this experiment. Since we have used the same dataset for the learning process, most of the time these accuracies are close. However, we train these models separately. Each ML model (batch and online) starts with a set of random values and then are trained on the data. These random initializations may have affected in changing the initial accuracies. More importantly, even though we use the same dataset for initial training, the training procedures are quite different. The online learning algorithm performs quick updates to the ML model while the batch learning algorithm does not perform such quick updates. These updates also affect the accuracies of online and batch learning. Sinhala POS Tagging Experiment We checked the applicability of the proposed Onlin CRF model for the Sinhala POS Tagging task. Figure 3 depicts the accuracy variation of the POS tagging experiment for both the batch learning and online learning approaches. The accuracy values of the online learning model were closely related to the batch learning technique. Further, in the first step of the experiment, the online learning model obtained slightly higher accuracies than the batch learning approach. However, in the other steps of the experiment, batch learning techniques demonstrate slightly higher accuracies. Sinhala-NER Experiment For the Sinhala-NER Experiment, Figure 4 shows the variation of precision, recall, and F1-measure of the batch learning technique with the online learning technique, for each step of the experiment. The precision values of the online learning model were closely related to the batch learning model except for the second step. When examining the recall values, the second and third steps of the batch learning technique gave higher values and in other steps, the online learning technique closely related to the batch learning technique. Also when it comes to F1-measure values, the online learning model was closely related to the batch learning model except for the second step. Sinhala-POS Tagging Experiment We also checked the applicability of the bidirectional LSTM-CRF model for the Sinhala-POS Tagging task. The variation in the accuracy of online and batch learning strategies are shown in Figure 5. The accuracy values of the online learning and batch learning techniques are closely related to each other as apparent from Figure 5.
As shown in Figure 6, the training time consumed by online learning models remains almost constant in each step. However, the training time of batch learning models has increased linearly in each step. One of the main objectives of this research was to find out how much data would be needed for the proposed models to reach acceptable accuracy. The research analyzed the performance variation of these experiments over a given mini-batch. For each experiment, we generate a function to approximate the performance variation. That approximated function is then used to predict the size of data that the proposed models require to obtain acceptable performance. For the NER experiment, we have considered the F1-measure variation since it is the combination of precision and recall. Since the POS tagging experiment has more than 22 labels to be predicted, it is convenient to use accuracy as the evaluation metric instead of precision, recall, and F1-measure. Hence, the accuracy variation is used for the dataset size prediction in the Sinhala POS tagging experiment.
According to Figure 2 and 4, we observed that the performance variation of NER follows a logarithmic scale. Hence, the F1-measure variation of Sinhala NER was approximated using a logarithmic regression. The results of the NER prediction are shown in Table 1. The first column of Table 1 contains the experiment name. The second column contains the approximated function using logarithmic regression. These logarithmic functions are defined in the range [0-100) exclusively. The independent variable of these functions is the number of steps in the experiment. Each step in the Sinhala NER experiment is trained using a separate data source (mini-batch) of 662 sentences. Thus the third column contains the number of data sources (the data source size is 662 sentences) that is needed to obtain near optimal performance. The final column shows how many sentences are needed to obtain such performance based on the fitted model.
We currently have a Sinhala NER dataset consisting of 3268 sentences. To obtain acceptable performance from the CRF model, the Sinhala NER data set needs to be at least 5296 sentences long, while the bidirectional LSTM-CRF model requires a total of 7282 sentences. Both experiments point to the need to approximately double the Sinhala NER dataset in order to obtain acceptable performance.
Sinhala POS tagging did not have a large enough dataset in the training phase, to observe the incremental improvement of accuracy in the POS tagging experiments. Hence, the dataset size prediction for this task were not possible to calculate. 6
We proposed two online learning algorithms: an Online Conditional Random Fields (CRF) and a Bidirectional Long Short Term Memory-Conditional Random Fields (LSTMCRF). Both models can be used across various NLP tasks such as NER and POS tagging. Since the models do not use any language-dependent features, these models can also be used across any natural language. The proposed models increase the performance in each incremental training step. The training time consumed by the proposed models remains constant over each incremental training step. The dataset size needed to reach acceptable performance for Bidirectional LSTM-CRF models is higher compared to that needed for the Online CRF model. However, the performance of the Bidirectional LSTM-CRF model is higher than the Online CRF model. The training time of the Online CRF model is low compared to the deep learning model. The analysis of four experiments showed that online learning techniques can reach batch learning performances. The training time for online learning methods remains nearly constant in each training step. However, the training time for batch learning increases linearly. The inclusion of the dropout layer for the proposed online learning model solved the stated key challenges (Catastrophic Forgetting, Concept Drift, and Stability-Plasticity Dilemma) of online learning. Most importantly, the dropout layer gives consistent growth to the online ML model.
The state-of-the-art Sinhala POS tagging experiment by Gunasekara et. al. [
Integrating the online learning based NER and POS tagging models with other NLP tasks can be implemented as future work. The actual usage of these online learning techniques become more worthwhile after converting the major NLP tasks such as Information Extraction, Machine Translation, Automatic Summarization, and Information Retrieval into the online learning strategy.