In recent years, deep learning has shown supreme results in many sequence labelling tasks, especially in natural language processing. However, it typically requires a large training data set compared with statistical approaches. In areas where collecting of unlabelled data is cheap but labelling expensive, active learning can bring considerable improvement. Sequence learning algorithms require a series of token-level labels for a whole sequence to be available during the training process. Annotators of sequences typically label easily predictable parts of the sequence although such parts could be labelled automatically instead. In this paper, we introduce a combination of active and semi-supervised learning for sequence labelling. Our approach utilizes an approximation of Bayesian inference for neural nets using Monte Carlo dropout. The approximation yields a measure of uncertainty that is needed in many active learning query strategies. We propose Monte Carlo token entropy and Monte Carlo N-best sequence entropy strategies. Furthermore, we use semi-supervised pseudo-labelling to reduce labelling effort. The approach was experimentally evaluated on multiple sequence labelling tasks. The proposed query strategies outperform other existing techniques for deep neural nets. Moreover, the semi-supervised learning reduced the labelling effort by almost 80% without any incorrectly labelled samples being inserted into the training data set.
Deep learning is achieving state-of-the-art performance in image or video processing, audio processing or natural language processing. However, without using a pretrained model, deep learning typically requires a large amount of data. To c 2019 for this paper by its authors. Use permitted under CC BY 4.0. Se2mi-SupTeormvia´sˇsedSˇaAbacttai,vJeuLraejaErndiunagrdinPSa´lel,qaunedncMe aLrtaibneHlloinlegnˇa obtain unlabelled input for deep networks in video processing, cameras and other sensors are increasingly available. In natural language processing, a lot of unlabelled inputs can be obtained for almost no cost by gathering them from web sites. Unfortunately, labelling such data is very time consuming and expensive.
In this situation, we can benefit from semi-supervised learning using a large unlabelled dataset along with a small labelled one. Another option is to use active learning wherein each iteration, a part of an annotation budget is spent on labelling the most informative unlabelled samples. The model is retrained including those new samples and the process repeats. The annotation budget is significantly lower than the total number of available unlabelled samples.
Although active learning is a promising way to benefit from unlabelled data, the most common query strategy, uncertainty sampling, requires a measure of uncertainty. In sequence labelling, the measure can be easily defined for statistical models, such as hidden Markov models or conditional random fields (CRF), as they provide a probability of the labelled sequence or a marginal probability distribution for each element of the sequence. For neural networks, defining an uncertainty measure is more complicated since the soft-max activation function, typically used in the last network layer, does not correspond to a real uncertainty of network predictions. To overcome this issue, one can use a Bayesian neural network or include a statistical model, such as CRF, as the last layer of the network.
In sequence labelling, query strategies can be divided into two groups. The first group computes the uncertainty of the sequence predicted by a model. Query strategies of the second group compute uncertainties of separated tokens and then aggregate them to express the uncertainty of the whole sequence.
Querying the most informative sequence means that the annotator has to label every token of the sequence. This is expensive and often not necessary because some tokens can be very reliably annotated automatically. This situation can be found in many natural language processing (NLP) tasks, where some words can be assigned to only one category and we can predict that without knowing the context. A similar situation can be found in a video where two consecutive frames often contain the same or similar information and labelling all frames might be inefficient.
In this paper, we propose an active learning algorithm for sequence labelling with deep neural networks that queries labels of the most informative tokens whereas other labels are labelled automatically.
In the following section, we summarize approaches addressing this topic. In section 3, we define the architecture of our sequence labelling models. In section 4, we describe details of the proposed algorithm. The algorithm is evaluated with experiments on tasks from natural language processing and the results are shown in section 5.
Sequence labelling models have been used in many areas such as part of speech
tagging (POS) or named entity recognition (NER) [
With the increasing amount of data and computational power, and with
formulating new network topologies, deep networks are more and more popular in
sequence labelling. This is especially true for long short term memory networks
(LSTM), which deal well with vanishing gradient problem and are able to
incorporate context far from the predicted token. One of the state-of-the-art
topologies in sequence labelling is the bi-directional LSTM network (BI-LSTM) [
bilistic models [
In active learning for sequence labelling, the most informative sequence is
labelled. The sequence is then added to the training set, the model is retrained
and the process repeats. This requires the whole sequence to be labelled at
once. In contrast, Tomanek [
Active Learning in Connection with Deep Learning Although active
learning has been applied to many ML tasks, application to deep learning is
marginal compared to probabilistic modelling. One of the main problems in deep
active learning is that many query strategies require some uncertainty estimate,
however, most kinds of deep neural networks rarely support it. In literature, we
can find several approaches approximating the model posterior: variational
inference [
A sequence labelling model assigns categorical labels to all members (tokens)
of a sequence of observed values. In general, it considers the optimal label for
a given token to be dependent on the choices of nearby tokens. The problem is
often simplified through the assumption that the sequence of labels is a Markov
chain. With that simplification, the problem can be modelled with a
probabilistic graphical model such as a hidden Markov model [
Deep Learning Models. In sequence labelling, various kinds of neural
networks are used. These networks are typically designed for a specific task. This is
particularly true for their first layers that extract features. In NLP, a character
level embedding layer extracts low-level features from the text. In video analysis,
feature vectors are extracted using pretrained convolutional networks. After the
first layer, a layer that incorporates contextual information from neighbouring
elements is plugged in. The most commonly used layers on this level are LSTM
cells [
Moreover, to avoid over-fitting, a dropout regularization technique can be used. In our experiments, we use dropout for non-recurrent connections (solid lines in Figure 1). The dropout enabled for each layer allows to estimate prediction uncertainties, as the following section describes.
Bayesian neural networks aim to tackle several drawbacks of neural networks
such as overconfidence about their predictions or tendency to overfitting. In
classification, the prediction probabilities obtained from the soft-max function
are often erroneously interpreted as model confidence [
Y2 Fully connected neural network o2(back) o2(for)
LSTM Backward
LSTM Forward X2 Y2
CRF2
(a) BI-LSTM with fully connected neural network on the top.
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
can be uncertain despite high values of the soft-max function and these values
require correct calibration [
Gal & Ghahramani introduced Monte Carlo dropout [
Z p(y = c|x, T ) = p(y = c|x, ω)p(ω|T )dω t=1
R
≈ R1 X p(y = c|x, ωˆt),
where ωˆt ∼ qθ∗(w), R is the number of Monte Carlo runs, and where qθ(w)
denotes the Dropout distribution [
Monte Carlo dropout does not affect the model training complexity, however, each point has to be inferred repeatedly to obtain prediction uncertainty. 4
Query strategies for sequence labelling models can be divided into several
frameworks such as uncertainty sampling (US), query by committee (QbC), expected
gradient length (EGL) or information density (ID) [
x The probability of the sequence y in a by model M given the input sequence x is denoted PM (y|x). The set of labelled sequences is denoted L and set of unlabelled sequences is denoted U . 4.1
Query strategies of the uncertainty sampling framework select the sequences
that have the most uncertain label. The uncertainty measure can be expressed
in several ways. Least confidence query strategy [
Margin query strategy [
The margin query strategy utility function is defined as:
φM (x) = − PM (y1∗|x) − PM (y2∗|x) , where y1∗ and y2∗ are first and second the most likely sequences.
Token entropy query strategy [
K HM (l) = − X PM (yl = k) log PM (yl = k),
k where yl is label of the sequence in time l and K is the number of possible labels, over its labellings to define the utility function for selecting the most uncertain sequence: where L is the length of the sequence. The utility function is normalized by the length of the sequence. Omitting this normalization, the strategy would lead to querying long sequences as they contain more information. The unnormalized utility function is called total token entropy.
Whereas the marginal probability for CRF can be calculated using forward
and backward scores, those scores are not available for neural networks. We
propose an approximation called Monte Carlo approximation token entropy,
which uses the idea of Bayesian inference with Monte Carlo [
PMMC(yl = k) log PMMC(yl = k)).
Sequence entropy query strategy computes the entropy of probabilities of
all possible sequences. This strategy is unfeasible for long sequences as the
number of possible sequences grows exponentially with the length of the sequence.
Furthermore, it is not possible to obtain probabilities of a particular sequence
Se8mi-SupTeormvia´sˇsedSˇaAbacttai,vJeuLraejaErndiunagrdinPSa´lel,qaunedncMeaLrtaibneHlloinlegnˇa
directly in neural network based models. For probabilistic graphical models, the
strategy can be approximated with the N-best sequence entropy [
While the probabilities of N most likely sequences can be obtained directly in probabilistic models, this cannot be done in neural networks. Therefore, we propose a Monte Carlo approximation of the sequence entropy:
PMMC (yˆ|x)logPMMC (yˆ|x), (1) where N MC = {y1, y2, . . . } is set of all sequences predicted by Monte Carlo sampling and C2 normalizes the probabilities to sum to 1.
In the query by committee framework, a committee of models C = {M (1), ..., M (C)},
representing different hypotheses, is maintained during the whole process of
learning. The committee is used to query the sequence over which the
members are most in disagreement about how to label it. The committee is usually
trained using bagging. In each round, the labelling set is sampled with
replacement to create a unique training set L(C) that is used to train model M (C). The
committee prediction is obtained by models voting. In the context of deep neural
networks, maintaining a committee is too expensive for practical use. Although
dropout can be considered as a form of bagging [
US and QbC strategies are prone to querying outliers as they are often uncertain for the model and the committee of models often disagrees about them. The framework, called information density(ID), can be used to avoid this problem. ID uses a base utility function φB(x) and weights it by samples‘ representativeness. All above defined utility functions can be used as base utility functions. ID utility function is defined: φID(x) = φB(x) × 1 |U| β
X sim(x, x(u)) , |U| u=1 (2) where sim(x, x(u)) is a chosen similarity function for two sequences and β a parameter that controls a relative importance of the representativeness term. The similarity measure differs from task to task. 4.2
In the standard AL approach, the annotator has to label the whole sequence
although the sequence can contain subsequences that do not add too much value
to the utility function. If the model is sufficiently learned, these subsequences can
be easily annotated automatically using model inference. The decision whether
a token can be labelled automatically can rely on some kind of model confidence
[
We propose to use a combination of active and semi-supervised learning. For models with CRF layer on the top, marginal probability represents the model prediction confidence. Otherwise, the Monte Carlo dropout estimates the model prediction confidence. First, the most informative sequence is found with a chosen query strategy. Tokens in which the model is confident are automatically labelled using semi-supervised learning and the rest is given to an annotator. The labelled sequence is added to the training set and the process repeats. Details of the approach are described in Algorithm 1. The confidence threshold has to be chosen according to the model, problem type and query strategy.
Algorithm 1: Sequential semi-supervised AL framework
Input: L: labelled set U : unlabelled set φ(·): query strategy utility function θ: confidence threshold M : model type begin train model m of type M on data set L while stopping criterion is not met do // Find the most informative sequence from U x∗ = argmaxx∈U φ(x) // label the sequence with the model or query the annotator yˆ = m(x∗) for i = 1 to length of x∗ do if Pm(yi = yˆi|x∗) > θ then
yi∗ = yˆi else end
yi∗ = query(xi∗) end L = L ∪ hx∗, y∗i U = U \ x∗ retrain model m on L end end 5
To evaluate the performance of the proposed approach, we have chosen three different sequence labelling problems: named entity recognition(NER), part of Se1m0i-SupTeormvia´sˇsedSˇaAbacttai,vJeuLraejaErndiunagrdinPSa´lel,qaunedncMe aLrtaibneHlloinlegnˇa speech tagging (POS) and chunking. Experiments were performed with two sequential models: BI-LSTM-FCN with Monte Carlo dropout and BI-LSTM-CRF, and various query strategies designed for each of those models.
In the paper, we report two experiments. The first experiment tests proposed query strategies against random sampling and least confident query strategies as a baseline. The second experiment is using sequential semi-supervised active learning framework to reduce the labelling effort. Our primary aim was reducing the amount of labelled data required for training, rather than labelling performance. Therefore, we did not extensively optimize hyper-parameters such as learning rate, batch size or momentum. 5.1
The experiments were performed on the publicly available benchmark dataset
CoNLL 2003 [
Both models use GloVe embeddings [
First, each model was trained on the labelled training set with 10% of the original size for 30 epochs. This model was used for experiments with all query strategies. For each query strategy, the model was used to find the most likely paths and their scores together with tokens prediction confidences for all unlabelled sequences. The most informative sequences were selected and annotated until the annotation budget was exhausted. We have defined the annotation budget of one AL cycle in two ways: the number of labelled sequences and the total number of annotated tokens. In the first scenario, 100 sequences were selected and annotated, whereas, in the second scenario, sequences were annotated until the total number of annotated tokens reached 1000. With the updated labelled training set, the model was updated by iterative training for one epoch, then the new score was calculated. This active learning cycle was repeated 20 times. In the second experiment, samples were sorted according to their confidences, and the threshold value was chosen to achieve 0% or 1% of incorrectly labelled samples.
Early results showed that proposed query strategies are prone to select
outliers. Therefore, the information density wrapping strategy was used for all of
them. Each sequence was represented by the average of embedding vectors. The
representativeness of the sequence was computed as an average cosine distance
to all other sequences in the unlabelled dataset. The cosine distance is claimed
to be an efficient similarity measure of the linguistic or semantic similarity of
corresponding words for the chosen embedding [
In experiments, we studied the achieved performance in terms of F-measure (specifically F1 score) and accuracy by particular query strategies and the number of tokens that can be labelled automatically by semi-supervised learning. We report the macro-averaged F1 score that is calculated:
F 1macro = 1
X F 1q, |Q| q∈Q where Q is the set of all possible labels and F 1q is the F1 score for the class labeled q considered as the positive class and all remaining classes as the negative class.
These scores were compared with the models learned on the whole labelled dataset and models learned on a small labelled dataset that was later used for active learning.
Query Strategies Comparison Query strategies were compared in two AL scenarios: an unlimited number of tokens and a limited number of tokens. Table 1a shows that the strategies MC total token entropy and MC sequence entropy outperforms other strategies in both F1 score and accuracy. The MC total token entropy, however, required more tokens to be labelled. In NER, it queried almost twice as many tokens. In the scenario with a limited number of tokens, the MC sequence entropy dominates over other strategies except in the Chunking.
Table 1b shows that for the BI-LSTM-CRF model, the least confident and total token entropy query strategies have shown better results compared to the token entropy query strategy. We conclude that total token entropy query strategy dominates in the scenario with an unlimited number of tokens, whereas least confident achieves better results in the scenario with a limited number of tokens. The sequence entropy query strategy is missing as our implementation was lacking n-best Viterbi algorithm.
Moreover, Table 1b shows that the MC sequence entropy query strategy is the best among the compared strategies in the NER and POS tasks during the whole AL loop if the number of annotated tokens is limited in each cycle of the AL loop (Figures 2a and 2b).
we studied a possible reduction of the labelling effort using semi-supervised learning. We report how many tokens were automatically labelled if the threshold is set to not allow errors propagate into the training dataset and if 1% of errors are allowed. The results in Table 2 indicate that the BI-LSTM-CRF model has a more reliable uncertainty measure for the marginal distribution than the BILSTM-FCN model. It can reduce the labelling effort up to almost 80% without any incorrectly labelled samples being inserted into the training data set. The labelling effort is reduced up to almost 84% with 1% of incorrectly labelled samples being inserted into the training dataset. Se1m2i-SupTeormvia´sˇsedSˇaAbacttai,vJeuLraejaErndiunagrdinPSa´lel,qaunedncMe aLrtaibneHlloinlegnˇa MC Sequence Entropy Random MC Total Token Entropy Least Confident MC Token Entropy
NER 0 2500 5000 7500 0.775 0.770 0.765 1 F0.760 0.755 0.750
Anno1ta0t0e0d0tokens12500 15000 17500 20000
In this paper, we presented an application of Monte Carlo dropout, an approximation of Bayesian inference for deep neural networks, to active learning strateSe1m4i-SupTeormvia´sˇsedSˇaAbacttai,vJeuLraejaErndiunagrdinPSa´lel,qaunedncMe aLrtaibneHlloinlegnˇa gies developed for probabilistic graphical models. We proposed two not yet used adaptations of token entropy and sequence entropy query strategies suitable for LSTM-type deep neural networks. Moreover, we tested a combination of active and semi-supervised learning for sequence labelling for that network.
The proposed query strategies have shown a substantial improvement over the until now used strategy in sequence labelling with deep neural networks, least confident. The proposed strategies outperformed the least confident in all three considered sequence labelling tasks in case of the network without a CRF layer. This is particularly true, if the annotation budget is limited for each active learning batch, which is a typical real-world situation.
The combination of active and semi-supervised learning allows us to achieve up to 80% labelling cost reduction for the BI-LSTM-CRF model. The uncertainty measure based on Monte Carlo dropout, however, still needs improvement to achieve labelling effort reduction comparable with BI-LSTM-CRF. To this end, we would like to study uncertainty measures provided by other approaches to Bayesian recurrent neural networks.
Although uncertainty sampling has shown to be applicable to deep neural networks, other active learning frameworks have not been enough studied. In the future, we would like to study, in the context of sequence labelling and deep neural networks, active learning based on expected gradient length. In addition to this, we would like to apply deep active learning to sequence labelling in video processing, where context is also very important information.
The work has been supported by the grant 18-18080S of the Czech Science Foundation (GACR).