This work applies active learning to the novel problem of automatically classifying user-generated logbook comments, published in online rock climbing forums. These short comments record details about a climber's experience on a given route. We show that such comments can be successfully classi ed using a minimal amount of training data. Furthermore, we provide valuable insight into real-world applications of active learning where the cost of annotation is high and the data is imbalanced. We outline the bene ts of a model-free approach for active learning, and discuss the di culties that are faced when evaluating the use of additional training data.
Crowdsourced annotations have proven to be useful when creating new training
sets from unlabelled natural language corpora [
Therefore, in this work we employ active learning methods [
After comparing alternative active learning approaches, we adopt the
modelfree, exploration based strategy EGAL [
In many supervised learning tasks, there are not enough labelled samples to create su cient training and test sets. Often the cost of labelling samples to create such sets is very high. In these cases active learning can be employed to reduce the number of samples that need to be labelled.
In typical supervised learning tasks, training data is a random or strati ed
sample selected from the larger space. In the case of active learning, the learner
builds the training set by choosing the samples which are most useful to the
classi cation process. This is to reduce the number of samples that need to
labelled. The key hypothesis is that, if a learning algorithm is allowed to choose
the data from which it learns, it will perform better with less training [
In the work by Lewis and Gale [
As methods that consider only density and/or uncertainty can fail to explore
the sample space, some selection strategies will seek to select queries that are
di er from the samples in the labelled pool. Exploration guided active
learning (EGAL) [
There has been much success in recent years using crowdsourcing to annotate
data. reCaptcha is one of the best known examples of crowdsourcing in classi
cation tasks [
In this section we describe the key methods used to identify the most appropriate active learning approach for our data, how we applied this approach to collect additional training data and nally, how we evaluated the resulting augmented training set. We also include a brief description of the dataset we have collected.
A dataset of 100,000 comments was collected from UKC.com in September 2019. These include ascents covering the period 1952{2019, and the average comment was just 15 words long. In addition to the comment text, we also collected additional non-textual features associated with each comment. It was hypothesised that these additional features might aid in comment classi cation. In the case of each comment, where the information was available, the location of the route, the location that the climber was local to and the maximum grade achieved by the climber that year were collected as comment metadata. Of the 100,000 comments collected, 304 (0.304%) of them were labelled as containing beta. On inspecting those comments and their labels, an expert human annotator agreed with only 170 of them. It is clear that the data source contains even fewer labelled comments than initially expected and that the quality of this labelling is very poor. However, manual re-annotation of these comments provided a labelled pool of comments that we were con dent were correctly labelled and could serve as a starting point for the active learning process. While 99.7% of the comments in this dataset remain without class labels, we believe there to be a large class imbalance in the data where comments containing beta are in the minority. We include two comments from the dataset below to provide insight into the nature of the data and the language used to give beta. Comment 1 contains beta, while Comment 2 does not. 1. \Laybacked the main crack until the prominent right foothold, then stepped up with right foot whilst reaching up with left hand for a sound hold." 2. \Damp when we rst arrived but soon brightened up. An excellent afternoon, rst visit to the crag." A cross-validation experiment conducted on the pool of 304 annotated comments showed that the classi cation of beta was a learnable task, but that a larger training would be required to improve classi cation performance and ensure the model would generalise well to unseen data. Our dataset may prove useful to other researchers, in particular those considering classi cation of user generated text, short text or datasets with a large class imbalance. 3.2
To expand our training set as e ciently as possible, we sought to nd an appropriate active learning strategy for our dataset. We evaluated six di erent experimental combinations: three selection strategies, each with two classi cation models. The models proposed were a Support Vector Machine (SVM) with a linear kernel and a Random Forest classi er as these were shown to perform well for our data. The selection strategies were uncertainty sampling, random sampling, and EGAL. In each case the comments were represented as TF-IDF vectors of uni-grams, bi-grams and tri-grams.
In order to evaluate the di erent approaches, we simulated active learning
on our pool of labelled samples. A small random labelled pool was removed and
used as a training set while the remaining data was considered unlabelled. On
each iteration, samples would be selected from the unlabelled pool, added to
the training set and used to train a classi cation model. As we lacked su cient
annotated data to remove a hold-out test set, at each iteration, the model was
evaluated on all of the available annotated data. Each selection strategy was
then evaluated by plotting the classi cation performance (balanced accuracy
score) over the size of the training set. The best performing approach would
be the one that produced the steepest learning curve, ie. the approach that
achieved the highest accuracy with smallest amount of training data. As each
model was tested on a test set that contained samples also present in the training
set, the resulting accuracy would likely overestimate the performance on unseen
data, but the overall shape of the learning curve would still be informative. This
approach to evaluation is common when the labelled dataset is small [
In each case, the learning process was repeated 10 times, each with a di erent labelled pool of 10 randomly-chosen samples. All base labelled pools were balanced with 5 positive and 5 negative samples. In each case classi cation was evaluated using a balanced accuracy score which was averaged over the 10 experiments. The results of this experiment are provided in 4.1. 3.3
The selection strategy that performed best in the experiment outlined above was used to query 100 comments from the un-labelled of comments. These comments were labelled by a committee of expert annotators and added to the labelled comments used for training. A web page to developed to allow for remote, reliable annotation of comments by the human annotators. A brief pilot study was conducted with three participants in order to evaluate this annotation tool. Participants were asked to log on, annotate 30 comments and complete a questionnaire about their experience. From this pilot study we asserted that the site was functioning properly and that we could expect annotators to label 100 comments over the course of a week.
After collecting the user judgements from the annotation environment, the
labels were evaluated to assess inter-annotator agreement and quantify
consensus. In addition to a simple percentage agreement calculation, a Cohen's kappa
statistic [
The active learning stage of the project resulted in 100 additional samples that had been queried by our learner and labelled by participants using the annotation environment. In this section we outline the experiments used to evaluate the quality of this additional training data, and investigate how it might improve our ability to perform text classi cation of comments from UKC.com. The additional training samples were evaluated using a hold out test set of the comments most recently posted to UKC. Two models were trained; a pre-active learning model, trained on the original pool of labelled comments and a post-active learning model, trained on the original comments and the additional training data. Using our unseen test set, we could evaluate how each classi er might perform in a real-world deployment scenario. In our evaluations we report both a Balanced Accuracy Score (BAS) and the Area under the ROC Curve (AUC). The BAS is used due to the class imbalance in the data and reports performance on an unseen test set, at a decision threshold chosen after parameter tuning. The AUC instead considers performance across all decision thresholds and the classi er that maximises AUC is considered to be better able to distinguish between the two classes. While cross-validation is a common approach to evaluating classication, particularly in cases where annotated training and test data is scarce, we deemed it inappropriate for evaluating our additional training data for the following reasons: 1. The additional training data should not be used as test data, as this would reward any selection strategy that queried samples similar to the already labelled training data and may result in trivial classi cation. 2. The original training data should not be used as test data as the additional training samples are chosen to augment the training set, not to replace it. Instead, we have used cross-validation only as a means of parameter-tuning. In the case of each model, a 10 fold cross-validation was performed using all of the data available to the model, i.e., the pre-active learning model was tuned using only the samples that had been labelled prior to active learning.
The test set was composed of 150 comments that had been most recently posted to UKC.com. These were annotated by three expert human annotators. We refer to this set as Set 1. This set was heavily imbalanced, as it contained only 17 samples that were labelled as beta. While this test set was representative of the typical comments posted on UKC, a more informative test set should contain more instances of the positive class. Although it may have been possible to nd more examples of beta by searching for key words or other textual features, it was deemed that this was likely to result in trivial classi cation.
We chose instead to use features not included in the classi cation process to attempt to nd more instances of the beta class. From our initial labelled set of comments, we found that there were some users who were very active on the site and were more likely to include beta in their comments. By collecting up to 15 of the most recent comments from 10 such users we collected a set of 139 comments which, on annotation, was shown to contain 37 positive samples. This nal set was added to the original test set to give a larger, richer test set of 289 comments. We refer to this set as set 2. Both pre- and post-active learning models were tuned, trained and evaluated on each set and the results are reported in Section 4.3. 3.5 In addition to the textual features, three non-textual features were extracted and evaluated. (i) local to, a boolean features indicating if the climber was local to the area where the route was climbed. (ii) comment length, the length of the comment. (iii) challenge, a feature describing how challenging the climber would nd the route, calculating by comparing the grade of the route to the max grade achieved by the climber in the same year. In the case of each non-textual feature, the impact of that feature on the target variable was measured using information gain. This was calculated using the annotated base set and the additional data from the EGAL queries. The unseen test set was omitted from these calculations. The e ect of including these features in classi cation was then evaluated using the post-active learning training set and the nal unseen test set. Given the relatively small number of additional features, it was possible to evaluate all possible combinations of features exhaustively. For each subset of features, the model was trained on all of the textual features and that subset. Performance on the test set was reported as BAS and AUC.
The results of the experiment comparing di erent active learning selection strategies are provided below. Table 1 reports the area under the learning curve for each selection strategy, up to a training set size of 100 samples. Figure 1 shows the learning curves of the best performing selection strategies: (i) uncertainty sampling using a random forest classi er; (ii) EGAL using a random forest classi er. The smoother learning curve achieved using uncertainty sampling is due to the larger batch size. Each model-based approach was implemented using a batch size of 10, while EGAL was implemented with a batch size of 1.
As outlined in Section 2.1, EGAL is a model-free approach, meaning the classi er does not need to be retrained between batches of queries. As a result, it can be implemented with a batch size of 1 and still, all 100 queries can be made before any annotation is received. This is not the case however for model-based approaches like uncertainty sampling. In the case of uncertainty sampling, each batch must be annotated before the next batch can be queried. In light of this, the batches used to evaluate uncertainty sampling may have been too small. It is anticipated that, even with a larger batch size of 20 samples, the extent of annotation that was received would have been reduced, since annotators could not provide labels as and when they wished. The next batch of queries could not be made until the previous batch had been fully annotated.
Selection Strategy Random Sampling (SVM) Uncertainty Sampling (SVM) EGAL (SVM) Random Sampling (RF) Uncertainty Sampling (RF) EGAL (RF)
Using EGAL, 100 comments were queried from the pool of unlabelled comments and annotated by 10 human experts using our annotation tool. Here we report the results of some brief experiments to evaluate consensus amongst these annotators. In total we received 680 labels, meaning on average comments would be assigned a class label aggregated over close to 7 votes. Using pairwise agreement we calculated a total agreement percentage of 76.7% and a Cohen's kappa value of 0.498. Percentage agreement is often criticized as a means of inter-annotator agreement as it does not account for chance agreement. If we assign 10 random labels to each of our 100 comments, we achieve a 50% percentage agreement. For this reason we consider Cohen's kappa in value in addition to percentage agreement. A kappa value between 0.4 and 0.6 is considered moderate agreement.
We also chose to consider the rate at which annotators disagreed with the majority. The average annotator disagreed with the majority between 8 and 12% of the time. However there we two outliers; annotators who disagreed with the majority 21 and 33% of the time. These annotators received kappa scores of 0.21 and 0.32 indicating very low agreement. After removing these annotators from the study we recalculated the agreement measures. Across the remaining annotators the percentage agreement was calculated to be 82% with a kappa of 0.63. This indicated substantial agreement.
On investigation it was found that both of the unreliable annotators had strong biases that lead to their poor agreement scores. 17 of the 18 errors made by one annotator were false positives, while the other problematic annotator made 17 false negatives. It is encouraging to nd that the unreliable annotators were biased and not simply inconsistent. Were these annotations very inconsistent, it may suggest that labelling beta is not a learnable task. The fact that they are seen to be biased suggests that, with better calibration, their agreement scores could have been improved. This calibration could be achieved by changing the instructions and examples provided to the annotators prior to annotation. 4.3
Table 2 reports the balanced accuracy score (BAS) and area under the ROC curve (AUC) for each model on each of the unseen test sets. We can see that in all cases the classi er trained post active learning outperforms the classi er trained before active learning. The balanced accuracy scores are achieved using the decision threshold selected from the cross validation and as such, represent our best estimate of the models performance were it to be deployed. The AUC reported is the area under the ROC curve and instead quanti es the classi ers performance across all decision thresholds. This may provide a more robust means of comparing the models as it is not subject to decisions made in the crossvalidation process, namely: choosing the best decision threshold. This choice of decision threshold may be over tting to our relatively small labelled set used in the cross-validation. While, there may be a better choice, that better generalizes to all comments, the balanced accuracy scores reported in table 2 represent our best e ort at labelling unseen data both before and after active learning.
Metric AUC Set 1 BAS Set 1 AUC Set 2 BAS Set 2
Pre-Active Learning Post-Active Learning 0.794 0.824 0.699 0.715 0.830 0.837 0.736 0.756 Each of the additional non-textual features were evaluated by calculating their information gain on the target class. These values are reported in table 3. The ve most informative textual features are included in this table for comparison. When we consider these textual features we see examples of the highly speci c language that is common to the comments containing beta. Words like `reach', `left', `right', and `foot' are all indicative of the instructive language used to give beta. We can see how these terms form the instructions that might help a climber succeed on a route. Table 3 shows the comment length to be the most informative, not only of the additional features but of any feature.
In addition to calculating the information gain, it was also proposed to evaluate the additional features by including them in the classi cation and assessing their e ect on performance. Given the relatively small number of additional features, it was possible to evaluate each possible combination of features. Table 4 reports the performance of the post-active learning classi er on the nal unseen test set with each combination of the additional features. These results show that the addition of any of the non-textual features resulted in an increase in performance. However, unsurprisingly the comment length feature proved to be the most bene cial, achieving a 4% increase in BAS and a 1.3% increase in AUC.
In an attempt to better understand the e ect of these non-textual features on comment classi cation, it was decided to repeat these evaluations using our pre-active learning classi er. Figure 3 shows how the non-textual features e ect classi cation performance in both pre- and post-active learning models. Each line in gure 3 reports the BAS before and after the non-textual features were included. The slope of the lines represent the e ect of these on the model's performance. It is apparent from the gure that the pre-active learning model sees less improvement from the additional features than the post-active learning model. This is a further testament to the quality of the active learning training data. In essence, with this additional training data, our model makes better use of the non-textual features. 5
We sought to apply active learning to the problem of automatically classifying
user-generated logbook comments from online rock climbing forums. UKC.com
implemented labels on their site which allow users to identify comments
containing beta yet, the majority of these comments remain unlabelled. We have shown
our nal classi cation model can identify beta in comments on UKC.com with
80% accuracy. We believe this to be close to the upper limit of what is achievable
for this classi cation task. As we saw n Section 4.2, expert human annotators
achieve 82% pairwise agreement in the same task. We have used Exploration
Guided Active Learning (EGAL) to identify the most informative samples to
add to our training data. By including non-textual metadata features in the
classi cation process, we have improved our model's balanced accuracy on an
unseen test set by 6%. We believe the techniques we have employed should
prove useful in similar tasks involving the identi cation of anomalies in
usergenerated text where annotation is limited. An obvious analog to our work is
identifying spoilers in movie reviews. Further, we have highlighted the necessity
for the use of model-free active learning approaches in cases where the cost of
annotation is high. Many model-based selection strategies are evaluated using
batches of ve or fewer samples [
Acknowledgement. This work was supported by Science Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289 P2.