Text classi cation typically requires large amounts of labelled training data; however, the acquisition of high volumes of labelled datasets is often expensive or unfeasible, especially for highly-specialised domains for which both training data (documents) and access to subject-matter expertise (for labelling) is limited. Language models pre-trained on large amounts of text corpora provide state-of-the-art performance against most standard natural language processing (NLP) benchmarks, including text classi cation. However, their prevalence over more traditional linear classi ers and domain-based approaches have not been investigated fully. In this paper, we address the combination of state-of-the-art deep learning and classi cation methods and provide an insight into what combination of methods t the needs of small, domain-speci c, and terminologicallyrich corpora. We focus on a real-world scenario related to a collection of safeguarding reports comprising learning experiences and re ections on tackling serious incidents involving children and vulnerable adults. Our aim is to automatically identify the main themes in a safeguarding report using three main types of classi cation. Our results show that for a very small amount of data a simple linear classi er outperforms state-of-the-art language models. Further, we show that the performance of classi ers is more a ected by the size of the training data rather than the amount of context given.
The performance of natural language processing (NLP) classi cation tasks is
heavily reliant on the amount of training data available [
We look at whether domain-trained embeddings are e ective even when trained on very limited corpus. Our main contribution is that we conduct a thorough analysis of what combination of embedding and language models and classi cation approaches t the needs of a small domain-speci c and terminologyrich corpus. Further, we also look at how deep learning approaches are a ected by training dataset size versus the amount of context given. 2
The purpose of a safeguarding report is to identify and describe related events
that precede a serious safeguarding incident | for example, involving a child
or vulnerable adult | and to re ect on agencies' roles and the application of
best practices. Each report contains key information about learning experiences
and re ections on tackling serious incidents. The reports carry great potential to
improve multi-agency work and help develop better safeguarding practices and
strategies [
Furthermore, in our particular case, the safeguarding collection is expected to grow signi cantly in the near future, with the additional resourcing of 500 historical reports, making the manual annotation of these additional documents unfeasible. Therefore, we aim to automate the process of document annotation.
The thematic framework [
At the time of development the corpus consisted of 27 full safeguarding reports. The annotations were carried out by a social science team following standard methodology in the eld. They used a qualitative analysis tool (NVivo) to label parts of documents with thematic annotations from 5 top-level themes according to the thematic framework described in Section 2. The annotation was performed by labelling di erent-length passages of the reports with themes from the thematic framework. The majority of report contents were labeled except appendices. The total number of sentences in the corpus was 3,421 (see Table 1) with unbalanced distribution between the di erent themes where sentences can be associated with multiple themes. We evaluated models performance using training, development (both training and development were randomly sampled from the 27 reports) and test sets. Both development and test sets were annotated at the passage-level. The test set was extracted from safeguarding reports di erent from the original 27 documents. The test set contained a 100 randomly selected passages where each passage consisted of 3 sentences. Due to the limited amount of reports available, we built and evaluated classi er models on a sentence level (i.e., results presented in Section 4.2). Thus, each sentence was assigned the label of the passage to which it belongs. Further, we ensured that the train and development set do not intersect by automatically selecting random non-overlapping partitions for the two subsets. We also performed analysis at the passage-level, presented in Section 5.
In our experiments, we perform multi-label classi cation to identify main themes within documents. We compare three classi ers | a simple count-based classi er, a linear classi er based on word embeddings, and a state-of-the-art language model. We perform experiments with pre-trained and corpus-trained embeddings as well as di erent methods for building feature vectors. We use an n-gram feature representation and a Naive Bayes classi er as our baseline.
Our method consists of four overall steps, described below.
Step 1: Pre-processing We extracted terms from the corpus using FlexiTerm
[
Step 3: Feature Integration (FI) We use several ways for combining the word embeddings into reduced sentence representations: In the rst approach, we average the embeddings of each word in a sentence along each dimension. In the second approach, we assign TF-IDF weights to the words in a sentence, and calculate the weighted average of the word embeddings along each dimension (where the contribution of a word is proportional to its TF-IDF weight).
Finally, we use Bidirectional Encoder Representations from Transformers
(BERT) [
We evaluate the performance of the machine learning algorithms by using precision,
recall, and F1-measure metrics. The summary results are calculated using
microand macro- based measures. Early experiments using Word2Vec embeddings [
The results in Table 2 show that a simple terminology-based pre-processing step leads to slight improvements over the baseline with micro F1 of 0.59 in comparison to baseline micro-F1 of 0.57. Despite the small amount of data, we found that corpus trained embedding provide a notable advantage over pre-trained embeddings in the classi ers performance. fastText classi er outperformed GNB model, especially when domain-based embeddings were used. A non-verbatim example of a sentence where fastText model, based on corpus-trained embeddings performs better than pre-trained embedding models is: 'The police received information that the subject was selling crack'. A potential reason for fastText to classify correctly this sentence versus the classi ers using pre-trained embeddings is that the word `crack' has the meaning of a `drug' in the reports. However, this is not the widely accepted meaning for this word and thus it cannot be interpreted correctly by pre-trained models. The GNB based on pre-trained BERT model outperforms the classi ers based on pre-trained embeddings, however it does not lead to improvements over the domain-based models. Fine-tuning BERT is the best performing classi er with micro-F1 of 0.64 and macro-F1 of 0.59 which gives 0.5 improvement over the baseline. The improvement in the results achieved by ne-tuning BERT indicate the importance of adapting even the more context-aware pre-trained language models to the speci c domain, especially when the domain contains highly specialised language. Further, the poor performance of classi ers based on pre-trained word models shows the lack of transferability of pre-trained embeddings for a highly specialised domain such as the safeguarding reports.
The three best-performing classi ers give similar average results between the dev and test set (see Table 3). Further, models tend to return higher results for some themes, especially `Mental Health Issues' for the test set rather than the dev set. A potential reason for this may be attributed to the fact that the test set has been annotated in a similar manner to the classi cation models, i.e., independent of the context of the entire documents. The BERT classi er returned results above 0.50 for the themes `Contact with Agencies', `Re ections' and `Indicative Behaviour' for the dev and test datasets with precision above 0.60 and recall above 0.70. In the preceding section we evaluated the performance of the classi cation approaches against the annotations generated by the creators of the thematic framework, who we refer to as the expert annotators. By creating a classi er that uses the annotations generated by expert annotators as a `ground truth', we aim to produce uni ed and comparable results across generations that are not susceptible to variations in annotations created by di erent human annotators interpreting the coding framework. Going further, we judge the predictive power of the models by comparing their performance against the annotations of expert validators : independent social scientists who did not participate in the creation of the thematic annotation framework. We aim to measure the ability of the learned models to conserve the knowledge of the expert annotators versus if the task was performed manually by independent social scientists who were not creators of the framework (Section 5.1). In this way, we will be able to judge whether automated approaches are reliable for labeling the reports.
We perform three main types of analysis. First, we compare the performance of the classi ers against the annotations of expert validators. Secondly, we compare the performance of the classi ers for di erent length of sentences to observe the classi ers suitability for various sequence lengths. We also measure the e ect of the training dataset size on the performance of the models (Section 5.2). Thirdly and nally, we look at the e ect of the number of training instances versus the amount of context provided per instance on the performance of the classi ers (Section 5.3). 5.1
The initial thematic framework was developed by annotating passages of the documents rather than individual sentences. However, our classi ers are trained with sentences. In order to fairly judge the predictive power of the models against human annotators for annotating sentences and passages of the reports, we performed a study comparing the performance of the classi cation models versus two independent expert validators on sentence- and passage-level. For these purposes we used two datasets | one consisting of sentences and one consisting of passages. The sentence set consisted of a sample of 100 randomly chosen sentences, while the passage set consisted of a 100 passages, each containing three sentences. The sentence set was extracted from the dev set while the passage set was extracted from the test set (see Table 1). We measured the inter-annotator agreement for predicting themes using Cohen's kappa (see Table 4). We also compare the average F1 measure per theme between the expert validators and the best performing classi er (BERT).
The Cohen's kappa scores showed moderate agreement between the validators with an average score 0.40 on sentence and a passage level. The highest level of agreement is for `Mental Health Issues' theme. However, the average expert F1 for this theme is surprisingly low. The reason for the discrepancy between the Cohen's kappa score and the F1 measure is the occurrence of sentences which mention mental health problems such as `depression'. Such sentences are labeled by the expert validators as `Mental Health Issues', however their true label is di erent because of the surrounding context. Surprisingly, a large portion of these sentences were correctly classi ed by BERT. The average F1 score for the expert validators signi cantly improves for passage-level classi cation with average F1 = 0.60 in comparison to sentence-level annotations where an average F1 = 0.45 (see Table 4). This suggests that humans need more context | i.e., to see the sentences embedded in paragraphs | to classify sentences correctly, compared to deep learning models that can generalize better in these cases with limited context thanks to what they learned from the training set. 5.2
Experiments comparing the best-performing classi ers for di erent sentence length and training set size showed that BERT performed better than the baseline method for any length of sentences. Further, BERT gave higher results than fastText and the baseline for shorter sentences. For long sentences, BERT and fastText had very similar performance with a di erence less than 1% (see Fig.2). The comparison between the classi cation models performance for di erent sizes of training set revealed that deep learning models (i.e., BERT) are highly in uenced by the size of the training set in comparison to linear models such as the baseline and fastText (see Fig.2). BERT performed worse than the baseline for the very small training set while fastText gave similar performance to the baseline. However, BERT's performance almost doubled as more sentences were added to the training set while GNB performance was not that heavily in uenced by the size of the training data, especially for a training set with more than 1,000 sentences. 5.3
In this section, we extend the analysis from Section 5.1 by looking at the e ect of context versus the number of training instances provided for the classi er models. In this experiment, we gradually increase the length of the training instances in order to judge the importance of the training size versus the context (in terms of passage length). We evaluate the models using sentences and passages where each test passage consisted of three sentences (see Fig. 3). The test sets for these experiments were extracted from the dev set while the training sentences and passages were extracted from the training set. Results showed that the performance of deep learning models is more in uenced by the amount of the training instances rather than the length of the training passages. Further, models trained on sentence-level with a higher volume of training data give better results when tested on small paragraphs than classi ers trained on passages but with less training data available. This signi es the importance of higher volume of labelled data for reaching good classi ers performance. 6
Through this work, we explored the problem of predicting the main themes in safeguarding reports using supervised machine learning approaches. We analysed the performance of state-of-the-art classi ers, feature extraction and feature integration techniques which allowed us to identify classi cation methods suitable for domain-speci c documents. Results showed that state-of-the-art deep learning model performance is highly dependent on the size of the training data in comparison to linear models as BERT's performance is worse than a simple Naive Bayes baseline and fastText for very small training datasets. Further, training word embeddings onto the speci c domain, even when the size of the corpus is very small, lead to much higher results in comparison to pre-trained embeddings. This shows the importance of targeting pre-trained models to the speci c corpus despite its size. The study comparing the expert validators' performance versus the automated models showed that the thematic analysis can be challenging even for subject-matter experts without prior knowledge of the thematic annotation framework. Further, humans need more knowledge about the context surrounding a sentence, compared to deep learning approaches. Experiments showed that BERT and fastText performance is more a ected by the size of the training data rather than the amount of context given. On this respect, sentence-level classi cation provides more training data and ne-grained distinction between themes, which in turn allows for an easier expansion of the models and faster annotation.
In the future, we want to improve theme detection for the safeguarding documents by using generative language models for arti cially augmenting the sparse data of the corpus. We will use the additional data as a training set in order to improve classi er performance. Further, we plan to look into developing and using knowledge graphs for improving classi cation. This will help re ne the query functionality of the application and help improve the identi cation of similar documents and common trends in the safeguarding collection.