We present the findings of an interdisciplinary project that implemented a document tagging support system for nursing care experts. We evaluate its performance and provide lessons learned. This project was particularly marked by a low inter-rater reliability of the document labels and use case understanding issues, but also of a good performance of a simple, BERT-based binary relevance approach.
Text mining and document classification are long-standing research areas [
We present the findings of a small-scale applied interdisciplinary project in the domain of tagging support for document labelling tasks. The project’s goal was building a document tagging support system for health experts tasked with tagging nursing care publications. As an applied machine learning project, it had a set of requirements and challenges that are quite diferent from standard text classification or information filtering tasks, on which we report here.
Researchers in nursing care are investigating how to promote continuous professional learning
in the daily practice of nursing [
In the business understanding eforts it was realized rather late in the project that helping experts to save on the time needed to inspect a document was not as relevant as motivating the expert to rethink his or her tagging decisions.
The data consisted of 1515 nursing care or medical publications each associated with one or several of 24 diferent tags. The number of publications per tag (see Fig. 1) arose historically and reflects a data taking campaign that did not specify the relative frequency of the tag categories in the data set. After dropping tags with less than 20 associated documents and the class others, 1293 documents in 18 relevant categories remained.
The tag frequency distribution was a data understanding indicator that had major implications on the project: Only few tags were associated per document, but a significant fraction of documents had more than one tag associated with it. 76% of all documents had 1 tag, 23% had two tags and 1% had three or more tags.
Therefore, a multi-label classification approach [
Furthermore, metrics such as precision@k need to be evaluated with respect to the multi-label case. In a UX workshop with the experts it was found desirable to present = 3 selected document tags. This implies that precision@3 values cannot reach a value of 100%. Also, most information retrieval or recommender systems have a much smaller percentage of relevant items to retrieve or recommend, so that our precision values will likely be much higher.
A complication arose with the early realisation that there was some disagreement between
experts on which labels to assign to a given document. This efect was expected based on
the results of Xia and Yetisgen-Yildiz, since medical training alone does not ensure high
interannotator agreement and no NLP researcher had been involved in the annotation process
until this project. [
The proposed system consists of text extraction and feature engineering steps that return the BERT sentence embedding using the Hugging Face model "paraphrase-MiniLM-L6-v2"2 of the paper abstract and a tag filtering step based on this vector to be detailed further in Sec. 3.2. 2See https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2
A feature selection decision was made by using BERT feature vectors of the publication abstracts only. They were extracted manually by the experts and have the advantage of fitting well within the 512 subword tokens limit imposed by BERT.
For multi-label classification, we used a binary relevance problem transformation method [
As part of the user interface design choice and as a consequence of the distribution of the number of tags per document, = 3 tags were selected and presented in red with the associated probabilities. Fig. 2 shows the user interface as it is currently implemented.
We discuss the evaluation of the data mining prototype on the statistical level. The user level evaluation is discussed in 4.2. Our main results are given in Tab. 1
The low precision@3 and MAP@3 values can be attributed to the fact that the number of relevant tags per document are mostly 1 or 2. The average precision@1 is higher, at 82%. From a practical point of view, the recall@3 value is presumably the most important one, as three tags can easily and quickly be judged by an expert. By looking only at these top-3 selections, however, the expert might miss relevant tags in 8% of the cases.
This result was deemed suficient for a first iteration and therefore terminated classifier optimization eforts, as early user experience feedback was deemed more important. However, 42% 82% 92% 61% the single-tag binary classifiers were far from perfect, as the precision-recall curves in Fig. 3 show. In fact, these curves show that the less frequent tags (with a prevalence of 2-4%) perform worse than the more frequent ones (appearing in 6-13% of the documents), suggesting that a larger data set will likely help improve the performance further.
Although useful as a rough performance estimate, recall@3 must be considered a flawed metric since the number of relevant tags varies mostly between 1 and 2 tags per document Given that ∼ 97% of all documents in the test data set had at most two associated labels, Tab. 2 considers the documents with one and two labels separately.
On the subset of ∼ 63% of documents with only one relevant tag, Tab. 2 lists the percentages of documents for which this relevant tag is not found, found at the first, second or third item in the filtered list. For the ∼ 33% of documents with two relevant labels, similarly the percentages of where the two relevant tags were found are given.
Thus, for documents with one relevant tag, precision@3 values of over 90% were reached,
Position
Classifier Type Support Vector Machine
Naïve Bayes
None 7% 8% 3 3% 3%
None 1% 3% 2 relevant tags 1 tag 1+2 1+3 18% 64% 12% 20% 61% 7% whereas for documents with two relevant tags, both tags were found in the top 3 recommendations in at least 92% of the cases. Using a naïve Bayes classifier instead of a support vector machine generally resulted in a minor performance reduction.
All tagged publications are made accessible for nursing practitioners on a TYPO3-based website. Data is stored on a Linux server and managed in a MariaDB database.
Experts administrate publications in the backend of the website. The Python-based filter system is started once a day with a cron job, retrieves newly uploaded publications via REST-API and writes the filtered tags (also via REST-API) into a designated table of the MariaDB database, from where they are displayed to the tagging experts (see Fig. 2).
The system has so far been in use for 6 months. Based on an interview conducted with the nursing care expert chiefly tasked with tagging documents, several crucial insights could be established. The most problematic one was that the expert viewed the filtered tags as almost authoritative recommendations. The expert looked at the system’s three filtered tags and checked their plausibility, instead of looking at the document and determining the relevant tags. The option of tagging the document with a non-filtered item and the tag probability indications were ignored. This was done mainly out of convenience and eficiency, believing that no tag could be relevant that was not highlighted. The expert also declared that he no longer looked at the abstract and/or the paper, instead he would rely solely on the filtering system and the title.
These practices must be registered with alarm, since apparently even with usage instructions and being involved in this project and knowledge of the statistical evaluation results, the expert seemd happy to uncritically pass on the responsibility for the correctness of the tags to the tag ifltering system.
We built a document tagging support system to aid nursing care experts in building a labelled dataset for on-the-job professional education.
Using a simple BERT-based feature engineering approach combined with a standard radial basis function support vector machine, recall@3 values of around 90% were reached. While the domain experts deemed this result good enough for deployment, issues regarding the specified inter-rater reliability and scarce training data for several tags show potential for improvements. Consequently, Fig. 1 showed that not all tags were reliably recognized by the system. It is likely that inter-labeller agreement was the limiting factor in this regard, in contrast to many CRISP-DM-projects where data collection, feature engineering and optimizing the modelling step often pose the key challenges.
Our results also demonstrate that building custom tagging support systems is already quite inexpensive. This again suggests that domain-adaptation eforts are reasonably likely to be successful when transformer-based sentence embeddings are used.
We cautiously try generalise these insights. When trying to judge the probability of success of a tagging support system, some important positive indications are if • the domain-specific dataset is at least moderately sized. In the case of publications, 20-50 abstracts per category might sufice. • the labels are easily distinguished based on the text only, possibly even by non-experts. • experts judge the reasoning necessary to distinguish between categories to be simple.
During our project, we designed the system with the the goal of challenging the experts to reconsider their tag selections with the aid of the tagging support system. The system was therefore designed around assisting expert labelling, rather than providing an expert opinion by itself. Despite respective eforts and instructions though, the expert interviewed after using the system for two months developed a significant inclination to uncritically adopt the system’s (alleged) "recommendation" out of convenience. This UX-challenge remains unsolved and ofers great potential for future research.
In summary, we list some lessons learned:
• Carefully think about how the system biases the expert’s labelling decision. The experts
might need training to correctly use the system. Otherwise there is a possibility that they
uncritically accept the system’s biases in the tagging process by trying to become more
eficient. Additionally regularly check the way it is used after deployment.
• Do not start a tagging or labelling process without defining clear labelling instructions.
• Monitor the label correlations in the labelling process, and discuss label categories among
experts to verify that they are discernible by all [