Keeping a professional knowledge database with scientific publications up to date requires continuous scanning and annotating of newly published articles by domain experts. To shorten this time-consuming process, we study experts' assistance through a domain-specific tag recommender. We introduce the real-life case of a knowledge management system for nursing practitioners and present its architecture and user interface for creating and assigning tag recommendations. While the original tagging interface was thought to assist experts in the tagging process and possibly challenge them to reconsider their tag selections or assign more tags to a document, a preliminary evaluation shows an uncritical adoption of the provided recommendations by experts. We conclude that future design iterations of the recommendation user interface should try to prevent blind trust in the system and encourage reflection on tag suggestions.
cost-eficient care. One way to achieve this is through
evidence-based practice, consisting of evidence from
research, context, patient preferences, and clinical
expertise. To provide evidence-based knowledge to nursing
practitioners at the point of care, the Eastern Switzerland
University of Applied Sciences ofers a knowledge
management system that informs about the latest scientific
work for practical use [
While the system has been attracting great interest among nursing practitioners, scanning and preparing relevant scientific work is challenging. One of the core tasks of the editorial team is the annotation of scientific publications which is both time-consuming and cost-intensive. To this day, the team labeled 1’515 nursing care or medical publications with one or several of 24 tags to categorize the works for later recommendation to practitioners.
In order to support the work of the editorial team, we
have been implementing a tag recommender system for
health- and care-related scientific publications. Using a
BERT-based feature engineering approach [
Besides such performance measures, we particularly (P. Fröhlich)
focused on the user interaction with the tag recom- ering potential tag selections. We followed user-centered © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License of the tag recommender system and report on first expeAttribution 4.0 International (CC BY 4.0). riences of the experts involved with the corresponding asked to re-tag 60 publications. Besides the functional tagging user interface. recommender described above, we implemented an additional version of the recommender which picked one of the 24 tags at random. Both experts were provided 2. System Architecture and recommendations from both variants during the tests, Tagging Interface without knowing which system recommended the tags to the publication they were currently reviewing. For The knowledge management system FIT-Nursing Care1 is each publication they tagged, they were asked to identify a TYPO3-based website that makes relevant publications which of the two systems generated the notification. available to nursing practitioners. The website as well as Their success rate in identifying the functional recomour Python-based recommender system are hosted on a mender was 77% and 85%, respectively. This showed that Linux server. Data such as relevant scientific publications the non-randomness of the recommendations is noticeare managed in a MariaDB database and made accessible able and the experts can assess whether they are shown via a REST-API. Novel suitable publications are identified a solid recommendation or not. by the editorial team and added to the system’s literature After the system had been in use for three months, we database. The recommender system is implemented as an interviewed the key expert mainly tasked with tagging external component that checks the database for newly documents about his experiences. He reported that he added publications once a day and determines tags to be had noticed a definite change in his tagging approach. recommended. While originally he scanned a publication first and then
Nursing care experts administrate these publications chose one or several adequate tags out of the list, he now
in the backend of the website. This includes uploading started to first check on the three recommendations made
new publications and assigning one or several of the and then check their plausibility against the publication,
24 available tags, selected from an alphabetically sorted in many cases only by examining the title of the
publicalist. For the last task, the recommender system comes tion. This process of quickly validating the tags
recominto play. Based on results from a previous co-design mended led to ignoring potential other suitable tags. In
workshop with the experts and as a consequence of the many cases, the option of assigning a non-recommended
distribution of the number of tags per document, three tag was overlooked.
tags are recommended and presented in red with the Furthermore, we found that the current version of
associated probabilities (see Fig. 1) [
The design shown in fig. 1 was deliberately chosen and expert reported to rather rely on the red highlighting, yet implemented to not mislead experts into just confirming neglecting the actual probability presented. Whether the the assignment of the tags recommended. With them still percentage provided was above 90% or between 60% and having to select tags actively, we aimed at supporting 90% would result in the same outcome - the assignment of the experts’ decision-making process without the system the tag. As described by the expert interviewed, this was being too authoritarian. In the human factors literature, mainly done out of eficiency and convenience. This is communicating the uncertainty (or reliability) of a sys- an interesting finding, which tends to be obtained much tem’s perceptions or predictions has proven to support more often in real-world trials like the one presented the formation of long-term trust [4, 5]. Ease of use and here, as compared to lab studies. Findings pointing in seamless integration into the existing tagging process this direction have been presented [6], but only little was achieved, in order to foster the continuous use of the empirical evidence has so far been gathered. system without any additional eforts for the experts.
Throughout the implementation of the recommender was informal and only involved the key expert mainly resystem, its performance was continuously evaluated on a sponsible for organizing and tagging the documents, we statistical level using common machine learning metrics still found a crucial adaptation efect in his work behav(such as precision, recall, map@3). ior. Despite his knowledge on the statistical evaluation
Besides that, we were interested in how the novel tag results and the overall functionality of the recommender, recommendation is used by the experts and how their he quickly started to have blind faith in the system. Only original tagging behavior is impacted by the novel func- focusing on the top three recommended tags and ignortion. We designed and conducted a small user study ing the probabilities shown, the expert accepted passing with two experts from the editorial team, which were on the responsibility for assigning the correct tags to the system.
For further iterations of the tag recommender user prototype and evaluate alternatives for the current user interface, we identify two design challenges: interface, while particularly trying to encourage reflec
Prevent blind trust: Users seem to trust the recom- tion on the tag suggestions. mender regarding the top three tags suggested while ignoring the probabilities displayed. How can we better point out uncertainties and create awareness for more References accurate manual checking? A potential solution could include diferent color codes for visualizing the diferent levels of certainty. Another approach could be to also show the uncertainties of the other, non-recommended tags, in order to increase users’ sensemaking of the data.
Also, diferent levels of trust indications could be experimented with: additionally to the tag-specific reliability, also explanations for the quality of tag results could be provided, as well as overall system reliability [7].
Encourage reflection: The recommender was supposed to assist in the tagging process, yet not to overrule or replace the experts’ opinion. However, recommended tags seem to influence the experts’ choice very strongly.
How can we encourage reflection on recommendations and combine automated recommendations and expert knowledge in the best possible way?