Technology-assisted review (TAR) is software that aids in high-recall information retrieval tasks, such as abstract screening for systematic literature reviews. Often, TAR systems use a form of Active Learning (AL); during this process, human reviewers label documents as relevant or irrelevant according to a screening protocol, while the system incrementally updates a classifier based on the reviewers' previous decisions. After each model update, the system uses the classifier to rerank the remaining workload by prioritizing predicted relevant documents over irrelevant ones, enabling a reduced workload. Recently, studies have been performed that study the ability of solely using Large Language Models (LLMs) to perform this task by supplying the LLM prompts that contain the task, screening protocol, and a document from the corpus. The LLM then provides a classification of the document in question. While the results of these studies are promising, the LLM's predictions are not error-free, resulting in a recall or precision that is lower than desired. In this work, we propose a new Active Learning method for TAR that integrates the results of the LLM in the review process that may correct some of the shortcomings of the LLM results, leveraging a reduced workload with respect to current TAR systems.
Technology-assisted review (TAR) is software that aids in high-recall (information) retrieval (HRR)
tasks. An example of such a task is performing a Systematic Literature Review (for example, in medicine
[
In these extensive studies, the researchers, attorneys, or investigators gather evidence or information by screening documents stored in large databases or corpora. The task is to find nearly all information relevant to the subject of the investigation. In the case of Systematic Literature Reviews, the researcher starts by using specialized search queries to select documents from databases. Formulating these queries is not a trivial task, as it is the objective to capture (nearly) all relevant documents. These queries should not be too restrictive to minimize the chance that a relevant document is missed; researchers often use disjunctions rather than conjunctions. Consequently, the resulting set of candidate documents the researchers process is often enormous, while the prevalence of relevant documents within these sets can be very low.
More formally, we can specify this task as follows: we have a dataset containing all the candidate
documents found after the initial keyword search. During the review process, these documents are read
by the domain experts and labeled as either relevant or irrelevant. Read documents are referred to as
labeled. During the process, we maintain two sets ℒ+ and ℒ− for the labeled relevant and irrelevant
documents. The remaining unlabeled documents belong to the set . Traditionally, researchers screened
all documents in . Technology-Assisted Review are then systems or algorithms that aid the reviewers
in reducing the reviewing workload [
Early TAR methods consisted of first creating a randomly sampled subset of and training a classifier
on the labeled dataset ℒ. Then, that classifier is used to classify the remaining documents in [
Within TAR, the model is used in a transductive setting only, i.e., the model is only used to retrieve
the relevant data within the pool. The model is not used after the retrieval task has been completed
[
For abstract screening, consists of title-abstract pairs, which the reviewers for eligibility for the researcher’s systematic review or meta-analysis. The researchers follow a protocol that consists of inclusion and exclusion criteria to determine the eligibility of a record (in Section 4 - Figure 2, an example of such a protocol is displayed). This protocol should be followed strictly to ensure fairness and mitigate bias. Typical statistics of this process are given in Table 1.
Eligibility cannot always be determined from the title-abstract pair only due to the limited amount of information stored there, so reading the full-text of the paper is necessary to decide on definitive eligibility. Reading the full-text is associated with a high cost. Title-abstract screening greatly reduces the number of papers that have to be screened fully. TAR systems then aid in reducing the number of irrelevant title-abstract pairs so that not all records have to be screened.
Recently, methods have been proposed that use generative Large Language Models (LLMs) systems
to perform title-abstract screening (inter alia [
LLMs are prone to hallucination, where the LLMs generate responses that seem plausible but are
factually incorrect [
In [
In the following section, we will briefly overview previous work on TAR, LLM classification and techniques for combining weak supervision and AL. After that, we will explain our method, which consists of an LLM classifier and an Active Learning method that incorporates its predictions. As the LLM classifier assigns labels to each specific criterion, we introduce a case study in which we study a novel dataset that contains labels for each record at the criterion level, enabling us to assess the performance of our method. Finally, we will present our initial experiments and results, followed by a discussion and suggestions for future work.
Most TAR approaches are based on the Continuous Active Learning (CAL) algorithm (see Algorithm 1)
[
AutoTAR [
CAL, as described in Algorithm 1, leaves the question of a Stopping Criterion open (i.e., the
StoppingCriterion procedure, line 15 in Algorithm 1, is not given). Formulating a good stopping criterion
is an area of active research. Some practitioners use pragmatic criteria based on time constraints or stop
when the returns diminish (e.g., when TAR proposes irrelevant documents in a row; however,
specifying is target and topic dependent)[
The classifiers that are used in these systems are often based on classical Machine Learning algorithms
like Multinomial Naïve Bayes, Logistic Regression (AutoTAR), and Support Vector Machines combined
with TF-IDF features. However, some recent studies explore using neural networks and deep learning
(e.g., [
This work focuses on applying TAR to aid abstract screening for systematic reviews. In this field,
state-of-the-art systems can find (nearly all) after screening 5 – 40 % of the corpus by using this general
methodology [
In contrast to the AL-based methods, after the popularization of generative Large Language Models
like ChatGPT-3.5 and GPT-4 [30], systems have been proposed that use these models to perform
screening tasks. The main approach is to prepare a prompt that delineates the task and specifies the
criteria, followed by the title and abstract [
There have been several works that combine or compare LLMs and Active Learning. For example, in
[
In [
In this section, we describe the general architecture of our method. Our TAR procedure consists of two main components: a method to obtain classifications from the LLM and an Active Learning procedure that is used to rank the records during the review phase. Our AL procedure, LLM+CAL, uses the results of the LLM to reduce the review workload further.
In [
Chain-of-thought prompting is a method to improve the accuracy of LLMs when performing complex
reasoning. With this method, it is specifically requested in the prompt to think step-by-step in addition
to a few examples of appropriate answers. The aim is to let the LLM reason about its “thought process”
verbosely, which results in a higher probability that the final answer is correct [
Ignoring instructions. Only a part of the screening protocol was used according to the LLM’s chain-ofthought response. Some LLMs have problems following all instructions in the prompt, especially when the instructions are long and complex. Larger models like GPT-4 are less prone to this but have a higher computational and financial cost.
Often, the LLM followed the protocol partially: consider a dataset with four criteria, the LLM considered three criteria correctly but mistakenly ignored one of them, causing a misclassification of
questions by citing evidence in the given text followed by a YES or NO or UNKNOWN decision. When there is no evidence in the title and abstract, decide with UNKNOWN. Only answer with NO if there is absolute evidence given that the answer is NO. In the absence of evidence or when nothing is mentioned, always answer UNKNOWN. Use the following format:
work your way to an answer. Your full reasoning and answer should be given in this field)
write down []). In this field, only directly cite from the TITLE and ABSTRACT fields. DO NOT USE YOUR OWN
[. . .]. Furthermore, the study aims to [. . .], suggesting that this is indeed the case. So, the answer to this question is YES.
- "Sentence evidence 1" - "Sentence evidence 2"
about [. . .] but do not mention anything about [. . .]. As there is no definitive evidence, the answer should be
about [. . .]. This statement rules out that [. . .]. As there is evidence to the contrary, the answer should be NO.
- "Sentence evidence 1"
TITLE: {title} ABSTRACT: {abstract}
QUESTION: {question} the whole instance due to a mistake. This setup makes it challenging to detect failures due to a specific criterion. Mistakes become only apparent by combing through the (semi-structured) LLM answers containing information on all criteria.
We aim to mitigate this by considering each criterion separately, making the set of instructions shorter and less complex, which results in a higher accuracy. The system can then infer the inclusion status of a record by applying a simple logical formula to the model’s decision on the criteria (for example, Figure 2).
Despite the reduced complexity, it is still possible that the LLMs make classification errors, for example, due to hallucination, possibly because of missing knowledge. We hypothesize that these errors will not always happen at random, especially for the latter cause. Suppose the LLM makes an incorrect classification for a specific criterion due to missing knowledge. In that case, the LLM will likely make a similar mistake for instances similar to the one in question. Collecting the rationales and chain-of-thought fragments of misclassifications and training models on them might aid in predicting when the LLM makes a mistake or a correct decision.
We used LangChain [
As in canonical TAR, we represent each document as a high-dimensional vector. A typical feature extraction method is a bag-of-words method like TF-IDF that TAR systems frequently use. Combining sparse feature matrices and classical machine learning methods ofers fast retraining and reranking of the documents in . The AutoTAR baseline uses TF-IDF combined with a Logistic Regression classifier. In our approach, we will also use TF-IDF and Logistic Regression to ensure that changes in performance are not due to changes in the document representation.
During the process, the labeling task is specified as follows: we have a feature space tiab, which contains the feature vectors of the title-abstract (tiab) records. Each document presented to the oracle gets, for each of the criteria (see Figure 2), a label in the space crit = {+, ?, ¬} corresponding to True (Yes), Unknown, False (No). The option Unknown is vital in this phase, as it is not always the case that the information needed to determine eligibility for a criterion is present in the title and abstract.
Our method, LLM+CAL, consists of two phases: the first phase is called LLMPreferred, which is - in essence - a version of the method AutoTAR, but in this version constrained to select from the unlabeled {+,?}). As initial training data, the whole screening documents that are included by the LLM ( ∩ ℒLLM protocol is given in addition to a random sample of 100 LLM-excluded documents (ℒ−LLM). This phase is applied until 25 consecutive irrelevant documents are proposed, which might indicate that the set of relevant documents may be exhausted.
Because the possibility exists that there are relevant documents that the LLM does not find, we will switch to the CriteriaWSA method, which can query all documents within . First, all labeled data ℒ from the first phase is transferred to this method. Then, several machine learning models are trained: Inclusion Judgment Classifier. A Binary Classifier trained on the labeled data after transforming the data to binary = {+, ¬}, trained on the data in ℒ, in a similar fashion as AutoTAR. The criterion judgments are transformed using the formula specified in Figure 2, which will result in a label in the space ternary = {+, ?, ¬}. We can then transform ternary to binary by changing each ? into a +.
Acceptance Classifier. A Binary Classifier that determines Acceptance for each inclusion criterion.
This is similar to a method presented in [
Given a TAR task that has four inclusion criteria ({, , , }), we obtain the following pairs for each labeled for each labeled record: • tiab × crit × crit × crit × crit • tiab × binary • tiab × ternary • ans × • ans × • ans × • ans ×
During each annotation round, a batch of ten documents is given to the oracle using relevance sampling based on the ranking produced by the inclusion judgment classifier. The batch size of ten is an initial default value for this parameter. Smaller, larger, and dynamic batch sizes can be explored in future work. Another ten documents are sampled based on a ranking that is based on the predictions of the LLM and the Acceptance Classifier using the following equation: score(^L LM, acc) = ⎧ 0.75 + 0.25acc if ^LL LLMM == ?+ ⎨ 0.5 + 0.25acc ⎩ 0.5(1 − acc) iiff ^^L LM = ¬ .
Equation 1 is calculated for each study criterion , where ^L LM is the LLM’s prediction for criterion and acc is the corresponding acceptance probability. The mean of those scores is calculated for each of the unlabeled documents. Then, this score is used to rank the remaining documents in . The rationale behind Equation 1 is that instances with a higher probability to be relevant (instances with criteria that have more True labels) are put before documents that have Unknown labels, followed by documents that have False labels. Labels that have False labels and a low acceptance probability will have a higher probability of being selected than documents with False labels that are certain. For the True and Unknown labels, the inverse holds if there is a higher acceptance probability, they are preferred over instances with lower acceptance probability. This is still an initial formulation that may not always work optimally; other options can be explored in future research.
After this batch of twenty documents has been prepared, they are given to the oracle for labeling unless the LLM has found exclusionary evidence for a specific criterion and its acceptance probability is above 80 % (unless that criterion is a reason for exclusion for all remaining documents in ); these examples are skipped but may be proposed again in another round if the acceptance probability drops below 80 %.
This process is repeated until a stopping criterion is triggered, the oracle decides to stop the review, or is exhausted. In our experiments, we will stop querying after reviewing |ℒL{L+M,?}| documents. (1)
In this work, we compare the performance of various TAR methods on a dataset that is collected for a
systematic review (at the time of writing in preparation) that aims to identify common latent groups or
classes of PTSS/PTSD (Post-traumatic Stress Symptoms / Post-traumatic Stress Disorder) trajectories,
as well as their prevalence and predictors, which may give a better understanding how and under what
circumstances PTSS/PTSD presentations may develop [
Title. Some documents can be excluded by considering the title only. For example, animal studies are never eligible, and the fact that a study is an animal study can become clear from reading the title. We only study the records that have not been excluded by title screening.
Criterion. The eligibility of a study for inclusion depends on four inclusion criteria (see Figure 2). For each criterion ∈ {, , , }, a label crit = {+, ?, ¬}, corresponding to True, Unknown, False can be given. In Figure 3, some statistics per criterion are displayed.
Title Abstract. Using the logical formula in Figure 2, an inclusion judgment can be made for each criterion, so this level can be derived from the criterion level without additional human efort. This will result in a label in the space ternary = {+, ?, ¬}. Because an instance can have an Unknown label for one or more criteria, the final eligibility of such a study must be determined by reading the entire paper without exclusionary evidence in the record.
Full-text level: Final eligibility depends on reading the full-text of the study. This level is not considered in this work because this label needs more information than is available in this dataset (i.e., the full-text of every record).
This dataset is unique compared to other frequently used datasets used for benchmarking TAR
systems (e.g., [
We will consider only the records of one reviewer after title screening here, which results in a set of 4836 records after some data cleaning. Our dataset then contains |{+,?}| = 183 records that are included on the title-abstract level, resulting in a prevalence of 3.78 %. One observation that can be drawn from Figure 3 is that criterion determines the title-abstract inclusion label (displayed as judgment) the most.
Label statistics and relations 5000 4000 s t n e c o D f for example, that only a tiny subset of the documents for which is False, criterion is True. Also, it becomes clear that criterion excludes the most documents of all the criteria.
We compare several methods in a small simulation study on the dataset described in the previous section.
• AutoTAR, a state-of-the-art TAR method, • The LLM Classifier, as described in Section 3.1, • LLM+CAL, our AL method that integrates the predictions of the LLM Classifier, as described in
Section 3.2).
In this study, we only compare retrieval eficacy as we leave the question of a good stopping criterion open. Therefore, we constrain the run to the number of documents that are predicted by the LLM to be still eligible for inclusion (i.e., the number of documents with the label for which the inclusion judgment prediction is True or Unknown, |ℒLLM | we can compare the performance of the LLM classifier and the AL-based methods with the same review efort. During the experiment, we will record when various recall levels are triggered. We will record {+,?} ). We let each algorithm run until this number is reached. Then, the following metrics (calculated in the space binary).
Recall. The percentage of relevant documents found based on the a priori knowledge from the ground truth dataset.
= |ℒ |
+
+
| |
Work Saved over Sampling. This metric expresses the work reduction over random sampling [
We calculate this as follows. We will record this value for several recall targets: = | | || − ︂( 1 − |ℒ | + )︂ + | | Equation 3 is used in the AL setting. In the context of a classifier, we equate to the set of documents predicted to be irrelevant (the reviewers do not read those documents). For the LLM (2) (3)
Classifier, we can adapt the equation as follows.
= |ℒ−LLM| − ︂( 1
− ||
+ |ℒLLM|
+ | | ︂) (4)
The rest of the section is structured as follows: first, we describe the results of the LLM classification, followed by the results of a simulation study in which we compare the aforementioned AL-based TAR methods.
In Figure 4, we display an example of an annotated record. After parsing the response, we can highlight the fragments the LLM used in its decision-making. This overview is available for every instance in the dataset. When used in an annotation interface, the LLM explanations might aid users in their decision-making process, possibly reducing the screening time per document.
In Table 2, confusion matrices per criterion are displayed. A clear observation from Table 2 is that the LLM is more cautious in excluding papers than the human reviewer: the confusion matrices show high numbers of studies with ground truth False and predictions Unknown for all criteria. One of the causes is that when there is no written evidence to make a decision about a criterion, for example, whether or not a PTSD trajectory analysis (criterion ) was performed, the LLM would predict Unknown. This might seem like the correct decision in this situation. However, experienced human reviewers might exclude a paper based on their knowledge of the field by inferring that from other characteristics (for example, when the abstract describes a methodology that makes it impossible to use one of the eligible methods).
The LLM’s definition of specific terms or the meaning of concepts might diverge from the reviewers’. For example, for criterion , in some cases, the LLM eagerly infers from the descriptions of the studied populations that these might be exposed to trauma, which might not explicitly be mentioned in the record. Fortunately, the number of falsely excluded documents per criterion is low.
When combining the LLMs prediction, we can infer the title-abstract level predictions using the logical formula specified in Figure 2. In Table 2, the confusion matrix for this level is displayed, both on the ternary and binary levels. On this level, we obtain an accuracy of 78.52 % (ternary level), with a recall of 91.26 % on the binary level. In absolute numbers, this results in the fact that only 16 studies were missed out of the 183. The precision on the binary level is 12.9 %, resulting in a Work Saved over Sampling of 64.48 % (with Equation 4). Document M8746 Result: ¬, ?, , ¬ vs. Ground Truth ¬, ?, , ¬
a latent class analysis
c Abstract: BACKGROUND: In conflict-afected settings, women and girls are
c c vulnerable to gender-based violence (GBV). GBV is associated with poor long-term mental health such c as anxiety, depression and post-traumatic stress disorder (PTSD). Understanding the interaction between current violence and past conflict-related violence with ongoing mental health is essential for improving mental health service d provision in refugee camps. METHODS: Using data collected from 209 women attending GBV d d case management centres in the Dadaab refugee camps, Kenya, we grouped women by recent experience of GBV d d using latent class analysis and modelled the relationship between the groups and symptomatic scores for anxiety, d depression and PTSD using linear regression. RESULTS: Women with past-year experience of intimate partner violence alone may have a higher risk of depression than women with past-year experience of non-partner violence c,d alone (Coef. 1.68, 95% CI 0.25 to 3.11). Conflict-related violence was an important risk factor
c,d c,d for poor mental health among women who accessed GBV services, despite time since occurrence (average time c,d c,d in camp was 11.5 years) and even for those with a past-year experience of GBV (Anxiety: 3.48, 1.85-5.10; c,d
intimate partner violence or conflict-related violence may be at increased risk of depression, anxiety or PTSD. Service providers should be aware that compared to the general refugee population, women who have experienced violence may require additional psychological support and recognise the enduring impact of violence that occurred before, during and after periods of conflict and tailor outreach and treatment services accordingly.
Reasoning: ¬ The title and abstract do not mention anything about a longitudinal/prospective study with multiple time-point assessments. As there is no evidence to suggest that the study is longitudinal/prospective with at least three-time point assessments, the answer is NO. ? The question asks if the study assesses PTSD symptoms as a continuous variable and provides a list of eligible and ineligible PTSD scales. The abstract mentions that the study modelled the relationship between groups of women and symptomatic scores for anxiety, depression, and PTSD using linear regression. However, the abstract does not mention any specific PTSD scales used in the analysis. Therefore, it is not clear whether the study used any of the eligible or ineligible PTSD scales provided in the question, and the answer should be UNKNOWN. The study examines the association between gender-based violence (GBV) and mental health, including anxiety, depression, and post-traumatic stress disorder (PTSD), among women in a Kenyan refugee camp. The abstract mentions that conflict-afected settings put women at risk for GBV, which is associated with poor long-term mental health such as anxiety, depression, and PTSD. The study also found that conflict-related violence was an important risk factor for poor mental health among women who accessed GBV services, despite time since occurrence. Therefore, it is likely that the individuals in the study were exposed to traumatic events. ¬ The study aimed to investigate the relationship between GBV and mental health among Somali women in a
After obtaining the LLM’s results, we conducted several simulation runs of the AutoTAR baseline and our LLM+CAL method. Because both methods contain components in which random sampling takes place, we performed 30 runs per method to account for this. We stopped each simulation run after supplying the oracle 1295 papers, which is the number of documents the LLM predicted to be included (|ℒL{L+M,−} |). Stopping at this moment allows a comparison of the LLM’s recall to those of these methods given the same human reviewing efort. The recall curves of the methods are displayed in Figure 5. The mean recall (after stopping the simulation) of the AutoTAR method is 96.52 %, which is above the recall obtained with the LLM given the same human review efort. With the combined method, a similar recall is obtained (96.68 %), finding 177 out of 183 documents, reducing the number of missed studies from 16 to 6.
The mean recall after stopping the simulation is roughly the same for both AL methods. However, when considering other recall targets, it is evident that our combined method outperforms the baseline. For example, at 95 % recall, our method has a mean WSS@95 of 80.53 % versus 71.41 % of AutoTAR. This indicates that using the LLM predictions gives an additional advantage in retrieving relevant documents faster. In Figure 6, we give an overview of the performance for several other targets, of which all indicate that the LLM+CAL method outperforms the AutoTAR baseline. 100 % recall (183) 95 % recall (174) Exp. found at random # by AutoTAR WSS@60 - 54.1 % WSS@65 - 58.6 % WSS@70 - 63.6 % WSS@75 - 67.8 % WSS@80 - 70.9 % WSS@85 - 73.5 % WSS@90 - 72.7 % WSS@95 - 71.6 % 250 s t n e m cu200 o d t n a lve150 e r d e ive100 tr e fr o re 50 b m u n
0 0 500 1000 number of read documents 0 500 1000 number of read documents (a) AutoTAR (baseline) (b) LLM+CAL (ours) Run on a dataset with 183 inclusions out of 4836
Run on a dataset with 183 inclusions out of 4836 6 0 % 6 5 % 7 0 % 7 5 % 8 0 % 8 5 % 9 0 % 9 5 %
Work Saved over Sampling (%)
We have shown some preliminary results on our method, which indicate that adding LLM predictions
is beneficial to obtaining relevant documents at a lower cost than with the state-of-the-art method
AutoTAR as our LLM+CAL method yields higher work savings at several recall targets. Moreover, a
reviewer could achieve a better recall and WSS than obtained using only the LLM classifier. We have
presented a system that builds upon earlier LLM methods for Systematic Literature Reviews by making
the predictions more fine-grained by addressing each inclusion criterion separately. Moreover, our
approach aims to make the predictions more accurate and explainable by leveraging chain-of-thought
reasoning and asking the LLM to cite from the title-abstract record directly. Our method takes some
ideas from [
We evaluated our method on a single dataset, which may impact the generalizability of our results. Unfortunately, testing on more datasets is not feasible at the time of writing, as our method requires that the dataset has criterion-level labels. It may be interesting if we can adapt the method to work with feedback on the binary inclusion level, which might enable us to consider more datasets that do not have labels this fine-grained. Another interesting avenue is comparing the performance of our method on diferent LLM results than presented here. The LLM predictions may slightly difer when another model is used or when alternative formulations of inclusion criteria and general instructions are used. Further investigation is needed to determine what impact non-optimal instructions have on the LLM’s accuracy and the ability of our method to correct lower-quality weak labels.
The method we presented here is still relatively simple; several extensions can be made that might
further improve the eficacy. For example, incorporating Transfer Learning (as in [
We currently do not use the criterion-level labels during model training and subsequently rank documents in with those models. Designing a good method that combines the results of the four classifiers in a ranking is not trivial. Equation 1 is a starting point (now applied to LLM only) but not optimal. Relations between criteria have also not been taken into account yet. For example, assume a scenario where, within nearly all labeled records in ℒ, the proposition ∧ ∧ → holds. When, for a new instance, the LLM predicts the following labels {, , ¬, }, this record may be an interesting example to review for the oracle because it is an exception to what has been seen so far.
So far, the LLM rationales have not been used to train the classifier. In [
As mentioned before, we have left the question of a stopping criterion open. One avenue could be to combine the method with an existing stopping criterion or to use the LLM predictions to determine an optimal stopping point.
During a review, regardless of whether it is performed in the traditional setting or with TAR, labeling
mistakes occur due to human error [
We believe several ideas presented here might also benefit research areas other than TAR. For example, the LLM framework presented here can be applied to text classification tasks in general. However, adapting our method to a canonical AL setting is more appropriate in this setting. The framework we presented here enables obtaining weak labels at a low cost, with little engineering efort besides writing a good labeling protocol, and chain-of-thought prompting may aid in spotting errors within them, enabling more eficient creation of text classification models.
We thank the anonymous reviewers for their insightful comments, which helped improve this article’s quality. This work was sponsored by a grant from the Dutch Research Council (Domain Social Sciences and Humanities [SSH]), with file no. 406.22.GO.048. Moreover, this work was sponsored by a grant from the Human-Centered Artificial Intelligence focus area at Utrecht University.