Introduction

Technology-assisted review (TAR) is software that aids in high-recall (information) retrieval (HRR) tasks. An example of such a task is performing a Systematic Literature Review (for example, in medicine [1]), but there are also applications in the legal domain (e.g., e-Discovery [2], but also the processing of Freedom of Information Act Requests, criminal investigation, etc.). For all these search tasks, it is important that nearly all relevant information is found, so these have a recall target of 75 -100 % [3].

In these extensive studies, the researchers, attorneys, or investigators gather evidence or information by screening documents stored in large databases or corpora. The task is to find nearly all information relevant to the subject of the investigation. In the case of Systematic Literature Reviews, the researcher starts by using specialized search queries to select documents from databases. Formulating these queries is not a trivial task, as it is the objective to capture (nearly) all relevant documents. These queries should not be too restrictive to minimize the chance that a relevant document is missed; researchers often use disjunctions rather than conjunctions. Consequently, the resulting set of candidate documents the researchers process is often enormous, while the prevalence of relevant documents within these sets can be very low.

More formally, we can specify this task as follows: we have a dataset 𝒟 containing all the candidate documents found after the initial keyword search. During the review process, these documents are read by the domain experts and labeled as either relevant or irrelevant. Read documents are referred to as

Table 1

Typical process statistics for Systematic Literature Reviews. Users query multiple databases using keyword search, which yields a candidate set of records 𝒟 which are screened. In this work, we aim to optimize this screening phase. After the title-abstract screening phase, the reviewers will read the full-text of the remaining set of documents (𝒟 + ), which will determine the definitive eligibility for inclusion in the review or meta-analysis. labeled. During the process, we maintain two sets ℒ + and ℒ − for the labeled relevant and irrelevant documents. The remaining unlabeled documents belong to the set 𝒰. Traditionally, researchers screened all documents in 𝒟. Technology-Assisted Review are then systems or algorithms that aid the reviewers in reducing the reviewing workload [4], while still aiming to find all relevant documents 𝒟 + .

Databases Keyword Search (𝒟) Title Abstract (𝒟

Early TAR methods consisted of first creating a randomly sampled subset of 𝒟 and training a classifier on the labeled dataset ℒ. Then, that classifier is used to classify the remaining documents in 𝒰 [5]. Many recent TAR systems use a form of Active Learning to update the classifier after each or several review decisions iteratively [6,7,8,9,10,11]. AL is a Machine Learning technique that is used to train a classifier with fewer labeled data points while retaining good performance. In this setting, the model can interactively query an oracle (i.e., the domain expert) to label data points with the desired output of the Machine Learning model (i.e., in the case of a classification task, the class of the data point). In our case, the model should predict each document's relevancy or inclusion status. In canonical Active Learning, the selection strategy aims to select the "most informative" examples from the perspective of the classifier. An example of such a strategy is Uncertainty Sampling [12]. The goal of canonical AL is to create a good inductive classifier, that can be used to classify previously unseen documents not found in the pool of potential training examples.

Within TAR, the model is used in a transductive setting only, i.e., the model is only used to retrieve the relevant data within the pool. The model is not used after the retrieval task has been completed [13]. Many TAR systems (e.g., [14,7,9,8]) use relevance sampling [15], a greedy batch sampling method that selects a batch ℬ with the top-𝑘 documents with the highest probability of belonging to the class of relevant documents according to the trained model. After the annotation of each document in ℬ, the model is retrained, and a new ranking for the documents in 𝒰 is produced. The objective is then to find all the remaining unlabeled relevant documents belonging to the set 𝒰 + , while minimizing reading documents that belong to the set 𝒰 − .

For abstract screening, 𝒟 consists of title-abstract pairs, which the reviewers for eligibility for the researcher's systematic review or meta-analysis. The researchers follow a protocol that consists of inclusion and exclusion criteria to determine the eligibility of a record (in Section 4 -Figure 2, an example of such a protocol is displayed). This protocol should be followed strictly to ensure fairness and mitigate bias. Typical statistics of this process are given in Table 1.

Eligibility cannot always be determined from the title-abstract pair only due to the limited amount of information stored there, so reading the full-text of the paper is necessary to decide on definitive eligibility. Reading the full-text is associated with a high cost. Title-abstract screening greatly reduces the number of papers that have to be screened fully. TAR systems then aid in reducing the number of irrelevant title-abstract pairs so that not all records have to be screened.

Recently, methods have been proposed that use generative Large Language Models (LLMs) systems to perform title-abstract screening (inter alia [16,17,11,18]). The main approach is to prepare a prompt that delineates the task and specifies the criteria, followed by the title and abstract. After supplying the prompt to the LLM, it will provide an answer and a decision on the inclusion status of that record. Obtaining results can be automated by making a program or script that automatically processes a dataset through the models' API. In [16], the authors report a mean accuracy of ± 90 % with a recall of 76 %. However, the performance varied per dataset, with recall scores ranging from 59 % to 100 %. In another study, the reported precision is low for some datasets [11], which may result in a higher screening workload than current AL-based systems offer.

LLMs are prone to hallucination, where the LLMs generate responses that seem plausible but are factually incorrect [19]. Moreover, LLMs are very eager to provide an answer even though there is no information provided in the LLM's training data or within the prompt to give a good answer [20]. With the current limitations, using the LLMs to determine the inclusion status of the title and abstract pairs may not be reliable enough.

In [21], the authors propose a system that combines (canonical) AL with Weak Supervision (e.g., noisy labels provided by a black-box model). To our knowledge, a TAR method that combines AL and noisy labels (e.g., from an LLM or another model) has not been presented yet. In this work, we propose a system that combines LLM classifications and Active Learning to improve the efficacy of the TAR procedure. Our main contributions can be summarized as follows:

1. A system that provides more detailed LLM classifications for all the criteria in the screening protocol instead of a single binary label for inclusion. 2. A system that makes LLM classifications more transparent by making the LLM provide a detailed explanation for each classification. 3. An Active Learning method that incorporates the LLM results to reduce the workload of the review. 4. A preliminary experimental evaluation of our method and several suggestions for future work.

In the following section, we will briefly overview previous work on TAR, LLM classification and techniques for combining weak supervision and AL. After that, we will explain our method, which consists of an LLM classifier and an Active Learning method that incorporates its predictions. As the LLM classifier assigns labels to each specific criterion, we introduce a case study in which we study a novel dataset that contains labels for each record at the criterion level, enabling us to assess the performance of our method. Finally, we will present our initial experiments and results, followed by a discussion and suggestions for future work.

Related Work

Most TAR approaches are based on the Continuous Active Learning (CAL) algorithm (see Algorithm 1) [22]. In this process, a model is trained on the documents that have already been reviewed. The model is then used to rerank the remaining documents in 𝒰. Several CAL procedures [8,23,9,7] require a set of seed documents provided by the reviewer. This set needs to contain at least one relevant document, but it does not need to be a document from 𝒟; it may also contain a description of the research topic as a pseudo-document. Additionally, one example of an irrelevant document is needed.

AutoTAR [14] extends the CAL procedure, which is still considered state-of-the-art and has been included in many studies as a baseline, for example, when studying ideal performance vs. the performance of a stopping criterion [10,24,25]. Instead of just training on the labeled documents ℒ + , ℒ − , it samples a set of documents from the unlabeled set 𝒰, which are temporarily assumed to be irrelevant; a fair assumption, given the low prevalence of relevant documents in most datasets. ASReview [9], opensource TAR software specialized for abstract screening, resamples the data to improve the performance in the presence of imbalanced training data. FASTREAD2 [7] modifies the CAL procedure with the goal of detecting human errors during the review procedure, as noisy human labels may occur [26].

CAL, as described in Algorithm 1, leaves the question of a Stopping Criterion open (i.e., the Stop-pingCriterion procedure, line 15 in Algorithm 1, is not given). Formulating a good stopping criterion is an area of active research. Some practitioners use pragmatic criteria based on time constraints or stop when the returns diminish (e.g., when TAR proposes 𝑘 irrelevant documents in a row; however, specifying 𝑘 is target and topic dependent) [27]. Several heuristics [14,7,28,27] (for example, characteristics of the recall curve) have been proposed, as well as methods that change the CAL procedure to allow the use of statistical methods that predict when a recall target has been achieved (inter alia [10,23,24]).

Algorithm 1

The Continuous Active Learning algorithm. The algorithm requires as parameters a dataset 𝒟, an unlabeled set of documents 𝒰, labeled documents ℒ + , ℒ − , a classifier 𝐶, a batch size 𝑘. The Active Learning procedure selects new documents according to the relevance predictions of the classifier 𝐶, which are updated after each batch of labeling decisions. The classifiers that are used in these systems are often based on classical Machine Learning algorithms like Multinomial Naïve Bayes, Logistic Regression (AutoTAR), and Support Vector Machines combined with TF-IDF features. However, some recent studies explore using neural networks and deep learning (e.g., [3,29]).

This work focuses on applying TAR to aid abstract screening for systematic reviews. In this field, state-of-the-art systems can find (nearly all) after screening 5 -40 % of the corpus by using this general methodology [8,9], but performance is dataset and query dependent. A frequently used metric to assess the efficacy of TAR systems metric is Work Saved over Sampling (WSS) which indicates the work savings over the use of random sampling (i.e., traditional screening) [5]. This metric can be calculated after the procedure was terminated after a stopping criterion was triggered or when a recall target has been achieved according to the ground truth; WSS@95, which indicates the the work savings over random sampling at the moment when 95 % recall is achieved, is a frequently used metric for TAR systems targeting Systematic Literature Reviews (inter alia [23,8,9]).

In contrast to the AL-based methods, after the popularization of generative Large Language Models like ChatGPT-3.5 and GPT-4 [30], systems have been proposed that use these models to perform screening tasks. The main approach is to prepare a prompt that delineates the task and specifies the criteria, followed by the title and abstract [16,17,31,18]. Many approaches use ChatGPT-3.5 or GPT-4 [16], several [11,18] use open-source LLMs such as Llama 2 [32]. In [18], a large simulation study is performed to assess the performance of several LLMs on popular TAR datasets (CLEF2017, CLEF2018, CLEF2019) [33,34,35]; however, in this study, the LLM predicts the inclusion status only on the title of the systematic review, not its screening protocol (the CLEF datasets do not offer a lot of information on the screening protocol, although the keyword searches are available and a topic description is available). Contrary to the other methods, [18] compares the next token probabilities of yes and no (which are used to indicate the inclusion decision), which can be used as a measure of confidence.

There have been several works that combine or compare LLMs and Active Learning. For example, in [36], the authors compare the performance of LLMs and models that have been trained with Active Learning. One of the findings is that with a limited number of labeled documents, the AL-trained models outperform the LLMs that perform zero-shot classification despite being significantly smaller in terms of training parameters. In [37], a method is proposed that integrates an LLM as an annotator for the creation of Named Entity Recognition (NER) models in underrepresented languages (e.g., African languages). Another work presents a method that generates synthetic data with LLMs, which are used to select the most interesting examples from the pool of unlabeled documents [38].

In [21], the authors present a method that combines AL with Weak Supervision and Transfer Learning. They present their results on training a classifier for classifying financial transactions (text data) in the presence of a black-box model (BBM) (a rule-based system). In this study, an annotator model is trained on agreement labels between the black-box model and the oracle's labels for each iteration along the typical classifier. The annotator model is used to determine per selected instance if the BBM's label can be trusted and accepted or if the human oracle should label it instead. With this method, the authors show that they could significantly lower annotation costs while retaining an accuracy close to the traditional AL setting.

Methodology

In this section, we describe the general architecture of our method. Our TAR procedure consists of two main components: a method to obtain classifications from the LLM and an Active Learning procedure that is used to rank the records during the review phase. Our AL procedure, LLM+CAL, uses the results of the LLM to reduce the review workload further.

Obtaining LLM classifications

In [31,16], a prompt contained the task and the full screening protocol. The task for the LLM was then to answer only with a final inclusion decision (e.g., choose between INCLUDE or EXCLUDE). This setup can be regarded as a black-box system, as it is impossible to determine any of its reasoning for making the decision. Also, the LLM does not provide any information about the confidence in its prediction besides a probability of predicting the token that represents the word INCLUDE or EXCLUDE over the space of all possible output tokens.

Chain-of-thought prompting is a method to improve the accuracy of LLMs when performing complex reasoning. With this method, it is specifically requested in the prompt to think step-by-step in addition to a few examples of appropriate answers. The aim is to let the LLM reason about its "thought process" verbosely, which results in a higher probability that the final answer is correct [39]. By adjusting the prompt to let the LLM respond with chain-of-thought steps in a structured way, we aim to make the process more transparent for the reviewer. In addition, we ask the LLM to provide rationales (i.e., select fragments cited directly from the record in question), which enables tracing the decision to the source document. In Figure 1, we display the prompt template that we use in our experiments, which containsbesides the instruction -a few examples of appropriate answers. We wrote a parser that parses the LLM answer into a structured datatype. In a real-world application, the rationales can be used to highlight fragments in the abstracts used in the LLM's decision-making, enabling easy verification and correction for the end-user in an annotation interface. Another significant difference between the studies in earlier work and ours is that we consider each criterion in the protocol separately. We noticed many classification errors in initial experiments when the whole screening protocol was considered. We list some major error categories below:

Hallucination. The model makes up factually incorrect but seemingly plausible answers.

Missing knowledge or context. The model does not know enough information about a topic that a human reviewer might know (e.g., technical jargon)

Incorrect reasoning. The information extraction works correctly, but the inclusion rules are not followed, causing a misclassification.

Ignoring instructions. Only a part of the screening protocol was used according to the LLM's chain-ofthought response. Some LLMs have problems following all instructions in the prompt, especially when the instructions are long and complex. Larger models like GPT-4 are less prone to this but have a higher computational and financial cost.

Often, the LLM followed the protocol partially: consider a dataset with four criteria, the LLM considered three criteria correctly but mistakenly ignored one of them, causing a misclassification of ASSIGNMENT: You are a helpful assistant who helps screen abstracts and titles of scientific papers. You answer questions by citing evidence in the given text followed by a YES or NO or UNKNOWN decision. When there is no evidence in the title and abstract, decide with UNKNOWN. Only answer with NO if there is absolute evidence given that the answer is NO. In the absence of evidence or when nothing is mentioned, always answer UNKNOWN. Use the following format: REASONING: (Think step by step to answer the question; use the information in the title and abstract and work your way to an answer. Your full reasoning and answer should be given in this field) EVIDENCE: (List sentences or phrases from the title and abstract used to answer the question in the previous field. Answer in bullets (e.g., -"quoted sentence"). Each quoted sentence should have its own line. If there is no evidence, write down []). In this field, only directly cite from the TITLE and ABSTRACT fields. DO the whole instance due to a mistake. This setup makes it challenging to detect failures due to a specific criterion. Mistakes become only apparent by combing through the (semi-structured) LLM answers containing information on all criteria.

We aim to mitigate this by considering each criterion separately, making the set of instructions shorter and less complex, which results in a higher accuracy. The system can then infer the inclusion status of a record by applying a simple logical formula to the model's decision on the criteria (for example, Figure 2).

Despite the reduced complexity, it is still possible that the LLMs make classification errors, for example, due to hallucination, possibly because of missing knowledge. We hypothesize that these errors will not always happen at random, especially for the latter cause. Suppose the LLM makes an incorrect classification for a specific criterion due to missing knowledge. In that case, the LLM will likely make a similar mistake for instances similar to the one in question. Collecting the rationales and chain-of-thought fragments of misclassifications and training models on them might aid in predicting when the LLM makes a mistake or a correct decision.

We used LangChain [40] to build our LLM classification pipeline. This package enables us to target multiple Large Language Models. In our experiments, we only worked with ChatGPT-3.5 (specifically, version 0301); however, the method can be applied to GPT-4 or models of other vendors, such as open-source models published on repositories like HuggingFace [41].

Active Learning method

As in canonical TAR, we represent each document as a high-dimensional vector. A typical feature extraction method is a bag-of-words method like TF-IDF that TAR systems frequently use. Combining sparse feature matrices and classical machine learning methods offers fast retraining and reranking of the documents in 𝒰. The AutoTAR baseline uses TF-IDF combined with a Logistic Regression classifier. In our approach, we will also use TF-IDF and Logistic Regression to ensure that changes in performance are not due to changes in the document representation.

During the process, the labeling task is specified as follows: we have a feature space 𝒳 tiab , which contains the feature vectors of the title-abstract (tiab) records. Each document presented to the oracle gets, for each of the criteria (see Figure 2), a label in the space 𝒴 crit 𝑖 = {+, ?, ¬} corresponding to True (Yes), Unknown, False (No). The option Unknown is vital in this phase, as it is not always the case that the information needed to determine eligibility for a criterion is present in the title and abstract.

Our method, LLM+CAL, consists of two phases: the first phase is called LLMPreferred, which is -in essence -a version of the method AutoTAR, but in this version constrained to select from the unlabeled documents that are included by the LLM (𝒰 ∩ ℒ {+,?} LLM ). As initial training data, the whole screening protocol is given in addition to a random sample of 100 LLM-excluded documents (ℒ − LLM ). This phase is applied until 25 consecutive irrelevant documents are proposed, which might indicate that the set of relevant documents may be exhausted.

Because the possibility exists that there are relevant documents that the LLM does not find, we will switch to the CriteriaWSA method, which can query all documents within 𝒰. First, all labeled data ℒ from the first phase is transferred to this method. Then, several machine learning models are trained: Inclusion Judgment Classifier. A Binary Classifier trained on the labeled data after transforming the data to 𝒴 binary = {+, ¬}, trained on the data in ℒ, in a similar fashion as AutoTAR. The criterion judgments are transformed using the formula specified in Figure 2, which will result in a label in the space 𝒴 ternary = {+, ?, ¬}. We can then transform 𝒴 ternary to 𝒴 binary by changing each ? into a +.

Acceptance Classifier. A Binary Classifier that determines Acceptance for each inclusion criterion. This is similar to a method presented in [21]. Here, for each criterion 𝑖, we obtain binary agreement labels 𝑧 ∈ 𝒵, where 𝒵 = {0, 1}. This is determined by comparing the LLM predictions and the labeled data in ℒ 𝑖 : each instance receives a label Accept (1) if the LLM prediction agrees with the human-annotated label. Otherwise, the label Reject (0) is given. However, contrary to the other models in our system and the method in [21], the model is not trained on the Title-Abstract records (𝒳 tiab ), but on the LLM's reasoning fragments 𝒳 ans 𝑖 (see Figure 4 for example data) of criterion 𝑖.

Given a TAR task that has four inclusion criteria ({𝑎, 𝑏, 𝑐, 𝑑}), we obtain the following pairs for each labeled for each labeled record:

• 𝒳 tiab × 𝒴 crit𝑎 × 𝒴 crit 𝑏 × 𝒴 crit𝑐 × 𝒴 crit 𝑑 • 𝒳 tiab × 𝒴 binary • 𝒳 tiab × 𝒴 ternary • 𝒳 ans𝑎 × 𝒵 𝑎 • 𝒳 ans 𝑏 × 𝒵 𝑏 • 𝒳 ans𝑐 × 𝒵 𝑐 • 𝒳 ans 𝑑 × 𝒵 𝑑

During each annotation round, a batch of ten documents is given to the oracle using relevance sampling based on the ranking produced by the inclusion judgment classifier. The batch size of ten is an initial default value for this parameter. Smaller, larger, and dynamic batch sizes can be explored in future work. Another ten documents are sampled based on a ranking that is based on the predictions of the LLM and the Acceptance Classifier using the following equation:

score 𝑖 (𝑦 ^LLM 𝑖 , 𝑝 acc 𝑖 ) = ⎧ ⎨ ⎩ 0.75 + 0.25𝑝 acc 𝑖 if 𝑦 ^LLM 𝑖 = + 0.5 + 0.25𝑝 acc 𝑖 if 𝑦 ^LLM 𝑖 = ? 0.5(1 − 𝑝 acc 𝑖 ) if 𝑦 ^LLM 𝑖 = ¬ . (1)

Equation 1 is calculated for each study criterion 𝑖, where 𝑦 ^LLM 𝑖 is the LLM's prediction for criterion 𝑖 and 𝑝 acc 𝑖 is the corresponding acceptance probability. The mean of those scores is calculated for each of the unlabeled documents. Then, this score is used to rank the remaining documents in 𝒰. The rationale behind Equation 1 is that instances with a higher probability to be relevant (instances with criteria that have more True labels) are put before documents that have Unknown labels, followed by documents that have False labels. Labels that have False labels and a low acceptance probability will have a higher probability of being selected than documents with False labels that are certain. For the True and Unknown labels, the inverse holds if there is a higher acceptance probability, they are preferred over instances with lower acceptance probability. This is still an initial formulation that may not always work optimally; other options can be explored in future research.

After this batch of twenty documents has been prepared, they are given to the oracle for labeling unless the LLM has found exclusionary evidence for a specific criterion and its acceptance probability is above 80 % (unless that criterion is a reason for exclusion for all remaining documents in 𝒰); these examples are skipped but may be proposed again in another round if the acceptance probability drops below 80 %.

This process is repeated until a stopping criterion is triggered, the oracle decides to stop the review, or 𝒰 is exhausted. In our experiments, we will stop querying after reviewing |ℒ {+,?} LLM | documents.

Case Study

In this work, we compare the performance of various TAR methods on a dataset that is collected for a systematic review (at the time of writing in preparation) that aims to identify common latent groups or classes of PTSS/PTSD (Post-traumatic Stress Symptoms / Post-traumatic Stress Disorder) trajectories, as well as their prevalence and predictors, which may give a better understanding how and under what circumstances PTSS/PTSD presentations may develop [42]. For this purpose, researchers reviewed a large corpus of records after querying several databases. During the review, the records were labeled on various levels, which we list below.

Inclusion Criteria:

𝑎 : Is the study a longitudinal/prospective study with at least three time point assessments? 𝑏 : Does the study assess PTSD symptoms as a continuous variable? [Followed by a list of eligible scales] 𝑐 : Does this study mention that individuals are exposed to traumatic events? 𝑑 : Did the study conduct a PTSD trajectory analysis? [Followed by a list of eligible methods]

A study 𝑠 can be included in the review when all criteria are satisfied (so, ∀𝑠 ∈ 𝒟 + , 𝑎(𝑠) ∧ 𝑏(𝑠) ∧ 𝑐(𝑠) ∧ 𝑑(𝑠)). Title. Some documents can be excluded by considering the title only. For example, animal studies are never eligible, and the fact that a study is an animal study can become clear from reading the title. We only study the records that have not been excluded by title screening.

Criterion. The eligibility of a study for inclusion depends on four inclusion criteria (see Figure 2). For each criterion 𝑖 ∈ {𝑎, 𝑏, 𝑐, 𝑑}, a label 𝒴 crit 𝑖 = {+, ?, ¬}, corresponding to True, Unknown, False can be given. In Figure 3, some statistics per criterion are displayed.

Title Abstract. Using the logical formula in Figure 2, an inclusion judgment can be made for each criterion, so this level can be derived from the criterion level without additional human effort. This will result in a label in the space 𝒴 ternary = {+, ?, ¬}. Because an instance can have an Unknown label for one or more criteria, the final eligibility of such a study must be determined by reading the entire paper without exclusionary evidence in the record.

Full-text level: Final eligibility depends on reading the full-text of the study. This level is not considered in this work because this label needs more information than is available in this dataset (i.e., the full-text of every record).

This dataset is unique compared to other frequently used datasets used for benchmarking TAR systems (e.g., [33,34,35]) have only binary inclusion information, sometimes only on the full-text level. Moreover, while these datasets are based on real-world search tasks, there is little to no information about the inclusion/exclusion criteria available. The SYNERGY [43] corpus consists of several systematic reviews (including an earlier version of the PTSS dataset [44]) with links to the publications from which the screening protocols can be obtained. Unfortunately, only inclusion labels on the full-text level are included, so we cannot study retrieval efficacy fairly (we can only consider recall of the set of papers that are included based on the full-text, which is a subset of the Title-Abstract included papers; therefore, we cannot distinguish title-abstract inclusions from the false positives). To our knowledge, the dataset used in this case study (for the systematic review in [42]) is the only systematic review with labels on the criterion level.

We will consider only the records of one reviewer after title screening here, which results in a set of 4836 records after some data cleaning. Our dataset then contains |𝒟 {+,?} | = 183 records that are included on the title-abstract level, resulting in a prevalence of 3.78 %. One observation that can be drawn from Figure 3 is that criterion 𝑑 determines the title-abstract inclusion label (displayed as judgment) the most. , which shows some of the relations between the labels, for example, that only a tiny subset of the documents for which 𝑏 is False, criterion 𝑐 is True. Also, it becomes clear that criterion 𝑑 excludes the most documents of all the criteria.

Experimental evaluation

We compare several methods in a small simulation study on the dataset described in the previous section.

• AutoTAR, a state-of-the-art TAR method, • The LLM Classifier, as described in Section 3.1, • LLM+CAL, our AL method that integrates the predictions of the LLM Classifier, as described in Section 3.2).

In this study, we only compare retrieval efficacy as we leave the question of a good stopping criterion open. Therefore, we constrain the run to the number of documents that are predicted by the LLM to be still eligible for inclusion (i.e., the number of documents with the label for which the inclusion judgment prediction is True or Unknown, |ℒ {+,?} LLM |). We let each algorithm run until this number is reached. Then, we can compare the performance of the LLM classifier and the AL-based methods with the same review effort. During the experiment, we will record when various recall levels are triggered. We will record the following metrics (calculated in the space 𝒴 binary ).

Recall. The percentage of relevant documents found based on the a priori knowledge from the ground truth dataset.

𝑅 = |ℒ + | |𝒟 + |(2)

Work Saved over Sampling. This metric expresses the work reduction over random sampling [5].

We calculate this as follows. We will record this value for several recall targets:

𝑊 𝑆𝑆 = |𝒰| |𝒟| − (︂ 1 − |ℒ + | |𝒟 + | )︂(3)

Equation 3 is used in the AL setting. In the context of a classifier, we equate 𝒰 to the set of documents predicted to be irrelevant (the reviewers do not read those documents). For the LLM Classifier, we can adapt the equation as follows.

𝑊 𝑆𝑆 = |ℒ − LLM | |𝒟| − (︂ 1 − |ℒ + LLM | |𝒟 + | )︂(4)

The rest of the section is structured as follows: first, we describe the results of the LLM classification, followed by the results of a simulation study in which we compare the aforementioned AL-based TAR methods.

LLM Classification results

In Figure 4, we display an example of an annotated record. After parsing the response, we can highlight the fragments the LLM used in its decision-making. This overview is available for every instance in the dataset. When used in an annotation interface, the LLM explanations might aid users in their decision-making process, possibly reducing the screening time per document.

In Table 2, confusion matrices per criterion are displayed. A clear observation from Table 2 is that the LLM is more cautious in excluding papers than the human reviewer: the confusion matrices show high numbers of studies with ground truth False and predictions Unknown for all criteria. One of the causes is that when there is no written evidence to make a decision about a criterion, for example, whether or not a PTSD trajectory analysis (criterion 𝑑) was performed, the LLM would predict Unknown. This might seem like the correct decision in this situation. However, experienced human reviewers might exclude a paper based on their knowledge of the field by inferring that from other characteristics (for example, when the abstract describes a methodology that makes it impossible to use one of the eligible methods).

The LLM's definition of specific terms or the meaning of concepts might diverge from the reviewers'. For example, for criterion 𝑐, in some cases, the LLM eagerly infers from the descriptions of the studied populations that these might be exposed to trauma, which might not explicitly be mentioned in the record. Fortunately, the number of falsely excluded documents per criterion is low.

When combining the LLMs prediction, we can infer the title-abstract level predictions using the logical formula specified in Figure 2. In Table 2, the confusion matrix for this level is displayed, both on the ternary and binary levels. On this level, we obtain an accuracy of 78.52 % (ternary level), with a recall of 91.26 % on the binary level. In absolute numbers, this results in the fact that only 16 studies were missed out of the 183. The precision on the binary level is 12.9 %, resulting in a Work Saved over Sampling of 64.48 % (with Equation 4). Depression: 2.26, 0.51-4.02; PTSD: 6.83, 4.21-9.44). CONCLUSION: Refugee women who experienced past-year intimate partner violence or conflict-related violence may be at increased risk of depression, anxiety or PTSD. Service providers should be aware that compared to the general refugee population, women who have experienced violence may require additional psychological support and recognise the enduring impact of violence that occurred before, during and after periods of conflict and tailor outreach and treatment services accordingly.

Reasoning:

¬𝑎 The title and abstract do not mention anything about a longitudinal/prospective study with multiple time-point assessments. As there is no evidence to suggest that the study is longitudinal/prospective with at least three-time point assessments, the answer is NO. ?𝑏 The question asks if the study assesses PTSD symptoms as a continuous variable and provides a list of eligible and ineligible PTSD scales. The abstract mentions that the study modelled the relationship between groups of women and symptomatic scores for anxiety, depression, and PTSD using linear regression. However, the abstract does not mention any specific PTSD scales used in the analysis. Therefore, it is not clear whether the study used any of the eligible or ineligible PTSD scales provided in the question, and the answer should be UNKNOWN. 𝑐 The study examines the association between gender-based violence (GBV) and mental health, including anxiety, depression, and post-traumatic stress disorder (PTSD), among women in a Kenyan refugee camp. The abstract mentions that conflict-affected settings put women at risk for GBV, which is associated with poor long-term mental health such as anxiety, depression, and PTSD. The study also found that conflict-related violence was an important risk factor for poor mental health among women who accessed GBV services, despite time since occurrence. Therefore, it is likely that the individuals in the study were exposed to traumatic events. ¬𝑑 The study aimed to investigate the relationship between GBV and mental health among Somali women in a Kenyan refugee camp. However, the methods section does not mention conducting a PTSD trajectory analysis.

Therefore, the answer is NO.

Figure 4:

Here, we display an annotated record (title and abstract from [45]) from the case study. This figure was generated by processing the responses generated by ChatGPT-3.5 to the prompts that were created for this record. The LLM cited the highlighted fragments as rationales for making the decision. Fragments containing evidence to include the record based on the criteria in Figure 2 are highlighted in green. Fragments highlighted in red are used to exclude a record. Yellow highlights indicate contradictory evidence, meaning the information is used as evidence for inclusion for one criterion and exclusionary evidence for another. Below the abstract, the reasoning of the LLM is listed per criterion. (Note that the breaks between highlights are automatically added to prevent overflowing lines during typesetting.) Run on a dataset with 183 inclusions out of 4836 100 % recall (183) 95 % recall (174) Exp. found at random # by AutoTAR WSS@60 -54.1 % WSS@65 -58.6 % WSS@70 -63.6 % WSS@75 -67.8 % WSS@80 -70.9 % WSS@85 -73.5 % WSS@90 -72.7 % WSS@95 -71.6 % Run on a dataset with 183 inclusions out of 4836 100 % recall (183) 95 % recall (174) Exp. found at random # by LLM+CAL # by LLMPreferred # by CriteriaWSA WSS@60 -56.8 % WSS@65 -61.5 % WSS@70 -66.3 % WSS@75 -69.8 % WSS@80 -74.0 % WSS@85 -77.9 % WSS@90 -81.3 % WSS@95 -81.6 % (b) LLM+CAL (ours)

Discussion

We have shown some preliminary results on our method, which indicate that adding LLM predictions is beneficial to obtaining relevant documents at a lower cost than with the state-of-the-art method AutoTAR as our LLM+CAL method yields higher work savings at several recall targets. Moreover, a reviewer could achieve a better recall and WSS than obtained using only the LLM classifier. We have presented a system that builds upon earlier LLM methods for Systematic Literature Reviews by making the predictions more fine-grained by addressing each inclusion criterion separately. Moreover, our approach aims to make the predictions more accurate and explainable by leveraging chain-of-thought reasoning and asking the LLM to cite from the title-abstract record directly. Our method takes some ideas from [21] in combining AL and the noisy labels from, in our case, an LLM annotator. We evaluated our method on a single dataset, which may impact the generalizability of our results. Unfortunately, testing on more datasets is not feasible at the time of writing, as our method requires that the dataset has criterion-level labels. It may be interesting if we can adapt the method to work with feedback on the binary inclusion level, which might enable us to consider more datasets that do not have labels this fine-grained. Another interesting avenue is comparing the performance of our method on different LLM results than presented here. The LLM predictions may slightly differ when another model is used or when alternative formulations of inclusion criteria and general instructions are used. Further investigation is needed to determine what impact non-optimal instructions have on the LLM's accuracy and the ability of our method to correct lower-quality weak labels.

The method we presented here is still relatively simple; several extensions can be made that might further improve the efficacy. For example, incorporating Transfer Learning (as in [21]). Another area that can be explored further is the sampling strategy. Currently, our sampling strategy is based on a binary Logistic Regression classifier and TF-IDF features (as in AutoTAR). Considering other classifiers like Neural Networks and text embeddings like SentenceBERT [46] might yield additional performance gains over traditional methods.

We currently do not use the criterion-level labels during model training and subsequently rank documents in 𝒰 with those models. Designing a good method that combines the results of the four classifiers in a ranking is not trivial. Equation 1 is a starting point (now applied to LLM only) but not optimal. Relations between criteria have also not been taken into account yet. For example, assume a scenario where, within nearly all labeled records in ℒ, the proposition 𝑎 ∧ 𝑏 ∧ 𝑑 → 𝑐 holds. When, for a new instance, the LLM predicts the following labels {𝑎, 𝑏, ¬𝑐, 𝑑}, this record may be an interesting example to review for the oracle because it is an exception to what has been seen so far.

So far, the LLM rationales have not been used to train the classifier. In [47], (human annotated) rationales were used as additional training data besides 𝒳 tiab for TAR for Systematic Literature Reviews, suggesting it might be beneficial to consider the LLM rationales during training as well.

As mentioned before, we have left the question of a stopping criterion open. One avenue could be to combine the method with an existing stopping criterion or to use the LLM predictions to determine an optimal stopping point.

During a review, regardless of whether it is performed in the traditional setting or with TAR, labeling mistakes occur due to human error [7,26]. As in [21], our method assumes that the oracle always makes the correct decision; however, this may not always be the case. Presenting the LLM rationales and chain-of-thought fragments (like in Figure 4) may help the oracle to make better decisions and prevent some mistakes, but the extent of this has to be further investigated. Also, the Active Learning part of our method could be adapted to consider the possibility of human errors.

We believe several ideas presented here might also benefit research areas other than TAR. For example, the LLM framework presented here can be applied to text classification tasks in general. However, adapting our method to a canonical AL setting is more appropriate in this setting. The framework we presented here enables obtaining weak labels at a low cost, with little engineering effort besides writing a good labeling protocol, and chain-of-thought prompting may aid in spotting errors within them, enabling more efficient creation of text classification models.