Although large language models (LLMs) are achieving impressive performance under zero- and few-shot learning configurations, their reasoning capacities are still poorly understood. As a step in this direction, we present several experiments on multiple-choice question answering, a setting that allows us to evaluate the stability of the model under diferent prompting, the capacity to understand when none of the provided answers is correct, and to reason on specific answering strategies (e.g., recursively eliminate the worst answer). We use the Italian medical specialty tests yearly administered to admit medical doctors to specialties. Results show that a gpt-3.5-turbo model achieves excellent performance in the absolute score (an average of 108 out of 140) while still sufering in certain reasoning capacities, particularly in failing to understand when none of the provided answers is correct.
conferences, and other scientific venues.
Instruction-tuned Large Language Models (LLMs) have analyzing results, stability, and consistency in Italian. We recently shown unprecedented results in various tasks do so by considering a case study: the Italian medical in diferent languages [ 1]. Beyond the impressive perfor- specialty tests. The test consists of 140 multiple-choice mance, their popularity derives from the possibility of us- questions in various medical areas, based on which Italing them with no or little training data for multiple tasks ian medical doctors are evaluated and ranked if they and languages. In fact, instruction-tuned LLMs go be- want to enroll in a medical specialty school. We chose yond the previously established learning paradigm based this test for several reasons. First of all, we believe the on transfer learning — where a model, first pre-trained task is intrinsically dificult. It requires domain-specific with no supervision, must be fine-tuned on downstream knowledge that doctors are expected to acquire after a task-specific data — and are typically used in a zero- or six-year-long university career; moreover, the test confew-shot manner. tains both fact-based questions (for example, the criteria
LLMs performance and ease of use have attracted in- for a diagnosis) and clinical cases, which also require reaterest from Natural Language Processing researchers and soning capabilities (for example, to decide on the most practitioners. However, most previous work has focused appropriate intervention given some symptoms). On the on the models’ performance and practical applications. other hand, the structured nature of the test makes it Less relevance has been given to the models’ stability more robust to the specific prompts used and allows us and reliability, e.g., in the variability of their outputs to measure performance easier and more reliably. or reasoning capacities in controlled settings. This is We perform experiments by using ChatGPT. This even more problematic since many of the most popular choice is due to several reasons: firstly, the model is and performative instruction-tuned LLMs are proprietary, undoubtedly very popular at the time of writing; secand the details of the exploited data, architecture, and ondly, according to our preliminary experiments, its pertraining procedures are at best supercfiially discussed in formance is superior to those of other open-source LLMs technical reports [2] rather than proper research papers, available, e.g., Alpaca [3]. While we are aware of the limCLiC-it 2023: 9th Italian Conference on Computational Linguistics, itations linked to the proprietary nature of the model, we Nov 30 – Dec 02, 2023, Venice, Italy believe its black-box nature, combined with its popular$ silvia.casola@unito.it (S. Casola); tlabruna@fbk.eu (T. Labruna); ity and practical importance in NLP-related applications, lavelli@fbk.eu (A. Lavelli); magnini@fbk.eu (B. Magnini) make an analysis of its capabilities, limitations, and sta0000-0002-0017-2975 (S. Casola); 0000-0001-7713-7679 bility even more urgent. (T. Labruna); 0000-0002-7175-6804 (A. Lavelli); 0000-0002-0740-5778 (B. Magnini)
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org)
In our work, we benefit from recent approaches to We collect questions from the 2022 Italian medical speinstruction-tuning of LLMs, out of which several prompt- cialty test. The test contains 140 short questions in Italbased techniques have been developed. ian, each with five possible answers. Only one answer is correct. A small fraction of the original questions reInstruction-tuned LLMs. In recent years, LLMs have quire considering a picture (e.g., an ECG or a medical been the focus of extensive research due to their ability image). We remove those questions. This leaves us with to learn from large amounts of data in a self-supervised 136 questions. Since we have collected questions and fashion and to achieve impressive results in various tasks corresponding correct answers from a published solu[4, 5]. A recent trend in utilizing LLMs is the development tion (where the correct answer was always the first), we of prompt-based techniques, where a textual prompt is randomize the order of the answers. Unless otherwise given to the model as input to generate the desired output. specified, the order of the answers is consistent for all Such techniques have shown to be highly efective, espe- experiments. cially for tasks that require specific outputs and have the After constructing a prompt, we input it to a gpt-3.5advantage of (i) not requiring any parameter updates in turbo model with 4K tokens of context. We set the temthe LLM; (ii) being human readable, and (iii) not requiring perature to 0 to avoid hallucinations and leave all other in-domain data, unlike fine-tuning techniques. An exam- parameters at their default value. Unless otherwise specple of such a model is GPT-3.5, a pre-trained language ified, the prompt is inputted through a user role, and no model that uses the Transformer architecture and an at- system role is used1. tention mechanism to generate natural language text. For We measure the model’s performance using accuracy; an extensive survey on prompt-based techniques, refer we also compute the associated test score (normalized to to [6]. Prompting has led to a shift from objective en- 140 answers to be comparable with human performance), gineering to prompt engineering: this includes both the which assigns one point to correct answers, -0.25 to inmanual design of templates [7] and automatic prompt correct answers, and 0 to unanswered questions. learning [8], as well as various options to ensemble [9] and compose [10] multiple prompts.
and ChatGPT. We leverage the “gpt 3.5 turbo” model, which is the basis of the interactive interface of ChatGPT To measure the model’s baseline performance on our [11], and part of the InstructGPT family [12] based on the task, we construct a simple prompt (see Example 12). GPT-3 language model [13]. Unlike standard GPT-3 mod- Since doctors are allowed not to answer questions for els, however, InstructGPT models are optimized for inter- which they do not feel confident enough, we also experiactive use, are particularly suited to take instructions as mented with adding an option (5-choice + IDK) to allow input prompts, and can modify their outputs when asked the model not to choose any of the options (F: I do not in a dialogue, making them more aligned with users’ re- know or there is not enough information to answer the quests. This is accomplished by a reward mechanism, question).
Reinforcement Learning from Human Feedback (RLHF) Finally, we also experimented with allowing the model [14] used to optimize the model. After unsupervised to select an answer according to which none of the propretraining, conversation data – generated by human vided answers were correct (F: None of the previous antrainers who act as both the user and the AI assistant – swers is correct). were collected; the model was then fine-tuned through Table 1 reports the results. For the cases in which supervised learning. Given several possible model re- the model was allowed, we also report the number of sponses to each prompt, human annotators ranked the questions for which it chose not to answer or to answer desirability and alignment of each response; a reward that none of the options were correct. model was thus trained to mimic their preference. Finally, the reward model was used to further fine-tune the LLM, making it more aligned with human preferences.
Taking advantage of the multilingual pretraining at the base of the GPT-3.5 models, ChatGPT is also available for Italian. 1Three diferent roles can be specified through APIs: ‘assistant’ i.e. the model (used to show expected responses in a chain of interactions); ‘system’ (used to give “developer-like" instructions and modify the overall behavior of the model), and ‘user’ (the user that is interacting with the model).
we will only report the English translation in the continuation of this paper.
mato json. Per esempio {“lettera”: <la tua scelta>}.
utare la gravità di un paziente afetto da cirrosi epatica?’
{ "lettera":"A", "contenuto": "GCS"} { "lettera":"B", "contenuto": "Chads-VASC"} { "lettera":"C", "contenuto": "ABCD"} { "lettera":"D", "contenuto": "Child-Pugh"} { "lettera":"E", "contenuto": "Curb-65"}
format. For example {“letter”: <your choice>}.
assess the severity of a patient afected by liver cirrhosis?’
{ "letter":"A", "content": "GCS"} { "letter":"B", "content": "Chads-VASC"} { "letter":"C", "content": "ABCD"} { "letter:"D", "content": "Child-Pugh"} { "lettera":"E", "content": "Curb-65"}
5-choice 5-choice + IDK 5-choice + None Acc. 81.62 77.94 78.68 Score 107.83 101.4 103.20 ? – 0 –
None – – 2
perature to 0, asking the model to repeatedly answer to the same exact prompt (free of modifications of any sort) can result in diferent outputs. To measure this efect, we ask the model to answer the test given the same inputs 5 times.
rable to that of the best-performing doctors.
Considering the minimal score needed to be admitted in diferent specialties in 2022, this performance would be suficient to be admitted in all but one medical specialty school (Dermatology) in at least one of the Universities Stability to the order of the answers. We want to ofering such specialty and to be able to choose among test whether changing the order of the given option afall University for 38 specialties (among the 51 available). fects the result and to measure the magnitude of such an efect. To do so, we show the same prompt to the 4.2. Stability model (see Example 1), but we randomly change the orIn this section, we consider the model’s stability, with a der of the answers for each run. Results are in Table 3. focus on the consistency of the results. While not dramatic, we notice that the diference in the order corresponds to a visible diference in accuracy. We also noticed that, in just 2 of the 5 runs, one question3 is blocked by the model due to the prompt triggering Azure
efect of using diferent prompts on the results. To this
Outputs were not consistent between runs. In most 3The question regards the correct action a family doctor has to take cases, the diferences were cosmetic (e.g., some answers when a person dies at home.
P1
P2 (9, 0)
P3 (8, 1) (11, 1) reported only the key “letter" with the corresponding letter answer in the output, while others also reported the key “content" with the corresponding answers). In several cases, however, the diferent runs correspond to diferent answers to the same questions.
Table 2 reports the experiment results. For each pair of predictions (, ), we report the number of total cases in which the answers have some diferences and the number of cases in which these diferences correspond to diferent answers. In most cases, this diference has no or negligible efect on accuracy, as mistakes tend to be compensated between runs. be adapted to this setting. Some questions, for example, require a relative judgment4; thus, we first manually selected adequate questions only. This leaves us with 130 questions, for which we build a counterfactual version.
prompt, where we specify that if none of the options seems correct, it must choose option E.
Table 5 reports the results of the experiment. We notice that the model performance drastically decreases in this setting: the model tends to very rarely pick the “No answer is correct" options, resulting in an accuracy that is only slightly above random. end, the authors of this paper constructed prompts independently. Table 4 reports the accuracy and related scores obtained by the diferent prompts.
can be expected, given the structured nature of the task —, there is a diference of more than 5 points in accuracy between the prompts that obtain the best and the worst performance.
To process the answer in an easier and more reliable way, all prompts try to condition the outputs to be structured or semi-structured. Using the first prompt, the output is always a valid JSON file; in 21 cases, however, the JSON does not only contain the letter (as required in the prompt) but also the “content" field (mimicking the way the possible answers are presented). For prompt 2, the output is not a valid JSON in 8 cases and presents other text (often corresponding to the answer text) outside brackets. In all cases, the JSON contains the field “letter" only. While prompt 3 requires the model to output the letter corresponding to the right answer only, in the vast majority of cases the output also included the answer text, e.g., in the format “D. Child-Pugh" (129 cases) rather than “D" (1 case) or “D." (10 cases). For all experiments in this paper, we take into account the correctness even of those outputs that are not perfectly formatted.
els to perform intermediate steps [15] or explicitly encouraging them to do so in the prompt [16] leads to improved performance. This methodology is commonly referred to as Chain of Thoughts (CoT).
While the direct application of this approach to the multiple-choice context is not straightforward, we sought to explore how a multi-step approach influences performance. We experimented with two diferent methods: (1) in a single prompt, we asked the model to remove one wrong answer at each step recursively and to give us the correct answer at the end of the process; the chain of thoughts and the resulting correct answers needed to be 4.3. No Correct Answer provided in the same output; (2) we asked the model to We want to understand whether the model is able to un- identify the answer most likely to be incorrect; we then derstand when none of the provided answers is correct. construct an identical prompt where the model choice
be the correct answer. Not all questions, however, can
carcinoma at the liver level, HER-2 positive (stage IV), which of the following treatments is the most recommended? 4.4. Recursive Reasoning
4.5. Prompt correction was removed and repeated the process until only two op- In all the experiments conducted thus far, we utilized tions were left. In this scenario, we prompted the model human-generated prompts to obtain the results from the 4 times in 4 diferent conversations. model. However, using such prompts introduces biases Examples 4 and 5 show our resulting prompts. Note that, and may not necessarily yield the most optimal results. in the first case, the prompt needed to be overengineered To explore the potential for improvement, we decided and pleonastic as the model was not able to follow instruc- to leverage ChatGPT itself to enhance the prompts. We tions with simpler versions consistently — in some cases, experimented with two diferent approaches: (i) we profor example, it would remove one option only, or output vided ChatGPT with all the human-generated prompts one answer only with no clear indication of whether it and requested it to improve upon them, and (ii) we considered it as wrong or correct. granted ChatGPT the freedom to choose the best prompt The recursive strategy does not seem complementary to independently, without any specific examples, but by the baseline one, as only in one case a question that is merely describing the required task. answered incorrectly by the baseline prompt is answered Both prompt versions are considerably long and elabcorrectly by using elimination. orated if compared to the human-generated ones. The
The results in Table 6 indicate a significant decrease ifrst version is shown in Example 6. The outcomes of in accuracy compared to the baseline experiment. We both approaches are summarized in Table 7. Interestingly, observed that the model particularly struggled to handle the results obtained from ChatGPT-generated prompts the high logical complexity required by understanding closely aligned with those from human-written prompts. the question and intentionally avoiding the correct an- Therefore, this particular approach yielded no significant swer by selecting a diferent one. This challenge was benefits, as the performance remained consistent with particularly evident when our request was performed on the original prompts. questions that themselves asked to identify the wrong option among the given ones; in fact, the model failed to recognize the need for a double negation. 5. Conclusions
and reasoning capacities of an LLM on a multiple-choice question-answering task in the medical domain and for
havior: the stability of the model (e.g., repeated questions, stability under diferent prompts and under different orders of answers), the capacity to understand counterfactual reasoning (e.g., when all answer choices are incorrect), the capacity to manage specific answering strategies (e.g., recursively eliminating wrong answers). Results show that a gpt-3.5-turbo model achieves excellent performance in terms of absolute score (an average of 108, out of 140), which is surprising given the technical nature of the test. The model is also relatively stable under diferent prompts. The model was also able to interpret and manage prompts asking to perform recursive reasoning, even though the resulting performance is considerably worse than the baseline. The major weakness that was found is related to understanding when none of the provided answers is correct, as the model performed only slightly better than random.
(enabling Clinical Research in Emergency and Acute care
expressed are however those of the author(s) only and do not necessarily reflect those of the European Union.