The current generation of artificial intelligence (AI) systems ofers tremendous benefits, but their efectiveness is limited by the inability of the machine to explain its decisions and actions to users. My dissertation will delve into the subject of explainable AI (XAI), exploring its various aspects and implications. I will focus on post-hoc local explainability of large AI models to provide human-readable explanations making users understand the automated decision-making of complex models. I aim to evaluate current large language models (LLMs) like ChatGPT or Llama on explainability and its implications, e.g., on education. My thesis also focuses on questions in the field of reasoning with LLMs, since reasoning is fundamental in LLMs for enhancing their understanding and generation of text, improving problem-solving capabilities, and facilitating natural human interaction. However, reasoning is a very dificult task for a computer and the capacities of LLMs regarding diferent reasoning tasks are not yet fully examined.
Reasoning stands as a core component of human intelligence, essential for problem-solving,
making decisions, and critical thinking. Recently, advancements in large language models
have made significant progress in the field of NLP, suggesting that these models might possess
reasoning capabilities, especially as they increase in size [
At the moment LLMs like ChatGPT are dominating AI and achieved remarkable results in various
tasks. In the education sector for example, students use this new technology for assignment
writing among other tasks [
According to the previous context and motivation the primary question of my research is: To what extent can users and developers benefit from post-hoc local explainability of large AI models?
• Can LLMs like ChatGPT handle commonsense reasoning in question answering tasks with near-human-level performance? • Are LLMs like ChatGPT able to generate good, human-understandable explanations for their decisions? In further research, I aim to study the syllogistic reasoning abilities of LLMs as deductive reasoning is very diferent from the inductive reasoning that is used for commonsense reasoning. Moreover, the goal is to analyse the impact of few-shot learning and chain-of-thought as well as the potential of Retrieval-Augmented Generation (RAG). RAG ofers a way to optimize the output of an LLM with specific information without changing the underlying model itself.
My objective is to rigorously test various LLMs across a carefully chosen set of reasoning tasks, aiming to compare these models’ performances against that of humans. Furthermore, I plan to examine explanations of LLMs for reasoning tasks firstly by humans and secondly with an automated evaluation mechanism employing diverse scoring metrics to assess the interpretability and coherence of these models’ reasoning capabilities. This comprehensive evaluation strategy seeks to illuminate the strengths and inadequacies of LLMs in mimicking human-like reasoning and understanding.
In my initial study on this subject, I delved into commonsense reasoning, using the LLM ChatGPT to evaluate 11 benchmark datasets. Employing a questionnaire, I compared ChatGPT’s responses against those of human participants, additionally questioning their evaluations of the explanations provided by the LLM. Later I broadened this inquiry by including additional open-source LLMs, such as Llama-3 by Meta and Gemma by Google, aiming for a deeper comprehension of LLMs’ reasoning competencies. This endeavour focuses on contrasting the reasoning explanations generated by various LLMs for the same tasks.
Beyond commonsense reasoning, my research intends to explore syllogistic reasoning with diferent LLMs to analyse the field of deductive reasoning. Moreover, I’m intrigued by the potential of employing chain-of-thought prompting in reasoning tasks, a method that could further elucidate how LLMs navigate complex reasoning processes. This multifaceted approach promises to shed light on LLMs’ reasoning abilities, paving the way for advancements.
To mitigate issues like hallucinations and allow better customization and scalability across various applications I aim to study RAG. I believe RAG can enhance the models’ ability to provide precise, up-to-date, and contextually relevant information.
In this section, I present the results of my paper Commonsense Reasoning and Explainable Artificial Intelligence Using Large Language Models [14] presented at the European Conference on Artificial Intelligence 2023.
We analysed 11 benchmark datasets specifically curated to challenge solvers lacking commonsense knowledge. We randomly select 30 examples from each dataset. These tasks span diverse domains, including medicine, physics, and scenarios from daily life. Our evaluation of ChatGPT’s capabilities using these QA benchmarks reveals a spectrum of performance outcomes. Notably, ChatGPT’s weakest performance was observed on the CommonsenseQA dataset, where it achieved an accuracy of 56.67%, while its strongest performance was recorded on the Story Cloze Test, reaching an impressive accuracy of 93.33%. A detailed representation of the performance on each of the eleven datasets is shown in Table 1. Over all datasets ChatGPT answered with an accuracy of 73.33%, 77 tasks were answered incorrectly (23.33%), and we did not get a valid response for 11 QA tasks (3.33%). Not valid means that ChatGPT does not respond which answer option is correct and instead asks for further context information. We conducted an error analysis and found that there are six kinds of problems where Chat- GPT still struggles with:
2. comparative reasoning 3. subjective reasoning 4. slang, unoficial abbreviations, and youth language 5. social situations 6. medical domain In our extensive questionnaire with 49 participants we found that the participants answered 73.72% of the 20 QA tasks correctly compared to ChatGPT’s 90.00% on the same questions.
Figure 2 compares ChatGPT’s performance to that of surveyed participants across the datasets. ChatGPT outperformed humans on six datasets, while humans excelled in four, notably struggling with CommonsenseQA where ChatGPT also had its lowest performance. The biggest performance gap was observed in the COPA and Cosmos QA datasets, with humans outperforming ChatGPT by 26.47% in COPA and ChatGPT surpassed humans by 19.53% in Cosmos QA. Interestingly, ChatGPT showed strength in Cosmos QA, which requires contextual commonsense reasoning, despite humans significantly outperforming ChatGPT in COPA, which demands an understanding of cause and efect, and selecting the most plausible option. The findings suggest ChatGPT struggles with comparative reasoning where multiple plausible options exist, hinting that traditional explicit reasoning approaches might fare better in such scenarios. We further evaluated the explanations given by ChatGPT with the help of a questionnaire and found that explanations were mostly rated “good” or “excellent” with 67.60% and only 42 times very poor. Explanations were rated “fair” or better with 84.80%. See Figure fig:explanation for more details.
The study demonstrates that ChatGPT achieved a 73.33% overall accuracy rate on eleven QA datasets, which require commonsense reasoning for correct answers. Despite certain issues, ChatGPT managed to surpass our survey participants in six of ten datasets (excluding the MedMCQA medical dataset), suggesting that LLMs like ChatGPT are approaching near-human performance in commonsense reasoning within QA tasks. Additionally, the research also delved into the explainability of LLMs, a critical facet in addressing the opacity of these black-box systems. According to our questionnaire, most of ChatGPT’s explanations were rated “good” or “excellent”, supporting our hypothesis that LLMs are capable of producing high-quality explanations.
A recent extension of this work [26], in which further LLMs have been analysed and a larger human-centred study was conducted, is currently under review.
My next steps for further research are: 1. Expand the analysis to more LLMs: Broaden the scope of the study to include a variety of LLMs to evaluate their capabilities in commonsense reasoning. This will provide a comparative understanding of diferent models’ strengths and weaknesses in this area. 2. Investigate syllogistic reasoning abilities: Conduct in-depth analysis of the syllogistic reasoning abilities of LLMs. This involves assessing how well these models can perform logical deductions based on premises, which is a critical aspect of human-like reasoning and decision-making. 3. Assess the impact of chain-of-thought prompting: Explore how chain-of-thought prompting influences the performance of LLMs. This approach, which involves prompting models to outline their reasoning step-by-step, could enhance both the accuracy of responses and the quality of explanations provided by LLMs. 4. Evaluate the quality of LLM-generated explanations: Systematically assess the quality of explanations generated by LLMs. This could involve several criteria such as clarity, completeness, and correctness. Understanding the explanatory capabilities of LLMs is vital for their applicability in education, decision support, and other areas requiring interpretability. 5. Study the influence of RAG to receive more accurate, customized and context-aware LLM responses. The aim is to identify the strengths and limitations of RAG. The focus could be again on both the explanation quality as all as the accuracy of information. These steps will contribute to a deeper understanding of the capabilities and limitations of LLMs, particularly in tasks requiring sophisticated reasoning and explanations.
I like to thank my supervisors Frieder Stolzenburg and Ute Schmid, for their invaluable guidance and support. Their expert advice and insightful feedback are crucial to this thesis. [12] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems 35 (2022) 24824–24837. [13] D. Paul, M. Ismayilzada, M. Peyrard, B. Borges, A. Bosselut, R. West, B. Faltings, Refiner: Reasoning feedback on intermediate representations, 2024. arXiv:2304.01904. [14] S. Krause, F. Stolzenburg, Commonsense reasoning and explainable artificial intelligence using large language models, in: European Conference on Artificial Intelligence, Springer, 2023, pp. 302–319. [15] N. Mostafazadeh, M. Roth, A. Louis, N. Chambers, J. Allen, LSDSem 2017 shared task: The Story Cloze Test, in: Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, 2017, pp. 46–51. URL: https://aclanthology.org/W17-0906.pdf . [16] Y. Onoe, M. J. Q. Zhang, E. Choi, G. Durrett, CREAK: A dataset for commonsense reasoning over entity knowledge, 2021. URL: https://arxiv.org/pdf/2109.01653. [17] M. Chen, M. D’arcy, A. Liu, J. Fernandez, D. Downey, CODAH: An adversarially-authored question answering dataset for common sense, in: Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP, 2019, pp. 63–69. URL: https://www.jaredfern.com/publication/ codah/. [18] S. Singh, N. Wen, Y. Hou, P. Alipoormolabashi, T.-L. Wu, X. Ma, N. Peng, COM2SENSE: A commonsense reasoning benchmark with complementary sentences, in: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Association for Computational Linguistics, 2021, pp. 883–898. URL: https://aclanthology.org/2021.findings-acl.78. [19] L. Huang, R. Le Bras, C. Bhagavatula, Y. Choi, Cosmos QA: Machine reading comprehension with contextual commonsense reasoning, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, 2019, pp. 2391–2401. URL: https://aclanthology.org/D19-1243/. [20] L. Du, X. Ding, K. Xiong, T. Liu, B. Qin, e-CARE: a new dataset for exploring explainable causal reasoning, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, 2022, pp. 432–446.
URL: https://aclanthology.org/2022.acl-long.33/. [21] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, O. Tafjord, Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, CoRR – Computing Research Repository abs/1803.05457, Cornell University Library, 2018. URL: https://arxiv.org/abs/1803.05457. [22] M. Sap, H. Rashkin, D. Chen, R. LeBras, Y. Choi, Social IQa: Commonsense reasoning about social interactions, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, 2019, pp. 4463–4473. URL: https: //aclanthology.org/D19-1454/. [23] M. Roemmele, C. A. Bejan, A. S. Gordon, Choice of plausible alternatives: An evaluation of commonsense causal reasoning., in: AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, 2011, pp. 90–95. URL: https://aaai.org/papers/ 02418-choice-of-plausible-alternatives-an-evaluation-of-commonsense-causal-reasoning/. [24] A. Pal, L. K. Umapathi, M. Sankarasubbu, MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering, ACM Conference on Health (2022). URL: https: //arxiv.org/pdf/2203.14371. [25] A. Talmor, J. Herzig, N. Lourie, J. Berant, CommonsenseQA: A question answering challenge targeting commonsense knowledge, 2018. URL: https://arxiv.org/pdf/1811.00937. [26] S. Krause, F. Stolzenburg, From data to commonsense reasoning: The use of large language models for explainable ai, arXiv preprint arXiv:2407.03778 (2024).