Commonsense Reasoning: how do Neuro-Symbolic and Neuro-only approaches compare?

Abstract

The representation of knowledge is a central task in Artificial Intelligence and has been an active topic of research since the beginnings of the field. Intensive research and labor has been put into producing resources which encode knowledge regarding diferent topics, structured in suitable formats so as to allow robust, automated reasoning over them. In Natural Language Processing, deep learning models are commonly given unstructured data and seek to learn the necessary knowledge and abstractions required to represent and understand the underlying mechanisms that govern the target language processing tasks. A popular method to address this issue is to expand the training process to include more tasks and data. Yet, it remains one of the challenges of deep learning. In this respect, a promising research path is to combine the rich knowledge encoded in structured resources with deep learning methods, enhancing them with the necessary means to more efectively learn the complexities of the target tasks. In this paper we set out to compare a Neuro-Symbolic model with mainstream Neuro-only models when they are tasked with solving commonsense reasoning problems, which heavily rely on appropriately represented knowledge: commonsense reasoning is an essential part of the human experience, encompassing human values and needs, and by resorting to it, we can organize sensible arguments and decide on efective actions. The results obtained indicate that there is no clear advantage to either approach, with the Neuro-Symbolic model being competitive amongst the Neuro-only models, but not superior.

Keywords

• eol>commonsense reasoning
• neuro-symbolism
• transformer

1. Introduction

• Selecting four prominent NLP tasks in common sense reasoning. Most commonsense reasoning

In this paper, we report on this experimental exercise and its results, and discuss the empirical evidence gathered in view of the research question motivated above: in what concerns commonsense reasoning, is there empirical evidence that hybrid models stand to the promise of having the potential of surpassing the performance of Neuro-only systems?

The results of our experimentation reported in the present paper do not uncover a clear winner, as in the performance scores obtained with the two approaches, no one stands out as having a clear advantage over the other, which comes in line with the findings from [ 1 ]. 1 tasks fall under two categories: multiple choice endowing it with some capacity to suitably process natquestion answering (Q&A) or machine reading ural language (or at least a good amount of linguistic comprehension. In this exercise, we concentrated phenomena); and (ii) fine-tuning , where the model is on Q&A commonsense reasoning tasks, cover- refined with respect to a specific language processing ing diferent topics and diferent reasoning types, task. setting a demanding environment to probe the Ensuing works would improve the methodology, introknowledge and ability of diferent language mod- ducing diferent pre-training tasks that aid downstream els. performance [4, 5, 7]. These eforts have allowed an • Selecting most prominent examples of non- accelerating increase in the state of the art on benchhybrid, Neuro-only models from the three difer- marks such as GLUE [13] and SuperGLUE [14]. Large ent families of Transformers, namely encoders- pre-trained language models were shown to be competonly, encoders-decoders and decoders-only; the itive with systems that access external knowledge, in models are, respectively, RoBERTa [4], T5 [5] and open-book Q&A challenges which require mostly trivia GPT-2 [6]. knowledge, by accessing their internal “memory” learned • Running these three models over the four selected during pre-training [15]. These results raise the questasks, finding the best performing one. tion of whether pre-training alone can endow language • Selecting a most promising hybrid system, models with enough commonsense knowledge to tackle namely COMET, which leverages a BART [7] commonsense reasoning tasks. model and enriches it with knowledge coming Motivated by this question, a study [16] was conducted from a given KB. The simplicity of the injecting to assess the intrinsic knowledge and reasoning capabilitask and the interesting results obtained on gener- ties of diferent pre-trained Transformer-based language ating commonsense knowledge makes COMET a models, without fine-tuning. The models were applied to promising method. We also include a fifth model, diferent commonsense reasoning tasks, by framing them BART-Large, in the experimentation, serving as as sentence scoring tasks. This can be achieved in the fola baseline for COMET(BART). lowing manner: for a given example (e.g. a question and • Finally, running this hybrid system over the same several possible answers), the perplexity of the language four tasks and comparing its performance against model when presented a particular question-answer pair the best Neuro-only found above. is measured, and the pair with the lowest perplexity (thus being the one that makes the most sense) is considered to be the response of the model. The method requires no fine-tuning, allowing “probing” into the knowledge contained in the model. It was found that the models consistently performed better than random, albeit close to it in some tasks, and that a noticeable gap remains between the models and the human baselines.

A promising approach to aid the learning process of commonsense reasoning tasks are Neuro-Symbolic models. This approach attempts to combines knowledge stored in KBs with neural networks, providing a richer prior knowledge.

Early attempts at enriching neural networks with knowledge coming from KBs made use of Graph Embedding techniques, representing concepts as vectors which would be used to initialize the embedding layer. Techniques such as Node2Vec [17], TransE [18] and WordNet Embeddings [19, 20] can leverage the nature of graph structures to produce embeddings for the nodes through the strength of the relations between them. Lexical knowledge coming from ontologies such as WordNet can be used as a way to enrich neural networks in a Neuro-Symbolic system [21].

More recently, with the advent of the Transformer, methods to blend KBs within its parameters leveraged the self-attention system. Embeddings from external KBs are

2. Related Work

The Transformer model [8] has become the all-embracing

and flexible approach to a wide range of NLP tasks. Extensive research has been devoted to refine its architecture and training methodology. It has become commonplace to section the training regime into two stages [9, 10, 11, 12], namely (i) pre-training, where a language model is trained on a large corpus of raw text,

1Code is publicly available at: https://github.com/nlx-group/

Commonsense-Reasoning-Neuro-only-vs-Neuro-Symbolic-Methods. combined with the internal states from the Transformer

3. Tasks

block, following some fusion function [22, 23, 24]. This usually requires the addition of a module that selects relevant entities, fetches their embeddings and includes We adopt four tasks related to commonsense reasoning, them in a self-attention operation with the sequence covering diferent demands and domains in terms of reahidden states. soning, framing a challenging environment to evaluate

These approaches are limited in three ways: (i) ad- the capacity of models addressing them, and the efecditional parameters required to be learned, making the tiveness of the Neuro-Symbolic COMET method. The search space for optimum performance rather expensive; four tasks are (i) Argument Reasoning Comprehension, (ii) given a trained model, accommodating additional enti- (ii) AI2 Reasoning Challenge, (iii) Physical Interaction ties, unseen during previous training, requires additional Question Answering and (iv) CommonsenseQA. training, which may prove costly; and (iii) the embedding matrix of KB entities can quickly become a problem, 3.1. Argument Reasoning especially as some popular and extensive KBs can encode Comprehension millions of entities, being prohibitively expensive to hold in memory. The Argument Reasoning Comprehension Task

In an interesting new wave of research on common- (ARCT) [26] tests the argument reasoning ability of a sense reasoning inspired by GOFAI,2, more complex KBs model, requiring not only language and logic skills but and methods are being developed to tackle the problem. also commonsense knowledge.

Open resources for commonsense reasoning, whether The underlying structure of an argument, whose unmanually built or automatically retrieved, tend to encode covering dates back to Aristotle and his study of argutaxonomic knowledge, which is a subset of the types of mentation, is defined as a series of premises (reasons) that commonsense knowledge. In ATOMIC [ 3, 25 ], we find an support a given claim (conclusion). In another model of approach that seeks to improve reasoning by going be- argumentation, an additional fundamental part, named yond taxonomic knowledge to encoding causal and infer- warrant, is included in the structure [27]. The warrant ential knowledge. This is important for an agent/model to establishes the connection between the premises and reason about what might be the causes for a certain event claims, such that the latter must logically follow from the to happen, and given that it did, what we can infer from it. former (sequitur). Warrants are often implicit, under the In its most recent update, ATOMIC encodes knowledge of assumption that they are shared knowledge between the social-interactions, physical-entity relations, and event- addresser and addressee [28]. This makes identification centered relations. ATOMIC is then used as a resource of warrants an exercise that requires commonsense. to build a dataset to fine-tune diferent generation mod- This task is defined as follows: given a reason and els on a tail generation task, through a method named a claim, choose the appropriate warrant from two posCOMET (Commonsense Transformers) [ 2, 3 ]. Tail gen- sible choices. One of the warrants is a distraction, not eration task is designed to enable models to learn the supporting the sequitur from reason to claim. knowledge contained in a KB. The task consists in pre- The dataset was constructed with data from the Room senting the model with a concept A and a relation, and for Debate, a section in the New York Times,3 where the model must generate a concept B that has such rela- knowledgeable contributors participate in debates retionship with concept A. garding contemporary issues. The authors selected 188

The reasoning behind the generation task lies in the debates of controversial issues and used crowdworkers fact that the universe of commonsense knowledge is so (referred as turkers) from Amazon Mechanical Turk to vast that we cannot hope to build a resource that is com- perform an 8-step pipeline to obtain the dataset instances, plete. Thus, we need our models to be able to generalize resulting in 1970 instances. and generate new knowledge. The results show that Problems of spurious correlations has been detected without the use of the ATOMIC KB, pre-trained models in the original dataset, so we will use a cleaned version fail to generate new knowledge, hinting at the possibility of it [29]. that the commonsense knowledge stored in them is very limited. When fine-tuned on the tail generation task, 3.2. AI2 Reasoning Challenge the COMET models are able to learn to generate new knowledge, even for entities that were not previously seen during training.

The AI2 Reasoning Challenge (ARC) [30] is a multi-choice

question answering task on the topic of natural sciences. The dataset is comprised of questions from 3rd to 9thgrade exams in the U.S. and other parts of the world. It is composed of two sets of questions: the easy and the

2Good Old-Fashioned AI

3https://www.nytimes.com/roomfordebate

challenge sets. The challenge set contains questions that cannot be trivially solved with token co-occurrence, as opposed to those in the easy set that can. Our experimentation will be carried out using the challenge set.

ARC demands models to possess knowledge in diferent dimensions: definitions, facts and properties, structure, processes and causal, teleology/purpose, algebraic, and many more; and diferent reasoning types: question logic, linguistic matching, multi-hop, comparison, algebraic, etc. The diversity in knowledge and reasoning types required to learn ARC makes it a highly challenging task.

3.3. Physical Interaction Question

Answering

parison, we pick a representative from each Transformer family of models:

Encoder-only We adopted RoBERTa [4] as an encoderonly exemplar. RoBERTa [4] is a derivative of BERT [12], conceptualized from a study on the optimization of BERT models.

Decoder-only The GPT series of models have gained

notoriety in NLP, and we select the most recent computationally afordable version, GPT-2 [6]. GPT-2 is a left-to-right language model, comprised of stacked Transformer decoders. It excels in text generation and boasts considerable capabilities in Natural Language Understanding (NLU) tasks.

Encoder-Decoder For this family of architectures, we

resorted to T5 [5]. T5 is conceptualized as a textto-text framework, meaning that both the input and output are entirely textual, regardless of the underlying tasks. This afords T5 with immense lfexibility.

The Physical Interaction Question Answering task

(PIQA) [31] tests the capabilities of models to answer commonsense questions regarding the physical world. Models are presented with a goal, mostly an everyday situation that a human might want to accomplish, and two possible solutions to attain the goal. Models will need to learn, from raw text only, physical commonsense knowledge.

For humans, acquiring physical commonsense knowledge is part of the human experience. We can interact with the world, manipulate objects and figure out how we might use them to solve problem, in a process called grounding [32]. Unlike humans, models as of yet cannot interact with the world to learn these properties, which makes it a real challenge for them to acquire physical knowledge from raw text only.

Large scale language models struggle with this task, with the state of the art achieving 83.5%4 accuracy, compared to the human 95% score.

To inject finer priors into a language model, in a

Neuro-Symbolic approach, we follow the COMET [ 2 ] method, which leverages a generative task to enrich the model with a commonsense knowledge base. In the current paper, we use COMET(BART), which is a BARTLarge [7] model trained with a tail generation task on ATOMIC2020 [ 3 ], the most recent version of the ATOMIC knowledge base. In order to better assess the influence of the knowledge base, we also include a standard BARTLarge in our evaluation.

A diferent fine-tuning technique is used for RoBERTa, GPT-2, COMET(BART) and BART-Large, which was shown as promising for Q&A based commonsense rea3.4. CommonsenseQA soning tasks. The fine-tuning process frames problems into sequence ranking problems [35]. In this framing, CommonsenseQA (CSQA) [33] is a multi-choice question given that the tasks are multiple-choice, the elements of answering dataset that requires commonsense knowl- input pairs of questions and candidate answers, (, ), edge in diferent formats, akin to ARC. It encompasses are separately given to the network, which produces a many diferent knowledge types: spatial, cause and ef- value, named the relevancy score. The pair with the fect, has parts, is member of, purpose, social, activity, maximum relevancy score is the answer given by the definition and preconditions. network.

It was built by resorting to ConceptNet [34], extracting Implementations for the experimentation are based subgraphs of concepts which are used to build questions, on Huggingface [36], along with their pre-trained model through crowdsourcing with Amazon Mechanical Turk. weights.

A sequential hyper-parameter search is employed for the learning rate and batch size. First, the learning rate is 4. Methodology determined through the selection of the model with the Designing a broad evaluation setting enables a richer best accuracy on the development set, after fine-tuning comparison between the Neuro-only pre-trained lan- for 10 epochs, with learning rate values picked from the guage models and the Neuro-Symbolic model. For com- set {1e-3, 1e-4, 1e-5, 2e-3, 2e-4, 2e-5, 3e-3, 3e-4, 3e-5}.

An appropriate batch size is subsequently searched for, 4Accessed on 2021/07/31: https://yonatanbisk.com/piqa/ using the previously determined learning rate. The same strategy is followed, searching for a batch size from the A juxtaposition appears concerning ARC and CSQA. set {4, 8, 16, 32}. The hyper-parameters found for each Despite both being multiple-choice problems with (up model are described in Appendix A. to) five possible answers, scores on ARC are half of those

Models are trained for up to 30 epochs, and the check- on CSQA. The commonsense knowledge types for CSQA point yielding the best accuracy on the development set is more broad, covering a large array of domains, being is selected for testing purposes. Due to known instability a more general task. ARC, on the other hand, focuses on across runs in pre-trained language models [12, 37], we more complex knowledge about the physical world, as it report the mean of five runs, each with diferent random was conceived from science exams, and as such features seeds, which are described in Appendix A. knowledge about physics and chemistry. This hints at

PIQA and CSQA, two of our proposed tasks, are active the possibility that ARC is a harder task to solve. competitions, and as such their test sets are kept private Regarding our primary research question, which is to so as to prevent cheating. Thus, for these tasks, we re- find empirical clues as to whether COMET efectively port results on the development set. In order to have a boosts performance on commonsense reasoning tasks, training set and a development set available, we split the the answer is that it is slightly better than its baseline. training set into two using stratified splitting to preserve COMET(BART), outperforms its baseline BART-Large the original distribution of classes, keeping 90% of the on all but the CSQA task. However, on the ARCT task, data as training data and setting aside 10% of the data as the standard deviation of BART-Large is so large due the development set. to instability that, despite the diferences in the mean,

All experiments were conducted on a single NVIDIA it is unclear whether COMET(BART) provides any real Titan RTX with 24Gb VRAM. advantage in this task.6 As such, given that BART-Large excels on CSQA, and on ARCT it could be that BARTLarge can perform at the level of COMET(BART), if it 5. Results and Discussion were not for the instability, COMET does not provide a sizeable diference over the baseline, but it does improve Performance (in accuracy) for the commonsense reason- over it. ing tasks is shown in Table 5. When comparing COMET(BART) with the diferent

A gap between human performance and the perfor- Transformer families, COMET(BART) consistently holds mance of the models is noticeable, albeit some tasks stand- the 2nd and 3rd best scores, reaching close to the best out as more challenging for the models than others. For score on PIQA. It was competitive with T5 (EncoderARCT, an encouraging gap of 0.094 accuracy separates Decoder) and GPT-2 (Decoder-only), but scored below the human upper bound from RoBERTa, which is the best RoBERTa (Encoder-only) in all tasks but ARC. RoBERTa, performing model for the task. CSQA and PIQA have a while not having had the knowledge injection for finer more significant margin to the human upper bound, with priors, still emerges as the most competent reasoner, hava gap of 0.156 and 0.151, respectively. Despite the ad- ing the best score in ARCT and PIQA, and is a close vances provided by Transformer and the “pre-train then second best in CSQA. The worst reasoner was GPT-2, ifne-tune” methodology, which have pushed the state obtaining poor results in all tasks. of the art further and the boundaries of sizes of model Despite the ability of the Neuro-Symbolic and training data, models are still a long way away from COMET(BART) to be competitive with most Neuro-only human aptitude.

5https://www.tau-nlp.org/csqa-leaderboard

6A t-test between COMET(BART) and BART-Large yielded

> 0.05 for the ARCT, ARC and PIQA results. methods, no clear advantage can be given to any of In this work, we establish a broad and challenging evalthe two approaches overall, for the proposed tasks. uation setting to gauge the eficiency of a Neuro-Symbolic This finding confirms previous work [ 1 ], which has method, COMET, at learning and applying the knowlreached the same conclusion for other tasks. One par- edge learned from the ATOMIC2020 knowledge base, ticular Neuro-only method stood out, RoBERTa, which and how it compares to Neuro-only methods. Five modconsistently outperformed other Neuro-only methods els (RoBERTa, GPT-2, T5, BART and COMET(BART)), and COMET(BART). RoBERTa was pre-trained on the covering the three families of Transformer models (Ensame corpus as BART, and as such, COMET(BART) was coder, Decoder and Encoder-Decoder), are tasked with exposed to the same data and more, due to its refinement four challenging commonsense reasoning tasks (ARCT, on the ATOMIC knowledge base, meaning diferences in ARC, PIQA and CSQA). the training data should not be a factor in the diferences The results show no clear advantage between Neurobetween RoBERTa’s and COMET(BART)’s performance. only and Neuro-Symbolic methods. COMET(BART) is We conjecture that two factors, alone or in combination, marginally better than its BART-Large baseline, and is could account for the observed diferences: competitive with most Neuro-only methods. RoBERTa emerges as the superior reasoner, despite not having been • The generative nature of COMET’s pre-training aforded with finer priors like COMET(BART). task, while being ideal for generating new com- These results call for future research on two diferent monsense knowledge, proves to be detrimental topics: (i) The application of the COMET method to other to classification tasks, in this instance, for com- models in the vast Transformer family. Despite COMET monsense reasoning. being inherently a generative method, its adaptation to • RoBERTa’s architecture, featuring only the En- models like RoBERTa, which has shown great promise coder, could be better adapted to perform com- in our results, could prove to be beneficial; and (ii) An monsense reasoning. Perhaps, a combination of intensive systematic review spanning diferent NeuroCOMET with RoBERTa could yield better results.7 Symbolic methods, each with their particular techniques, and their application to commonsense reasoning. This

Overall, the models behaved like capable reasoners, review would help uncover what type of techniques more performing well above the random baseline, despite the efectively transfer knowledge to models, and aid their gap to human capabilities. One can pose the question of learning. whether this performance is consistent across diferent types of reasoning in the diferent tasks. We have explored the question of consistency in previous work [38]. Acknowledgments We have found evidence that models are inconsistent because they seem to be solving the tasks not with reason- The research reported here was supported by PORing but with shortcuts present in the data. In one instance, TULAN CLARIN—Research Infrastructure for the Sciwe removed the question portion of the input from the ence and Technology of Language https://portulanclarin. tasks, and the models were still able to learn to select the net, funded by Lisboa 2020, Alentejo 2020 and FCT— correct answers to questions they were not presented Fundação para a Ciência e Tecnologia under the grant with. Adversarial attacks also demonstrate that minimal PINFRA/22117/2016. superficial changes to the input have a significant impact on their performance. COMET(BART) was just as sus- References ceptible to shortcuts as other models. However, one can expect that with further research into neuro-symbolism, the enrichment procedure should endow models with ifner priors such that they should become more resistant to this behavior.

6. Conclusion

Neuro-Symbolic methods are a promising path to integrate the rich knowledge represented in structured knowledge bases with deep learning models, enhancing their learning capabilities.

7Assuming an adaptation of COMET’s tail generation task to a classification task is possible. [4] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, J. Michael, F. Hill, O. Levy, S. R. Bowman, SuperO. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, GLUE: A stickier benchmark for general-purpose Roberta: A robustly optimized bert pretraining ap- language understanding systems, arXiv preprint proach, arXiv preprint arXiv:1907.11692 (2019). 1905.00537 (2019). [5] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, [15] A. Roberts, C. Rafel, N. Shazeer, How much knowlM. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the edge can you pack into the parameters of a language limits of transfer learning with a unified text-to-text model?, in: Proceedings of the 2020 Conference on transformer, Journal of Machine Learning Research Empirical Methods in Natural Language Processing 21 (2020) 1–67. (EMNLP), Association for Computational Linguis[6] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, tics, Online, 2020, pp. 5418–5426.

I. Sutskever, Language models are unsupervised [16] X. Zhou, Y. Zhang, L. Cui, D. Huang, Evaluating multitask learners (2019). commonsense in pre-trained language models., in: [7] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mo- AAAI, 2020, pp. 9733–9740.

hamed, O. Levy, V. Stoyanov, L. Zettlemoyer, BART: [17] A. Grover, J. Leskovec, node2vec: Scalable feature Denoising sequence-to-sequence pre-training for learning for networks, in: Proceedings of the 22nd natural language generation, translation, and com- ACM SIGKDD international conference on Knowlprehension, in: Proceedings of the 58th Annual edge discovery and data mining, 2016, pp. 855–864. Meeting of the Association for Computational Lin- [18] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, guistics, Association for Computational Linguistics, O. Yakhnenko, Translating embeddings for modelOnline, 2020, pp. 7871–7880. ing multi-relational data, in: Neural Information [8] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Processing Systems (NIPS), 2013, pp. 1–9.

L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, At- [19] R. Branco, J. Rodrigues, C. Saedi, A. Branco, Assesstention is all you need, in: Advances in neural ing wordnets with WordNet embeddings, in: Proinformation processing systems, 2017, pp. 5998– ceedings of the 10th Global Wordnet Conference, 6008. Global Wordnet Association, Wroclaw, Poland, [9] A. M. Dai, Q. V. Le, Semi-supervised sequence 2019, pp. 253–259.

learning, in: C. Cortes, N. Lawrence, D. Lee, [20] C. Saedi, A. Branco, J. Rodrigues, J. Silva, Wordnet M. Sugiyama, R. Garnett (Eds.), Advances in Neural embeddings, in: Proceedings of the third workshop Information Processing Systems, volume 28, Cur- on representation learning for NLP, 2018, pp. 122– ran Associates, Inc., 2015. 131. [10] J. Howard, S. Ruder, Universal language model fine- [21] M. Salawa, A. Branco, R. Branco, J. António Rotuning for text classification, in: Proceedings of the drigues, C. Saedi, Whom to learn from? graph- vs. 56th Annual Meeting of the Association for Compu- text-based word embeddings, in: Proceedings of tational Linguistics (Volume 1: Long Papers), Asso- the International Conference on Recent Advances ciation for Computational Linguistics, Melbourne, in Natural Language Processing (RANLP 2019), INAustralia, 2018, pp. 328–339. COMA Ltd., Varna, Bulgaria, 2019, pp. 1041–1051. [11] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, [22] M. E. Peters, M. Neumann, R. L. Logan, R. Schwartz, Improving language understanding by generative V. Joshi, S. Singh, N. A. Smith, Knowledge enhanced pre-training, 2018. contextual word representations, in: EMNLP, 2019. [12] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: [23] A. Yang, Q. Wang, J. Liu, K. Liu, Y. Lyu, H. Wu, Pre-training of deep bidirectional transformers for Q. She, S. Li, Enhancing pre-trained language replanguage understanding, in: Proceedings of the resentations with rich knowledge for machine read2019 Conference of the North American Chapter of ing comprehension, in: Proceedings of the 57th the Association for Computational Linguistics: Hu- Annual Meeting of the Association for Computaman Language Technologies, Volume 1 (Long and tional Linguistics, Association for Computational Short Papers), Association for Computational Lin- Linguistics, Florence, Italy, 2019, pp. 2346–2357. guistics, Minneapolis, Minnesota, 2019, pp. 4171– [24] Z. Zhang, X. Han, Z. Liu, X. Jiang, M. Sun, Q. Liu, 4186. Ernie: Enhanced language representation with in[13] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, formative entities, in: Proceedings of the 57th AnS. Bowman, Glue: A multi-task benchmark and nual Meeting of the Association for Computational analysis platform for natural language understand- Linguistics, 2019, pp. 1441–1451. ing, in: Proceedings of the 2018 EMNLP Workshop [25] M. Sap, R. Le Bras, E. Allaway, C. Bhagavatula, BlackboxNLP: Analyzing and Interpreting Neural N. Lourie, H. Rashkin, B. Roof, N. A. Smith, Y. Choi, Networks for NLP, 2018, pp. 353–355. Atomic: An atlas of machine commonsense for if[14] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, then reasoning, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 2019, langue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funpp. 3027–3035. towicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, [26] I. Habernal, H. Wachsmuth, I. Gurevych, B. Stein, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, The argument reasoning comprehension task: Iden- M. Drame, Q. Lhoest, A. M. Rush, Transformers: tification and reconstruction of implicit warrants, State-of-the-art natural language processing, in: in: Proceedings of the 2018 Conference of the North Proceedings of the 2020 Conference on Empirical American Chapter of the Association for Computa- Methods in Natural Language Processing: System tional Linguistics: Human Language Technologies, Demonstrations, Association for Computational Volume 1 (Long Papers), Association for Compu- Linguistics, Online, 2020, pp. 38–45. tational Linguistics, New Orleans, Louisiana, 2018, [37] M. Mosbach, M. Andriushchenko, D. Klakow, On pp. 1930–1940. the stability of fine-tuning {bert}: Misconceptions, [27] S. E. Toulmin, The Uses of Argument, Cambridge explanations, and strong baselines, in: International

University Press, 1958. Conference on Learning Representations, 2021. [28] J. B. Freeman, Argument Structure:: Representa- [38] R. Branco, A. Branco, J. Silva, J. Rodrigues, Shorttion and Theory, volume 18, Springer Science & cutted commonsense: Data spuriousness in deep Business Media, 2011. learning of commonsense reasoning, in: Proceed[29] T. Niven, H.-Y. Kao, Probing neural network com- ings of the 2021 Conference on Empirical Methods prehension of natural language arguments, in: Pro- in Natural Language Processing (EMNLP), Associaceedings of the 57th Annual Meeting of the Asso- tion for Computational Linguistics, 2021. ciation for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 4658–4664. A. Training Hyper-Parameters [30] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, O. Tafjord, Think you have The random seeds used for evaluation purposes were the solved question answering? try arc, the ai2 rea- following: 42, 1128, 1143, 1385 and 1415. soning challenge, arXiv preprint arXiv:1803.05457 The table below describes the hyper-parameters (Batch (2018). Size, Learning Rate, Epochs) used in the experiments. [31] Y. Bisk, R. Zellers, R. LeBras, J. Gao, Y. Choi, Piqa:

Reasoning about physical commonsense in natural language., in: AAAI, 2020, pp. 7432–7439. Task Model Hyper-parameters [32] Y. Bisk, A. Holtzman, J. Thomason, J. Andreas, RoBERTa-Large 16, 1e-5, 25 Y. Bengio, J. Chai, M. Lapata, A. Lazaridou, J. May, GPT2-Medium 8, 2e-3, 18 A. Nisnevich, N. Pinto, J. Turian, Experience ARCT T5 8, 2e-5, 17 grounds language, in: Proceedings of the 2020 BART-Large 16, 2e-4, 12 Conference on Empirical Methods in Natural Lan- COMET(BART) 8, 1e-4, 25 guage Processing (EMNLP), Association for Com- RoBERTa-Large 8, 1e-4, 16 putational Linguistics, Online, 2020, pp. 8718–8735. GPT2-Medium 4, 1e-3, 26 [33] A. Talmor, J. Herzig, N. Lourie, J. Berant, Common- ARC T5 8, 2e-5, 12 senseQA: A question answering challenge targeting BART-Large 8, 1e-4, 27 commonsense knowledge, in: Proceedings of the COMET(BART) 8, 3e-5, 22 2019 Conference of the North American Chapter of RoBERTa-Large 16, 3e-3, 28 the Association for Computational Linguistics: Hu- GPT2-Medium 8, 1e-3, 22 man Language Technologies, Volume 1 (Long and PIQA T5 8, 1e-5, 9 Short Papers), Association for Computational Lin- COBAMRETT-L(BaArgReT) 342,, 13ee--34,, 1196 guistics, Minneapolis, Minnesota, 2019, pp. 4149– 4158. RoBERTa-Large 8, 3e-4, 13 [34] H. Liu, P. Singh, Conceptnet—a practical common- GPT2-Medium 8, 1e-3, 14 sense reasoning tool-kit, BT technology journal 22 CSQA T5 8, 2e-5, 5 (2004) 211–226. COBAMRETT-L(BaArgReT) 88,, 31ee--44,, 1184 [35] X. Liu, P. He, W. Chen, J. Gao, Multi-task deep neural networks for natural language understanding, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 4487–4496. [36] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. De

References

[1] N. Lourie , R. Le Bras , C. Bhagavatula , Y. Choi, Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark , AAAI ( 2021 ).

[2] A. Bosselut , H. Rashkin , M. Sap , C. Malaviya , A. Celikyilmaz , Y. Choi , Comet: Commonsense transformers for automatic knowledge graph construction , in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 2019 , pp. 4762 - 4779 .

[3] J. D. Hwang , C. Bhagavatula , R. Le Bras , J. Da , K. Sakaguchi , A. Bosselut , Y. Choi , Comet-atomic 2020 : On symbolic and neural commonsense knowledge graphs , in: AAAI'21 , 2021 .