For solving NLP tasks, most researchers turn to using pre-trained word embeddings (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017) as a key component of their models. These representations map each word of a sequence to a real valued vector of fixed dimension. Drawbacks of these kinds of externally learned features are that they are (i) fixed, i.e. can not be adapted to a specific domain they are used in, and (ii) context independent, i.e. there’s only one embedding for a word by which it is represented in any context. More recently, transfer learning approaches, as for example convolutional neural networks (CNNs) pre-trained on ImageNet (Krizhevsky et al., 2012) in computer vision, have entered the discussion. Transfer learning in the NLP context means pretraining a network with a self-supervised objective on large amounts of plain text and fine-tuning its weights afterwards on a task specific, labelled data set. For a comprehensive overview on the current state of transfer learning in NLP, we recommend the excellent tutorial and blog post by Ruder et al. (2019)1.

With ULMFiT (Universal Language Model Fine Tuning), Howard and Ruder (2018) proposed a LSTM-based (Hochreiter and Schmidhuber, 1997) approach for transfer learning in NLP using AWDLSTMs (Merity et al., 2017) . This model can be characterised as unidirectional contextual, while a bidirectionally contextual LSTM-based model was presented in ELMo (Embeddings from Language Models) by Peters et al. (2018).

The bidirectionality in ELMo is achieved by using 1https://ruder.io/state-of-transfer-learning-in-nlp/ biLSTMs instead of AWD-LSTMs. On the other hand, ULMFiT uses a more "pure" transfer learning approach compared to ELMo, as the ELMoembeddings are extracted from the pre-trained model and are not fine-tuned in conjunction with the weights of the task-specific architecture. The OpenAI GPT (Generative Pre-Training, Radford et al., 2018) is a model which resembles the characteristics of ULMFiT in two crucial points. It is a unidirectional language model and it allows stacking task specific layers on top after pretraining, i.e. it is fully end-to-end trainable. The major difference between them is the internal architecture, where GPT uses a Transformer decoder architecture (Vaswani et al., 2017) .

Instead of processing one input token at a time, like recurrent architectures (LSTMs, GRUs) do, Transformers process whole sequences all at once. This is possible because they utilize a variant of the Attention mechanism (Bahdanau et al., 2014) , which allows modelling dependencies without having to feed the data to the model sequentially. At the same time, GPT can be characterised as unidirectional as it just takes into account the left side of the context. Its successor OpenAI GPT2 (Radford et al., 2019) possesses (despite some smaller architectural changes) the same model architecture and thus can also be termed as unidirectional contextual. BERT (Bidirectional Encoder Representations from Transformers, Devlin et al., 2019) , and consequently the other two BERT-based approaches discussed here (Liu et al., 2019; Lan et al., 2019 ) as well, differ from the GPT models by the fact that they are bidirectional Transformer encoder models. Devlin et al. (2019) proposed Masked Language Modelling (MLM) as a special training objective which allows the use of a bidirectional Transformer encoder without compromising the language modelling objective. XLNet (Yang et al., 2019) on the contrary relies on an objective which the authors call Permutation Language Modelling (PLM) and is also able to model a bidirectional context despite being an auto-regressive model. 2

In their stimulating paper, Raffel et al. (2019) take several steps in a similar direction by trying to ensure comparability among different Transformerbased models. They perform various experiments with respect to the transfer learning ability of a Transformer encoder-decoder architecture by varying the pre-training objective (different variants of denoising vs. language modelling), the pretraining resources (their newly introduced C4 corpus vs. variants thereof) and the parameter size (from 200M up to 11B). Especially, their idea of introducing a new corpus and creating subsets resembling previously used corpora like RealNews (Zellers et al., 2019) or OpenWebText (Gokaslan and Cohen, 2019) is a promising approach in order to ensure comparability.

However, their experiments do not cover an important point we are trying to address with our work: Focussing on only one specific architecture does not yield an answer to the question which components explain the performance differences between models where the overall architecture differs (e.g. Attention-based vs. LSTM-based). Yang et al. (2019) also address comparability to some extent by performing an ablation study to compare their XLNet explicitly to BERT. They train six different XLNet-based models where they modify different parts of their model in order to quantify how these design choices influence performance. At the same time they restrict themselves to an architecture of the same size as BERT-BASE and use the same amount of lexical resources for pre-training. Liu et al. (2019) vary RoBERTa with respect to model size and amount of pre-training resources in order to perform an ablation study also aiming at comparability to BERT. Lan et al. (2019) go one step further with ALBERT by also comparing their model to BERT with regard to run time as well as width and depth of the model.

Despite all these experiments are highly valuable steps into the direction of better comparability, there are still no clear guidelines on which comparisons to perform in order to ensure a maximum degree of comparability with respect to multiple potentially influential factors at the same time. 3

First, we present the different corpora which were utilised for pre-training the models and compare them with respect to their size and their accessibility (cf. Tab. 1). Subsequently, we will briefly introduce benchmark data sets which the models are commonly fine-tuned and evaluated on. While conceptual differences between the evaluated models have been addressed in the introduction, the models will now be described in more detail. This is driven by the intention to emphasise differences beyond the obvious, conceptual ones. 3.1

This corpus, actually introduced as a benchmark data set by Chelba et al. (2013), combines multiple data sets from the EMNLP 2011 workshop on Statistical Machine Translation. The authors normalised and tokenized the corpus and performed further pre-processing steps in dropping duplicate sentences as well as discarding words with a count below three. Additionally, they randomised the ordering of the sentences in the corpus. This constitutes a corpus with a vocabulary of 793.471 words and a total word count of 829.250.940 words.

et al. introduced the BooksCorpus, which is heavily used for pre-training language models (cf. Tab. 1). In their work, they used the BooksCorpus in order to train a model for retrieving sentence similarity. Overall, the corpus comprises 984.846.357 words in 74.004.228 sentences obtained from analysing 11.038 books. They report a vocabulary consisting of 1.316.420 unique words, making the corpus lexically more diverse than the 1B Word Benchmark, as it possesses a by 66% larger vocabulary whereas having a word count which is only 19% higher. 2https://research.google/pubs/pub41880/ 3https://yknzhu.wixsite.com/mbweb Wikitext-103 (Merity et al., 2016a,b) The authors emphasised the necessity for a new large scale language modelling data set by stressing the shortcomings of other corpora. They highlight the occurrence of complete articles, which allows learning long range dependencies, as one of the main benefits of their corpus. This property is, according to the authors, not given in the 1B Word Benchmark as the sentence ordering is randomised there. With a count of 103.227.021 tokens and a vocabulary size of 267.735, it is about one eighth of the 1B Word Benchmark’s size concerning token count and about one third concerning the vocabulary size. Note, that there is also the smaller Wikitext-2 corpus (Merity et al., 2016c) available, which is a subset of about 2% of the size of Wikitext-103.

a specific subset of the CommonCrawl data based on questions from common sense reasoning tasks. They extracted nearly 1M documents, most of which are taken from longer, coherent stories.

Stanford Question Answering Dataset (SQuAD) 1.1 consists of 100.000+ questions explicitly designed to be answerable by reading segments of Wikipedia articles. The task is to correctly locate the segment in the text which contains the answer. A shortcoming is the omission of situations where the question is not answerable by reading the provided article. Rajpurkar et al. (2018) address this problem in SQuAD 2.0 by adding 50.000 handcrafted unanswerable questions to SQuAD 1.1. The authors provide a train and development set as well as an official leaderboard. The test set is completely held out, participants are required to upload their models to CodaLab. The SQuAD 1.1 data is, in an augmented form (QNLI), also part of GLUE.

ReAding Comprehension Dataset From Examinations (RACE) contains English exam questions for Chinese students (middle/high school). In most of the articles using RACE for evaluation, it is described to be especially challenging due to (i) the length of the passages, (ii) the inclusion of reasoning questions and (iii) the intentionally tricky design of the questions in order to test a human’s ability in reading comprehension. The data set can be subdivided 7https://rajpurkar.github.io/SQuAD-explorer/ 8http://www.qizhexie.com/data/RACE_leaderboard.html in RACE-M (middle school examination) and RACE-H (high school examination) and comprises a total of 97.687 questions on 27.933 passages of text. 3.3

Tab. 2 gives an overview on the amount of computational power needed to pre-train a given architecture on given pre-training (lexical) resources. In Tab. 3 we will directly try to relate model architecture and size as well as usage of lexical resources to model performance.

One thing we can learn from Tab. 2 is the lack of details when it comes to reporting the computational resources used for pre-training. While Howard and Ruder (2018) do not provide any information on the computational power utilised for pre-training, the other articles report it to different degrees. Unfortunately, there are no clear guidelines on how to appraise this when it comes to evaluating and comparing models. This may be attributed to the rapidly growing availability of hardware, but in our opinion it should nevertheless be accounted for, since it might pose environmental issues (Strubell et al., 2019) and also limits portability to smaller devices.

Further, it is important to consider the differences displayed in the Tab. 2 and Tab. 3 when comparing the model performances. Considering two models of approximately the same size (BERT-BASE vs. GPT), the superior performance of BERT-BASE seems to originate purely from its more elaborated architecture because of the similar size. But one should also be aware of the larger lexical resources (BERT-BASE uses at least twice as much data for pre-training) and the unknown differences in usage of computational power. We approximated the latter as the pfs-days (cf. Tab. 2), resulting in an estimation for BERT-BASE being not less than the one for GPT.

Another aspect which should not be ignored when evaluating performance is ensembling. As can be seen in the first column of Tab. 3, the three model ensembles outperform both of the BERT models by a large margin. Only parts of these differences may be attributed to the model architecture or the hyperparameter settings, as the ensembling as well as the larger pre-training resources might give an

As can be seen from the above analysis, there is a lack of a concise guideline for fair comparisons of large pre-trained language models. It is not sufficient to just rank models by their performance on the common benchmark data sets as this does not take into account all the other factors mentioned in this analysis. Further aspects worth reporting are the use of resources (time and compute) spent on model development (including all experimental runs and trials) and hyperparameter tuning during pre-training. In our opinion, this is important with respect to two facets: On the one hand side it is important to take into account environmental considerations when training deep learning models (Strubell et al., 2019) , on the other hand side it is also a signal to the reader/user how difficult it is to train (and to fine-tune) the model. This might have implications for the usage of a model as transfer learning model for diverse downstream tasks. Models that have already been tuned to a high degree during pre-training to reach a certain level of performance, may have, in the long run, less potential for further improvements compared to models which do so without much hyperparameter tuning. To conclude, we unfortunately cannot say with determination which one of the influential factors (architecture or amount of pre-training resources) is more important, but we think that a substantial amount of the recent improvements can be attributed to larger pre-training resources. A detailed disentanglement of the influence of the different components stays an open research question which might be answerable by carefully designed benchmark studies.

We would like to thank the three anonymous reviewers for their insightful comments and their feedback on our work.