Introduction

According to the mainstream LLM development pipeline, Transformer-based architectures [5] outperform sequential training models, like LSTM [6], in various NLP tasks. When small-sized training data are available, optimization becomes necessary [7], [8], but common optimization techniques neglect the linguistically relevant fact that these models (i) conflate semantic/world knowledge with morpho-syntactic competence, (ii) require unreasonable training data compared to that needed by children during language acquisition, (iii) the higher their performance, the lower their return in cognitive/linguistic terms [9]. In this paper we address these three issues, starting from the observation that while world knowledge uses all training data available, and the more the better, structural (morpho-syntactic and compositional semantic) knowledge might require a much smaller dataset (from 10 to 100 million words, according to [10]). We explore this intuition further and, based on prolific literature from the '80s showing that typical child errors are structurally sensitive and never random [11], we model networks' architecture to bias learning towards plausible structural configurations, possibly preventing these "small" language models (SLM) from producing wrong linguistic generalizations. We started from a mild revision of the LM training and evaluation pipeline for Italian including alternative approaches to tokenization based on pseudo-morphological decomposition ( §2.2); we then approached a more structurally-driven update of the cell state in LSTM networks, which we will call eMG-RNN variants ( §2.3); we finally adopted a precise testing benchmark for specific linguistic contrasts in Italian following BLiMP design [12] ( §2.4). We will first set the stage in section ( §2) and discuss one alternative tokenization strategy (MorPiece). A simple modification to the gating system in LSTM is proposed that mimics certain linguistic constraints. Then, we will describe the relevant experiments we have run ( §3) and draw some conclusions based on the observed results ( §4). A general discussion with a description of the next steps will conclude this paper ( §5).

Revisiting LM training pipeline

LM training pipeline is relatively rigid: after corpus cleaning (i), the data are prepared/optimized for tokenization (ii), then the tokenized input is batched for training autoregressive models (iii), mostly feeding transformer-based architectures (iv). Once the models are trained, the evaluation step requires their assessment using some standard tasks (v). In the next sub-sections, we will identify various criticalities in this pipeline, eventually proposing strategies to mitigate these problems and, in the end, training linguistically more informative SLM.

Corpus creation and cleaning

The primary data we collected for Italian replicates plausible linguistic input that children may be exposed to during acquisition, in line with [1]. It consists of about 3M tokens divided into child-directed speech (CHILDES Italian section), child movie subtitles (from OpenSubtitles), child songs (from Zecchino D'Oro repository), telephone conversations (VoLIP corpus, [13]), and fairy tales (all from copyright expired sources). Simple cleaning consisted of removing children's productions from CHILDES files as well as any other metalinguistic annotation (speakers' identification, headers, time stamps, tags, links, etc.). Dimension and rough lexical richness of each section are reported in Table 1 (Type-Token Ratio, TTR) before and after the cleaning procedure.

Table 1

Corpus profiling before (bc) and after (ac) cleaning.

Tokenization: MorPiece (MoP)

Popular vLLMs use either Byte-Pair Encoding (BPE) [14], [15] or (fast)WordPiece (fWP) [16] algorithms for tokenization. The simplicity and computational efficiency of these approaches contrast with the limited morphological analysis they provide. In rich inflectional languages (e.g., Italian) and agglutinative languages (e.g., Finnish), this might induce linguistically unsound generalizations. Here, we explore a more morphologically informed strategy, inspired by the Tolerance Principle (TP) and Sufficiency Principle (SP) [17], aiming to break words into potentially relevant morphemes without relying on morpheme tables [18]. The experiments we conduct compare the impact of different strategies when integrated into various network architectures. We refer to MorPiece (MoP) as a TP/SP-based strategy, which can be algorithmically described as follows: each token is traversed from left to right to create a "root trie," and from right to left to create an "inflectional trie" [19]. Each time a node N of the trie is traversed (corresponding to the current character path in the word), the frequency counter associated with this node (Nc) is updated (+1). Nodes corresponding to token endings (characters before white spaces or punctuation) are flagged. Once both tries are created, the optimization procedure explores each descendant, and for every daughter node Dk its frequency k is compared to HN, the approximation of the harmonic number for N used both in TP and SP [17], where c is the frequency of the mother node Nc:

HN = c/ln(c) (F1)

If k > HN and c ≠ k, a productive boundary break is postulated (based on the inference that since there are different continuations and some of them are productive, i.e. sufficiently frequent according to SP, those might be real independent morphemes). We can check if this break respects HD for the relevant nodes Dj and Ni in the "inflectional trie". This means there exists a path where the frequency i of the daughter node Ni (in the "inflectional trie" the dependency between D and N is reversed) is lower than j/ln(j), where j is the frequency of the mother node Dj. If this is the case, the continuation is not considered "an exception", in the sense of TP [17], suggesting that the continuation is, in fact, a productive independent morpheme. A "++" root node is then activated, the node Dk linked to it, and so on recursively, following the FastWordPiece tokenization strategy [20].

During recognition, the LinMaxMatch identification approach is adopted, as in FastWordPiece. Figure 1 illustrates the relevant morpheme breaks (indicated as "||") obtained by applying this morpheme-breaking procedure in the root and infl tries fragments. Various parametric controls have been considered to tune this procedure: (i) a branching factor (bf) parameter that excludes nodes with an excessively high number (> bf) of continuations (the rationale being that when too many continuations are present, they are unlikely to correspond to inflections; this often happens near the root of each trie); (ii) a cutoff parameter indicating the lower frequency boundary for a mother node (this is necessary to ensure a minimum number of observations; for example, if cutoff = 8, we exclude from the "root" trie any branching daughter with a frequency < 5). As in BPE, minimum frequency control for tokens is also implemented to exclude infrequent dictionary entries. Consider the word "cerca" ("to search for") represented in the "root" trie. In the last "c-a" the relation between Hfc and "a" frequency indicates that a break might exist between the nodes "c" (frequency=1813) and "a" (frequency=1307), since Hfc = 1813/ln(1813) and 1307 > Hfc. This hypothesis is confirmed by the failure of the Hfc check at the relevant "infl" "a-c" segment ("a" frequency=10121, "c" frequency=466619): 10121 < 466619/ln(466619). If Hfc had been greater than "a" frequency, then no segmentation advantage would have been observable.

The proposed algorithm has a linear time complexity of O(2n), as each trie must be explored deterministically exactly once to evaluate the HN/D frequency relation. The best linguistic results (relatively linguistically coherent segmentations) for our Italian corpus were obtained with cutoff=100 and bf=10. We found that it was unnecessary to filter the proposed inflectional breaks using the infl trie double check (TP) since the LinMaxMatch strategy already efficiently filtered out initially overestimated breaks. However, as an anonymous reviewer correctly pointed out, this strategy does not guarantee total inclusion of every token of our training corpus (in contrast to BPE, for instance). We acknowledge this limitation, but we emphasize that our goal was to produce a smaller, potentially more efficient lexicon. In our experiments, while BPE generated a lexicon of 96028 tokens (67169 when the minimum lexical frequency was set to 2), MoP produced a lexicon of just 55049 tokens (cutoff=100, bf=10).

Revisiting LSTM architecture

Despite many variants of the standard LSTM architectures, notably Gated Recurrent Units [21] or LSTM augmented with peephole connections [22], and the discouraging equivalence results for these variations [23], we observe a recent revival of RNN-based model architectures [24]. We believe, in fact, that the core intuition behind the LSTM architecture may be linguistically relevant and worth exploring further, although generally more performant models (for instance in terms of GLUE benchmark, [25]) are usually preferred [26]. The linguistic intuition is that the "longterm memory" (cell state C in Figure 2) in LSTM networks could effectively model various types of nonlocal dependencies using a single mechanism. Linguistically speaking, filler-gap dependencies (1) and co-referential dependencies (2) are both "non-local dependencies" but they are subject to non-identical locality conditions: (2) a. [il panino]i, chi credi che loi abbia mangiato? the sandwich, who (you) believe it has eaten? b. *[il panino]i, chi credi che _i abbia mangiato? the sandwich, who (you) believe has eaten? the sandwich, who do you believe have eaten *(it)?

While both dependencies require C(onstituent)command generalizations to be captured [27], the adjunct island in (1), [28], but not clitic left-dislocation in (2), [29], can, for instance, be licensed with a(n extra) gap (1).b'. Aware of these differences, we decided to simply alter the gating system to allow the LSTM to create distinct pathways: one to "merge" new tokens, the other to decide if a long-distance dependency is necessary, and subsequently to "move" the relevant items [30]. The processing implementation of these operations is inspired by expectation-based Minimalist Grammars formalism, eMG [31], and it is then named eMG-RNN. Following this implementation, merge applies incrementally, token by token, and move means "retain in memory". In more detail, the cell of an eMG-RNN network performs the forward processing described in the computational graph in Figure 2: (i) the input at time t (xt) is linearly transformed to a lower dimension vector (E, loosely used for "embedding"), then concatenated (C) with the previous hidden state/output, if any (ht-1). Two pathways, both transformed using a sigmoid function (σ), lead, on the one hand, to the move gate, on the other, to the merge gate. In the first case, the result of the sigmoid transformation is multiplied (⊙, the Hadamard product) with the input (this either erases or allows some component of the original vector to be added (+) to the previous (if any) context/cell state (ct-1) as in LSTM forget gate). The merge gate, on the other direction, will privilege the new token if the result of the sigmoid combination of the incoming token and the previous hidden state is low, otherwise (1 -this activation, as in GRUs update gate) will favor items in the context/cell state (transformed through a tanh function to simulate memory decay). This architecture is the most performant compared to various alternatives tested for the BabyLM 2024 challenge [32].

A linguistically informed evaluation

The last step in the pipeline requires a linguistically advanced set of oppositions to verify that the structural generalizations can be captured coherently. We adopted the lm-eval package [33] and we included a specific task based on English BLiMP [12]. Most of the contrasts are derived from the COnVERSA test [4]. They consist of minimal pairs ordered following an increasing complexity metric that considers the number of operations necessary to establish a dependency and the locality of such dependency. The examples below illustrate this point by comparing a local agreement dependency with, (3).b, or without, (3).a, a (linear) intervener and a more complex dependency that requires to process an object relative clause (4): The one who the students listen to Vs. Quello che ascolta gli studenti.

The one who listens to the students Four kinds of dependency (agreement, thematic role assignment, pronominal forms usage, questions formation and answering) are considered for a set of 32 distinct syntactic configurations (a total of 344 minimal pairs to be judged, [4]).

Materials and Methods

We trained our models on the IUSS High-Performance Cluster with 2 GPU nodes, each with 4 A100 NVIDIA devices and 1T RAM. Each network has been trained with the full corpus using various batched strategies. CUDA drivers v.12.4 were used. The most relevant configurations tested are discussed in the next session.

Configurations tested

Three different tokenization strategies (BPE, FastWordPiece, and MorPiece) are compared using the best-performing LSTM network [35] , which consists of 650 units for the embedding layer and 650 nodes for each of the two hidden layers. Five different network architectures are compared, with the GroNLP GPT-2small pretrained model [36] constituting our "top LLM performer". This model was re-adapted to Italian from the GPT-2 English trained model, which was originally trained on approximately 10 billion token corpus, namely various orders of magnitude bigger than our corpus. We then trained on our corpus a comparable bidirectional transformer (BERT), two LSTM networks, respectively with 1 and 2 LSTM layers, and a one-layer eMG-RNN network (Table 2), as described in §2.3.

Table 2

Network architectures

Results

Comparing BERT and LSTM architectures, LSTMx1 qualifies as the most performant configuration (both in training and in minimal pair judgments). Considering training, the only batching regimen performing sufficiently well is the fixed sequence length (loss=0.8877 with LSTMx1 vs. conversational loss=4.0240 or naturalistic regimen loss=4.5884). All networks reached a learning plateau around 10-12 epochs. Comparing the performances on COnVERSA, we realized that the results does not improve after 3 epochs of fixed sequence length (60 tokens) training regimen (this result is compatible with the overfitting hypothesis, [37]). Focusing on tokenizer training results with LSTMx1, we observed that BPE and FastWordPiece have comparable performance. MorPiece performs slightly worse, even though the tokenization seems linguistically more coherent (e.g., "farlo" -"to do it" is tokenized both by BPE and fWP as a single token, while it is split in two in MorPiece: "far" "+lo") and the training faster (Table 3). This, however, only marginally impacts on minimal pairs contrast judgments, performing slightly better, overall, just in certain agreement cases.

Table 3 Impact of the tokenization strategy on LSTM training

We then adopted the BPE tokenizer for architectural comparisons. Network training performances are summarized in Table 4 and graphically represented in Figure 3 for linguistic dimensions comparison.

Discussion

Overall, LSTM networks significantly outperform Bidirectional Transformers in this minimal pairs test on Italian. This finding is consistent with results previously discussed in the literature and suggests a clear advantage of recurrent, sequential model architectures (e.g., LSTM) over Bidirectional Transformers in terms of linguistic generalizations [38] and partially justify the renewed interest for RNN networks that we have been observed in the last couple of years [24], [26]. As far as the tokenization procedure is concerned, it is somewhat premature to draw definitive conclusions from our experiments, as MorPiece has not yet been fully optimized or tested. Specifically, the optimal cut-off threshold and minimum branching factor have not been systematically evaluated. Nevertheless, a more morphologically coherent segmentation is expected to enhance sensitivity in certain minimal contrasts.

Similarly, the eMG-RNN architecture could be further explored and optimized, particularly considering specific contrasts, which may help determine whether our linguistic modeling is on the right track. Evidence to the contrary is attested by the judgments of sentences with missing thematic roles, which are often incorrectly preferred by most models, including our eMG-RNN.

In the end, our results suggest that Loss/Accuracy performance registered in training is not a significant predictor of the performance on the COnVERSA test, or more generally, of the linguistic coherence of the LM trained. Likewise, the models' dimension is not a clear predictor either: Transformers trained on the same small dataset perform randomly (in all dimensions their performance is round 50%) while eMG-RNN, which has a number of parameters similar to LSTM-2, outperforms both LSTM-2 and LSTM-1 (half size of eMG-RNN). The training size remains a striking difference compared to the input received by children: this difference of one order of magnitude suggests that the bias considered in eMG-RNN are not yet satisfactory and that our Language Acquisition Device is still more efficient; in this sense, the Poverty of Stimulus Hypothesis remains unrefuted [39] by these results. Next steps will consider extending to 10M tokens the training corpus (to match the English counterpart [1]) and further exploring the effects of optimized tokenization procedures or other minimal modifications, and optimizations [24], of recurrent neural networks.