The popularity of creating artificial intelligence models has been incredibly rising. These tools can be used in many diferent areas, including text analysis. Most modern models ofer great accuracy in many diferent text-based tasks but are limited by a huge number of data required to not only pre-train but also fine-tune these models. This problem only deepens in models trained on data from low and mid-resource languages, such as Slovak. In this paper, we examined the fine-tuning process of such models and tried to enhance it by connecting to Epinet to create Epistemic neural network, a relatively new concept that helps the model to detect its own uncertainty to make better decisions in the long run.
SlovakBERT is the first monolingual Large language
model (LLM) trained on Slovak text data. It was
introduced in the year 2021, although many diferent
BERTline of models were known for a longer period of time.
As it is in fact RoBERTa model, its strength comes from
the clever design of Transformer architecture [
Epistemic neural network (ENN) difers from the traditional network by taking additional input called epistemic index. This index, usually sampled by some reference distribution such as normal distribution or a finite set of values, is used to help the model recognize its own epistemic uncertainty. This uncertainty is easier to show in the joint probability distribution, where multiple samples (inputs to the network) are considered. are the same - 0.5. This shows the model’s uncertainty for the prediction, but the reason for this uncertainty is not clear. If we took the text sequence and put it as an input for the neural network a second time, we can model joint probability distribution by the table of size 2x2. In this case, we can get two diferent edge cases (the tables on the right). In the upper table, all possible combinations of classes when given two input samples show the same probability of 0.25. Summing up for marginal prediction, for each sample network assigns a probability of 0.5. This case suggests aleatoric uncertainty which most probably won’t be solved with future training. The more interesting case is shown in the table at the bottom right. Here, the networks’ choice is always consistent when the class for the first sample is chosen, the same class is assigned for the same sample for the second time. The only problem is that the network is not sure which class to assign in the first place. The sum of the marginal predictions is, as in the first case, 0.5, but the diference in distribution is visible. This second example suggests epistemic uncertainty, i.e. more training could resolve models’ uncertainty.
The epistemic index comes as an additional input to ENN to help with epistemic uncertainty. In figure 2, we see two diferent use cases. In the first case, ENN doesn’t use the epistemic index to its benefit and thus indicates ambiguous data. The possible use of the epistemic index is shown in the tables below it. Here, according to the sign of epistemic index, the class is chosen - that holds true not only for prediction of one input but also when the same input is given two times in a row. If a prediction of the network can be manipulated in this way by some additional input, then we can assume that future training will resolve some of its uncertainty.
tional neural network takes as an input a text sequence
and outputs a vector containing probability distribution
for 2 classes - positive or negative sentiment. In an edge
case (the table in the middle), the probabilities assigned
One concrete use of ENN was introduced with the
Epinet [
In most cases, these networks are standard Multi-layer perceptrons with the same architecture but diferent starting parameters. The prior network has no trainable parameters and aims to introduce initial knowledge about uncertainty. Over time, learnable networks’ parameters are updated, whereas prior networks’ parameters stay the same. Both of these networks take two inputs: epistemic index and so-called features ˜ of the base network. These features are usually the last layers or (parts of them) of the base network, which we are trying to train. These two inputs are concatenated and fed into the learnable and prior network, which results are then summed. The final result of ENN is the sum of the Epinet networks result and result of the base network : (˜, ) = (˜, ) + (˜, )
(, ) = () + (˜, )
In our experiments, we use the base version of the pretrained SlovakBERT network. On top of it, we add a Multi-layered perceptron (MLP) with one hidden layer of size 50. When connecting to Epinet to create an Epistemic neural network, the result of the SlovakBERT network present in the representation of [CLS] token at the last layer is used as the input to this MLP network and also as ’features’ for the Epinet.
In Epinet, we follow the original implementation in
[
The prior network has no trainable parameters. Its design is the same as for the learnable network: (˜, ) = ( (˜, ))
Both of these networks are the same in size as the MLP network connected to the SlovakBERT. The epistemic index is sampled from a uniform distribution.
For the size of epistemic indices, we iterate through multiple possible values. In our experiments, we try to sample this vector with sizes 10, 30, 50, and 100. The number of indices sampled in each stochastic gradient descent step is always 5. With each size we fine-tune the model from scratch, giving us the best setting for a reasonable comparison.
For optimization we use Adam [
We fine-tune our model for sentiment analysis classiifcation task on CSFD [ 11] dataset, a publicly available collection of movie reviews. For this, we preprocess the dataset in two ways: • we keep only two data features: text review and rating • as the rating column contains values ranging from 0 to 6, we edit these values to get sentiment analysis task of 3 possible classes: 0 - negative, 1 - neutral, and 2 - positive. We do this by mapping original values to predefined classes - values 0 and 1 are mapped to 1, values 2 and 3 are mapped to 1, and values 4 and 5 are mapped to 2.
No other changes to the text data itself were made. We
ifne-tune the model for 5 epochs on 25k train data and
validate the model after every epoch on 5k validation
data. The loss function used is the standard cross-entropy
function. For determining joint logarithmic loss value
during validation we use the vanilla version of the dyadic
sampling heuristic as described in [
We recognize that other settings could be potentially better suited for our experiments, but this exact preprocessing allowed us to easily train the model while keeping reasonable values for the potential sentiment in the text.
In this section, we present the results of our experiments.
To easily recognize the networks, we will always refer to SlovakBERT connected to only standard MLP as ’base SlovakBERT’ and SlovakBERT connected not only to standard MLP but also to Epinet network as ’SlovakBERT Epinet’ or simply ’Epinet’. Each diferent SlovakBERT Epinet will also contain in its name the size of the epis- not find any significant change in accuracy while using temic index used. Epinet. Although it is logical as in the case of training
Training accuracy nor training loss did not improve accuracy, longer training could still potentially reveal the significantly when using SlovakBERT Epinet. The result- problem with under-training. On our small dataset, no ing values did not difer in a major way, which holds major changes were discovered, though. not only when comparing to base SlovakBERT, but also diferent index size parameters of SlovakBERT Epinet.
As calculated training loss is in its nature marginal (each
sample is independent), it seems logical that no major
improvements are observed. Results can be seen in
figure 4 and 5. Similarly as in [
on validation dataset, results show that the networks
using epistemic index performed much better than base
SlovakBERT without Epinet in average. We also found
out that the networks are really sensitive to the size of
the epistemic index. In the table 1 below, we present
the percent value of the final joint logarithmic loss of
Epinet networks when compared to the value of the base
SlovakBERT.
(i.e. lower) joint loss values as networks with small epis- LLMs in terms of raw joint log-loss values but also
postemic indices sizes such as 10. It doesn’t mean that each sibly lower the amount of annotated data needed in the
consecutive epoch in training helps in lowering joint loss ifne-tuning process, e.g. with the help of Active learning
value, though. This can be also seen in the final loss val- methods. As most of these methods try to estimate
modues for Epinet 1 and 30, where the final value was actually els’ uncertainty on the input data, their combination and
higher than the final loss value of base SlovakBERT. As in uncertainty estimation with epistemic index and Epinet,
[
This research has been supported by grants APVV-210114 and UK/96/2023.
As the results suggest, ENN can serve as an
improvement to conventional neural networks, but our results
indicate some sort of instability, possibly caused by a lack
of fine-tuning data or weak adaptability of the network
parameters when used with a low-resource language
model. Both of these possible causes can be solved with
additional research, though. Still, finding this instability
is an interesting aspect of the ENN network, which was
not presented yet. The argument that in [
We showed that Large language models pre-trained in low-resource languages, such as Slovak, can benefit from modeling their uncertainty in the fine-tuning process to reach better results. This was made possible by connecting this model to the Epinet network, creating an Epistemic neural network. Results don’t indicate a major improvement in classification accuracy, but joint logarithmic loss seems to improve substantially. More research and experiments on this topic are needed, though. Our next potential step is to not only enhance these kinds of tic optimization, arXiv preprint arXiv:1412.6980 (2014). [11] Fewshot-goes-multilingual, 2022. Data retrieved from Huggingface site, https://huggingface. co/datasets/fewshot-goes-multilingual/sk_ csfd-movie-reviews.