For many applications, analyzing the uncertainty of a machine learning model is indispensable. While research of uncertainty quantification (UQ) techniques is very advanced for computer vision applications, UQ methods for spatio-temporal data are less studied. In this paper, we focus on models for online handwriting recognition, one particular type of spatio-temporal data. The data is observed from a sensor-enhanced pen with the goal to classify written characters. We conduct a broad evaluation of aleatoric (data) and epistemic (model) UQ based on two prominent techniques for Bayesian inference, Stochastic Weight Averaging-Gaussian (SWAG) and Deep Ensembles. Next to a better understanding of the model, UQ techniques can detect out-of-distribution data and domain shifts when combining right-handed and left-handed writers (an underrepresented group).
[10], and Laplace Approximation [11]. Another widely used technique are Deep Ensembles [12], which often Traditional machine learning (ML) algorithms assume yield well-calibrated models while being relatively easy training and test datasets to be independently and identi- to implement. cally distributed [1, 2]. For many real-world applications, data often changes over time and space, and hence, training and test data originate from diferent distributions.
This can cause ML models to fail due to a domain shift between training and test data [1]. Transfer learning [3, 4] and domain adaptation [5, 6] techniques can compensate for this domain shift. A first step in adapting for this domain shift is its detection, e.g., by having reliable uncertainty estimates of the model predictions [7]. Thus, to estimate the uncertainty of the model, a robust uncertainty quantification (UQ) technique is required that runs in real-time. temporal information such as writing speed and direc- aNet [28] addresses this issue for spatio-temporal human tion [20]. While many recording systems make use of a mobility forecasting by injecting controllable uncertainty. stylus pen together with a touch screen surface, sensor- This allows insights to both, UQ and weak supervised enhanced pens, e.g., [21, 22, 23, 24], based on inertial mea- learning. [29] focused on the spatio-temporal uncertainty surement units (IMUs) enable new applications. These of urban prediction (where and when a piece of land bepens stream data from accelerometer, gyroscope, mag- comes urban). [7] argue that the feature statistics such netometer and force sensors in real-time represented as as mean and standard deviation (the domain characterisspatio-temporal multivariate time-series (MTS). The ad- tics of the training data), can be manipulated to improve vantage of exploiting this temporal information is the the generalizability of DL models by modeling the unability to better distinguish between similarly shaped let- certainty of domain shifts with feature statistics during ters from dynamic information (number of strokes etc.). training (that follow a multivariate Gaussian distribuSpatio-temporal data can further help to identify certain tion). In the context of domain adaptation, [19] adressed characteristics in the data. [25], e.g., showed the domain the extraction of domain-invariant representations for shift between right-handed and left-handed writers by MTS classification. analyzing feature embeddings of their model for OnHW data.
Contribution. In this paper we evaluate the uncer- [21] initially proposed the OnHW-chars dataset and evaltainty of OnHW model predictions with SWAG [9] and uated machine and DL techniques for the OnHW MTS Deep Ensembles [12] for spatio-temporal reasoning, as- classification task. The dataset contains right-handed sessment of out-of-distribution detection, and pattern and left-handed writers with a domain shift between and failure recognition. We use uncertainty decomposi- both groups of writers (i.e., domains). [25] showed that tions based on the method by [15] and [16] to evaluate transfer learning from small adaptation datasets results the UQ techniques. Our claims are further supported in poor model performances. Hence, their domain adapby utilizing confidence and accuracy metrics to estimate tation approach transforms features from left-handed the expected calibration error (ECE) [26]. For an OnHW writers into the domain of features from right-handed task with domain shift between right- and left-handed writers by optimal transport techniques. A reliable UQ writers, we evaluate uppercase, lowercase and combined method could identify out-of-distribution samples and character classification tasks. Our source code will be only apply the transformation on samples for which the available upon publication.1 model has a high uncertainty. [22] combined ofline and
The remainder of the paper is organized as follows. online handwriting recognition with a cross-modal repSection 2 discusses related work. In Section 3, we de- resentation learning technique by increasing the dataset scribe the background of Bayesian modeling and approx- size by using generative models. A robust uncertainty esimate inference. The experimental setup is described in timation technique could select samples with high model Section 4, and results are discussed in Section 5. uncertainty.
We first present related work of UQ with focus on spatiotemporal reasoning in Section 2.1. Section 2.2 summarizes state-of-the-art results for OnHW recognition.
In the following we describe Bayesian model averaging in Section 3.1 and the two employed Bayesian UQ methods in Section 3.2. The decomposition of total predictive uncertainty into aleatoric and epistemic uncertainty is discussed in Section 3.3. 1 Code and datasets: www.iis.fraunhofer.de/de/f/lv/dataanalytics/ anwproj/schreibtrainer/onhw-dataset.html [27] analyzed Bayesian and frequentist UQ methods for 3.1. Bayesian Model Averaging spatio-temporal forecasting on network trafic, epidemics and air quality datasets. Their evaluation shows that Bayesian approaches in DL naturally represent uncerBayesian methods are typically more robust in mean pre- tainty by placing a distribution over model paramediction, while confidence levels from frequentist meth- ters and then marginalizing these parameters to form ods provide better coverage over data variations (i.e., a predictive distribution (Bayesian model averaging) [30]. out-of-distribution data). Furthermore, traditional learn- Let ( |) be the posterior distribution over model paing schemes lack knowledge about uncertainty. STU- rameters , i.e., real-valued weights in the NN, given training dataset , and let (* |* , ) denote the probability distribution over model outputs * (predicted classes), given sample * , and model weights . For the OnHW classification task, the sample * is an MTS U = {u1, . . . , u} ∈ R× , an ordered sequence of = 13 streams with u = (,1, . . . , ,), ∈ {1, . . . , }, where = 64 is the length of the MTS. The training set is a subset of the array = {U1, . . . , U } ∈ R × × , where is the number of time-series. The aim is to predict an unknown class label * ∈ with classes (i.e., character labels) for a given MTS. The predictive distribution of the target variable is then given by (* |* , ) =
(* |* , )( |).
∫︁ In practice, we can approximate this integral by drawing Monte Carlo samples from the posterior distribution: (* |* , ) ≈
1 ∑︁ (* |* , ) , ∼ ( |). =1
(
In order to apply Bayesian inference to an NN, we need
to compute the posterior ( |) of the NN weights. As
the computation of the posterior is usually intractable,
a (local) approximation is often used. This can be
addressed by SWAG and Deep Ensembles with the latter
abstaining from explicitly modeling ( |) –
nevertheless, this method can be considered to be in the field of
approximate Bayesian inference.
In the literature two sources of uncertainty are
commonly considered [13]: (
The definition proposed by [ 15] is based on considerations by [14] and presents a novel way to estimate predictive uncertainty by breaking it down into
1 ∑︁ diag(ˆ) − ˆˆ⊤ + 1 ∑︁(ˆ − )¯( ˆ − )¯ ⊤, =1 =1 ⏟
⏞ aleatoric uncertainty ⏟
⏞
epistemic uncertainty
(
SWAG [9] is a Bayesian inference technique for DL that builds on Stochastic Weight Averaging (SWA) [31]. SWA computes an average of stochastic gradient decent (SGD) iterates to obtain information about the geometry of ( |) from its trajectory. This posterior is then approximated by a Gaussian with simplified covariance structure and reduced dimensionality.
Interpretation. Equation 3 yields two × matrices with diferent interpretations. For the aleatoric part, diagonal values are in {− 2 | ∈ [0, 1]}, with the maximum uncertainty for = 0.5. If the model is “unsure”, meaning that the model neither displays confidence that a prediction corresponds to a certain class nor displays confidence that it is not, we expect high aleatoric uncertainty. The of-diagonal elements consist of values in Deep Ensembles. Deep Ensembles are a committee of {− · | , ∈ [0, 1]}, which yields values on the interindividual NNs initialized with a diferent seed [ 12]. The val [− 0.25, 0]. Lower values correspond to higher data initialization serves as the only source of stochasticity in uncertainty. For the epistemic part, the diagonal contains the model parameters which are otherwise not random; the squared diference to the mean softmax outputs (over Deep Ensembles can optionally be coupled with a difer- samples). The of-diagonal has positive values when ently shufled data loader. In contrast to SWAG, results the softmax values coincide and negative values if the are obtained by averaging the predictions of indepen- softmax values display an inverse relationship. dently trained networks instead of explicitly modeling a posterior and sampling from it. [32] point out that even an ensemble size of = 5 performs well, strengthening its reputation as a “gold standard” for accurate and well-calibrated predictive distributions.
= (︀ (* |* , ))︀ ≈
=1 ︁( 1 ∑︁ (* |* , ) .
︁) Efectively, this is the entropy of the averaged categorical predictions, and it includes the two sources of uncertainty we are interested in.
Aleatoric Uncertainty (AU), Entropy. press aleatoric uncertainty as the expectation over the entropies of sampled conditional predictive distributions with fixed weights, i.e., ≈
=1 1 ∑︁ ((* |* , )).
Finally, epistemic uncertainty emerges as the diference of total and aleatoric uncertainty = − is equivalent to the mutual information (MI): , and = ︁( 1 ∑︁ (* |* , ) − =1
︁) =1 1 ∑︁ ((* |* , )).
Intuitively, epistemic uncertainty stands for the information gain about the model parameters that would be obtained when observing the true outcome. MI is always non-negative, zero in case of perfect independence of * and , and positive when model uncertainty is present at prediction time.
In our order to evaluate the eficacy of UQ methods for spatio-temporal handwriting datasets, we use the OnHW dataset (Section 4.1) and fit diferent network architectures (Section 4.2). Our evaluation approach is given in Section 4.3. For architecture and training details and SWAG parameters, see Appendix A.1. For Deep Ensembles, we choose = 10 (for a study on number of base learners in Deep Ensembles vs. SWAG performance, see [9]).
The OnHW-chars [21] dataset consists of recordings from Another way to decompose predictive uncertainty into a sensor-enhanced ballpoint pen providing 14 sensor meaan aleatoric and epistemic part is by following [17] and surements: two accelerometers (3 axes each), one gyrosimilarly [16]. Based on principles from information thescope (3 axes), one magnetometer (3 axes), a force sensor ory, the Shannon entropy () = − ∑︀
=1 2() is utilized as a common measure of “informedness” of a sin- time steps. 119 right-handed and nine left-handed writgle probability distribution with outcomes/classes ers participated in the data collection. Each person was and the associated probabilities for each -th class ; tak- instructed to write the English alphabet on plain paper ing the logarithm to base 2 yields values measured in bits. six times. This results in 31,275 right-handed and 2,270 The total predictive uncertainty (TU) of the predictive left-handed samples. The task is to either classify low(with which the pen tip touches the surface), and the distribution (* |, ) can then be quantified by and
accuracy() = with denoting the set of indices of sampled softmax outputs falling into the interval (, ].
Commonly, the softmax output range is divided into ten bins 1 || ∈
∑︁ ˆ
1
|| ∈
∑︁ 1(ˆ = ),
(7)
(8)
We can ex- tion and combine it with one unit for spatio-temporal
(
We use a modified CNN from [ 21, 24] for feature extracclassification to extract important temporal features. This unit is added before the last dense layer. We compare a standard long short-term memory (LSTM) cell with 100 neurons, a bidirectional LSTM (BiLSTM) cell with 100 neurons, and a temporal convolutional network (TCN) with 120 neurons. The last dense layer contains 26 neurons for the lowercase and uppercase tasks, or 52 neurons for the combined task. We interpolate the time-series to 64 time steps without sensor normalization.
Calibration can be understood as the degree of reliability of a model. According to [18], a predictor is well-calibrated if the derived predictive confidence represents a good approximation of the actual probability of correctness – meaning that 20% of all predictions with a predictive confidence of 80% should actually be false. Calibration is thus a notion of uncertainty, measuring the discrepancy between the model’s forecasts and (empirical) long-run frequencies [12]. Using the definitions of confidence and accuracy [ (interval sizes of 0.1). We can now make statements whether our model is under-confident (︀ accuracy() > confidence ()︀) or over-confident (︀ accuracy() < confidence ()︀) . It has been shown that softmax outputs of deep NNs are in general not well calibrated and are often either over- or under-confident [ 26]. Ideally, accuracy() ≈ confidence (), allowing the user to interpret softmax outputs as probabilities and thereby quantify the prediction uncertainty.
Method WLoDwercWaseI WUpDpercWasIe WCoDmbinWedI SWAG 8T1C.8N5 7T4C.2N4 8T4C.9N2 7T9C.5N8 7T0C.3N7 6T3C.6N4 Deep 80.55 71.41 88.07 78.65 73.31 62.14 Ensembles LSTM TCN TCN TCN TCN TCN the model is under-confident. We additionally plot the histogram [34] of the softmax outputs to get an overview of the distribution.
In the following, we summarize the main results. In general, the models perform better on WD classification tasks than on WI tasks. Architectures with TCN units outperform LSTM and BiLSTM units on most tasks. Reliability Diagrams. We visualize Equations 7 and 8 as reliability diagrams [33] for selected models. Generally, a model is over-confident if the black bars (displaying the accuracy for one bin) are below the dashed bisectors. Consequently, if the black bars are above the bisectors,
Expected Calibration Error (ECE). The ECE sum- handed writers). SWAG and Deep Ensemble models marizes how far away the confidence is from the actual perform very similarly to frequentist models proposed in (empirical) accuracy [26]. It can be defined as [21] in terms of predicitve accuracy (see Table 2), being at most 3% points below and 5% points above a respecECE() = ∑︁ || |accuracy() − confidence ()|, tive frequentist model. When applying models trained =1 with right-handed data on the left-handed datasets, the (9) performance ranges from 33.27% to 49.87% accuracy (see with being the number of predicted softmax outputs, Table 1) which is substantially better than “pure guessand being the number of bins. Note that this met- ing” – our models make informed decisions after shifting ric does not give any information about over- or under- domains, albeit at a lower standard. A possible reason confidence – only how far away the expected accuracy is that certain sensors produce nearly identical signals is from the confidence. Ideally, the ECE is 0. regardless of the orientation of the pen. For example, the accelerometer at the bottom of the pen should give the same readings for left-handed writers when writing "I" and "i" as for right-handed writers, since it is simply a downward motion regardless of the writing hand. Evaluation on Handedness (trained on right- and confidence is not empirically justified when looking at left-handed writers). When evaluating performance the reliability diagrams in Figure 2, which point out that on right-handed data, models trained only on right- models trained without left-handed writers data are mishanded datasets consistently outperform models trained calibrated and therefore overconfident. Models trained on both datasets combined and yield between 2% points on the combined writers (Figures 2a and 2c) provide more and 12% points higher accuracies (see Table 3). This per- realistic accuracies when applied to the left-handed data formance loss is compensated by a performance gain for (ECE of 6.72). The ECE is even higher (24.24) for leftleft-handed data. Still, the performance is not up to par handed evaluation without left-handers in the training with right-handed data; this gap may be due to a “writing set (see Figure 2d). For a separate evaluation for each style” particular to every writer that especially influences character, see Appendix A.2. the gyroscope and magnetometer measurements. More importantly, left-handed writers have a writing style dif- 5.1. Uncertainty Analysis based on [Kwon ferent to right-handed writers which is perhaps exactly et al.] what the right-handed models never learned in order to address the style of left-handed writers, underlining the need for a suficient amount of samples to get a good representation of various writing styles.
In Figure 3 we visualize the aleatoric and epistemic uncertainty as well as the confusion matrix for the Deep Ensemble model and the combined task. For SWAG model results, see Appendix A.3. In the aleatoric uncertainty Analysis of Uncertainty. Figure 1 shows the MI and heatmap (Figure 3a) we observe a trace with negative entropy for SWAG and Deep Ensemble models evalu- values at the lower end of the scale. Note that for ofated on the left-handed data. The barplots show that diagonal values, the aleatoric uncertainty is higher for the models trained on only right-handed data display lower softmax values. Here, two softmax outputs (with lower uncertainty (i.e., higher confidence) compared to the highest values) coincide on average (see Section 3.3.1). models trained on combined data. However, this higher This means that the model tends to confuse the two A B C D E F G H JI K L M N O P Q R S T U V W X Y Z 0.020 0.015 80 60 []% y c ra 40 ccu
A 20 0 100 80 classes. The most prominent of-diagonal strip corre- such pairs with high similarity lead to negative values. sponds to the upper- and lowercase pairs. This makes When one entry of the softmax output values is below intuitive sense since, e.g., the lowercase "u" and upper- and another entry above the respective sample mean, case "U" are written similarly. This efect is consequently negative epistemic uncertainty is implied. This leads to not present for less similar pairs like "a" and "A". We some kind of discriminative power due to the negative can see this efect also for "l" (lowercase "L") and "I" “covariance” for which there is little justification. We thus (uppercase "i"). A very similar pattern can also be ob- advise caution when interpreting the epistemic uncerserved in the confusion matrix (see Figure 3c), confirming tainty in this context. that the trained model is not only unsure about how to classify these pairs, but is also empirically worse in the 5.2. Uncertainty based on Information respective classification task.
These patterns allow for further interesting insights. Theory For example, one might expect this pattern to occur for We further highlight the trade-of when using informa"i" and "j", but the corresponding heatmap entries lack tion theory-based measures to decide whether a sample signs of confusion of the model. Similarity between char- is too uncertain to classify correctly. This is depicted by acters consequently hinges on the similarity of motions Figure 5a showing the relationship between classification while writing. Two characters with small diferences are accuracies and diferent threshold values. We choose the written similarly but in diferent sizes. This also holds entropy as the target metric for uncertainty evaluation for specific parts of the letters. For example, "n" and (MI would work analogously). On the x-axis is the accu"h" have a higher aleatoric uncertainty in Figure 3a; the racy of the samples above the threshold, i.e., samples our major diference being that one tiny part of "h" is longer. model feels confident about classifying correctly. On the
Somewhat puzzling is that we see the same efect in y-axis is the accuracy for the samples below the threshthe epistemic uncertainty heatmap (see Figure 3b), where old. These values would be considered as too inaccurate
We employed SWAG and Deep Ensembles for OnHW
recognition with left- and right-handed writers, a
spatiotemporal MTS classification task with domain shift. We
critically evaluated aleatoric and epistemic uncertainty
75Ac8c0. >85Th9re0sh9o5ld100 0 0 En1trop2y per3Sam4ple 5 gursainmgs.coInnfidseunmcemcaarlyib,r(a1t)iothne, EmCoEdealnpderrfeolirambailnitcye doinal-y
(a) Sabamovpeleanacecnutrroapciyesthbreeslohwolda.nd (b) Henitsrtoopgyradmistrviibsuutailoinzi.ng the apraertolyverre-lacotensfidteontthaenhdanmdiesdcanleibssraotfewd rwitheresn, (o2n) loyutrrmaiondeedls
Figure 5: Accuracy and entropy for the Deep Ensemble with right-handed writers and evaluated on left-handed
CNN+TCN model trained on the combined WD (right-handed writers, (
This work was supported by the Federal Ministry of Education and Research (BMBF) of Germany by Grant No. 5.3. Summary and Limitations 01IS18036A (David Rügamer) and by the research program Human-Computer-Interaction through the project Uncertainty Decomposition. Neither uncertainty “Schreibtrainer”, Grant No. 16SV8228, as well as by quantification method shows notable diferences between the Bavarian Ministry for Economic Afairs, Infrastrucaleatoric and epistemic uncertainty. The heatmaps ex- ture, Transport and Technology through the Center for hibit the same “strip” for similar characters and give no Analytics-Data-Applications (ADA-Center) within the hints to diferent sources of uncertainty (data-driven or framework of “BAYERN DIGITAL II”. systemic confusion). The benefits of this kind of uncertainty diferentiation are limited, but measuring the total uncertainty can still be useful for domain adaptation or References the detection of wrong labels.
Real-World Link. Since the models trained on
rightand left-handed writers lead to lower data confidence
compared to models trained only on right-handed writers
(see Figure 1), it is unclear how well the measured MI and
entropy translate to the real-world uncertainty.
Therefore, verifying uncertainty remains a limitation in our
interpretation. While we can discriminate between the
entropy associated with diferent samples, pre-defining
thresholds for uncertain samples is challenging due to
the following reasons: (
JMLR, volume 80, 2018, pp. 2611–2620. [23] F. Ott, D. Rügamer, L. Heublein, B. Bischl, [9] W. J. Maddox, T. Garipov, P. Izmailov, D. Vetrov, C. Mutschler, Joint Classification and Trajectory A. G. Wilson, A Simple Baseline for Bayesian Un- Regression of Online Handwriting using a Multicertainty in Deep Learning, in: NIPS, 2019, pp. Task Learning Approach, in: WACV, Waikoloa, HI, 13153–13164. 2022, pp. 266–276. [10] C. Blundell, J. Cornebise, K. Kavukcuoglu, D. Wier- [24] F. Ott, D. Rügamer, L. Heublein, T. Hamann, J. Barth, stra, Weight Uncertainty in Neural Network, in: B. Bischl, C. Mutschler, Benchmarking Online ICML, volume 37, 2015, pp. 1613–1622. Sequence-to-Sequence and Character-based Hand[11] E. Daxberger, A. Kristiadi, A. Immer, R. Eschen- writing Recognition from IMU-Enhanced Pens, in: hagen, M. Bauer, P. Hennig, Laplace Redux - Efort- arXiv:2202.07036, 2022.
less Bayesian Deep Learning, in: NIPS, 2021. [25] F. Ott, D. Rügamer, L. Heublein, B. Bischl, [12] B. Lakshminarayanan, A. Pritzel, C. Blundell, Sim- C. Mutschler, Domain Adaptation for Time-Series ple and Scalable Predictive Uncertainty Estimation Classification to Mitigate Covariate Shift, in: Using Deep Ensembles, in: NIPS, 2017, pp. 6405– arXiv:2204.03342, 2022.
6416. [26] C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, On [13] E. Hüllermeier, W. Waegeman, Aleatoric and Epis- Calibration of Modern Neural Networks, in: ICML, temic Uncertainty in Machine Learning: An Intro- volume 70, 2017, pp. 1321–1330. duction to Concepts and Methods, in: Machine [27] D. Wu, L. Gao, X. Xiong, M. Chinazzi, A. Vespignani, Learning, volume 110, 2021, pp. 457–506. Y.-A. Ma, R. Yu, Quantifying Uncertainty in Deep [14] A. Kendall, Y. Gal, What Uncertainties Do We Need Spatiotemporal Forecasting, in: arXiv:2105.11982, in Bayesian Deep Learning for Computer Vision, 2021.
in: NIPS, volume 30, 2017, pp. 5580–5590. [28] Z. Zhou, Y. Wang, X. Xie, L. Qiao, Y. Li, STUaNet:
[15] Y. Kwon, J.-H. Won, B. J. Kim, M. C. Paik, Uncer- Understanding Uncertainty in Spatiotemporal
Coltainty Quantification Using Bayesian Neural Net- lective Human Mobility, in: WWW, 2021.
works in Classification: Application to Ischemic [29] J. A. Gómez, C. Guan, P. Tripathy, J. C. Duque, S.
PasStroke Lesion Segmentation, in: ICLR, 2018. sos, M. Keith, J. Liu, Analyzing the Spatiotemporal
[16] L. Smith, Y. Gal, Understanding Measures of Uncer- Uncertainty in Urbanization Predictions, in:
Retainty for Adversarial Example Detection, in: UAI, mote Sensing, volume 13(512), 2021.
2018. [30] J. A. Hoeting, D. Madigan, A. E. Raftery, C. T.
Volin[17] S. Depeweg, J.-M. Hernandez-Lobato, F. Doshi- sky, Bayesian Model Averaging: A Tutorial, in:
Velez, S. Udluft, Decomposition of Uncertainty Statist. Sci., volume 14(
AAAI, volume 216, 2014, pp. 76–102. [34] M. Hollemans, Reliability Diagrams, https://github. [20] R. Plamondon, S. N. Srihari, On-line and Of-line com/hollance/reliability-diagrams, 2020. 0.006 ba 0.005 dec 0.004 hgf 0.003 kij
l 0.002 mno 0.001 qpr 0.000 ust
v 0.001 w x y z 80 0.020 0.015
A.1. Model and UQ Method Parameters For reproducibility, we state all general model architec- (a) Sample accuracies below and (b) Histogram visualizing the ture parameters and propose training parameters for the above an entropy threshold. entropy distribution. SWAG model. For all experiments we use Nvidia Tesla Figure 7: Accuracy and entropy for the SWAG CNN+TCN V100-SXM2 GPUs with 32 GB VRAM coupled with Intel model trained on the combined WD (right-handed only) Core Xeon CPUs and 192 GB RAM. dataset. 4 6 321 iittrsyobpnnE 3to01unC45312 70 75 80 85 90 95 100
Acc. > Threshold Model Parameters. We use a CNN with dropout rate 20%, convolutional layers with kernel size 4 and filter size 200. The temporal cell (LSTM, BiLSTM or TCN) contains 100, 100 or 120 neurons, respectively. We interpolate the time-series to 64 time steps, and train the model for 2,000 epochs with early stopping and a batch size of 50.
SWAG Parameters. We initialize the stochastic gradient descent (SGD) optimizer with initial learning rate 10− 2, a momentum of 0.9, and weight decay of 10− 4.
The stochastic weight averaging (SWA) burn-in period was run for 10 epochs. SWAG showed a training process with fast convergence. "V", and "T" and "X". These confusions can be identified with the aleatoric and epistemic uncertainty and correspond with the classification accuracies. Overall, the uncertainty for lowercase characters is higher (see Figure 6a) since the writing style of lowercase characters is oftentimes quite similar, e.g., "r" and "v", "u" and "v", "h" and "n", and "d" and "q". This also leads to a lower classification accuracy (see Figure 6c). ion1.00 t a rm0.75 o f in0.50 l tau0.25 u M0.00 s itb1.50 n in1.25 o i ta1.00 rom0.75 f lin0.50 a tu0.25 u M0.00
A C EG I KMOQ SUWY a c e g i kmo q s u w y
Letters (a) Mutual information per letter.
ACEG I KMOQSUWY a c e g i kmo q s uw y
Letters (b) Entropy per letter. A.3. SWAG Model Results between SWAG and Deep Ensemble models. Figure 9 shows the MI and entropy for the SWAG model with the This section provides plots for the SWAG model that same pattern as for the Deep Ensemble model with lower can directly be compared to the previously shown Deep absolute values. In Figure 10, we see the same overconfiEnsemble model plots. We observe very similar results a b c d e f g h ij k l m n o p q r st u v 0.005 w x y z 0.005 0.000 A B C D E F G H JI K L M N O P Q R S T U V W X Y Z dence on left-handed data for SWAG models that have characters only, and in Figure 13 for lowercase characters never seen this data similar as for Deep Ensemble mod- only of the SWAG model show the same pattern as the els. The ECE by the SWAG model is marginally lower heatmaps for Deep Ensemble models. than the ECE by the Deep Ensemble model, but follows the same trend. The heatmaps in Figures 11 for lowercase and uppercase characters, in Figure 12 for uppercase 0.006 BA 0.005 DCE 0.004 HGF 0.003 KJI
L 0.002 MNO 0.001 QRP 0.000 UST
V 0.001 W
X Y
Z 0.006 ba 0.005 dec 0.004 hgf 0.003 kij
l 0.002 mno 0.001 pqr 0.000 ust
v 0.001 w x y z 60 ][% y c ra 40 ccu
A 20 0 100 80
(c) Confusion matrix of accuracy.