Automatic and unobtrusive inference of a user's cognitive load could revolutionise interaction with computing systems. Wireless ranging represents an attractive alternative to the current state-of-theart solutions relying on wearable physiological sensors. Yet, which wireless signal features will be the most informative for cognitive load inference remains an open research question. Due to their way of deconstructing input features into intermediate hidden layers, and reconstructing them into an output vector as similar as possible to the input data, autoencoders are on paper perfectly suited to help highlight the most important features in input data. In our study, we discover that using autoencoder for cognitive load inference from wireless signals can indeed sometimes beat classical machine learning classifiers in terms of accuracy. Nevertheless, despite the theoretical advantage, we find that autoencoders often struggle to reach the same accuracy levels as advanced traditional machine learning solutions. Consequently, we warn against over-reliance on deep learning methods in human-computer interaction research.
(V. Pejović)
Thirty years after Mark Weiser’s vision of human-computer interaction (HCI) being as seamless
as a walk in the woods is still not a reality [
Eforts to enable automatic cognitive load inference often rely on the physiological response
a person experiences when put under high cognitive load. In such cases, the person’s heart
activity becomes faster and more uniform, skin temperature varies, onsets of sweating appear,
and pupils dilate, to name a few relevant physiological reactions. Initial attempts to infer a
user’s cognitive load relied on lab-based equipment that required a user to be strapped to
specialised sensors, thus were of little practical utility in the field of HCI [
A few recent attempts have been made to wirelessly sense cognitive load [
In this work, we use automatic feature extraction to try and find new features in order to
improve cognitive load inference from wireless signals. For this purpose we use the
autoencoder [
Using data obtained with wireless ranging of 23 participants solving six diferent task types at various dificulty levels, we assess the autoencoder’s ability to derive features enabling high-accuracy cognitive load inference. Specifically, the contributions of our work include: • We craft four diferent types of autoencoders with the goal of automatic feature extraction from wireless phase signals; • We systematically evaluate the above autoencoders on various flavors of the cognitive load inference problem; • We compare the autoencoder-based solutions with a range of conventional machine learning and deep learning solutions to assess the potential of autoencoders for real-world wireless cognitive load inference.
Contrary to the expectations, our results show that the traditional breathing and heart rate features usually trump autoencoders and produce the highest prediction accuracy. There are a few cases, however, where autoencoders can match or improve the prediction accuracy. We suspect our findings demonstrate that, despite their modelling power, deep learning-based methods need not be the most appropriate machine learning solution in cases where the data is noisy, or when the amount of data is not enough, and where a substantial body of expert knowledge on feature extraction already exists.
Cognitive load is defined as “ a multidimensional construct representing the load that performing
a particular task imposes on the learner’s cognitive system” [
Harnessing the physiological reaction catalyzed by an increase in the cognitive load,
researchers have devised methods for inferring cognitive load from various physiological signals.
A study from 2010 used a contactless eye tracker, an ECG-enabled armband, a wireless EEG
headset and a wireless HR monitor to measure a participant’s physiological signals while solving
elementary cognitive tasks. Using some of those signals the authors were able to achieve over 80
% classification accuracy of the participant’s cognitive load defined as a two-category construct
(high vs low load) [
Moving towards in-situ inference, of-the-shelf wearable devices have been recently used
for cognitive load inference. In [
Wireless cognitive load inference has recently been investigated in the Wi-Mind study [
In the accompanying paper, the authors of Wi-Mind conducted a study with 23 volunteers
who, while seated in front of a computer, individually solved tasks of six diferent task types,
which are specifically designed to elicit a cognitive load response. Each task type has three
dificulty levels – low, medium and high – for a total of 18 tasks. The tasks fall into the group of
elementary cognitive tasks (described in [
To infer cognitive load from wireless signals the authors of Wi-Mind used carefully engineered features representing diferent aspects of a person’s breathing and heartbeat activity. The resulting features were then used in various classifiers, yielding a 75% accuracy rate in diferentiating between busy (working on a task) and relax times, and a 65% accuracy rate in detecting transitions between two task dificulties, while achieving 52% accuracy of discerning between low and high cognitive engagement (corresponding to low and high task dificulty). A more fine-grained distinction among easy, medium, and high dificulty tasks was not possible.
Autoencoders (AE) are a type of neural networks [
Similarly to standard neural network models, AEs are trained through backpropagation, with the diference being that the input provided to the AE is also the target output, whereas standard neural networks have a target output diferent to their input. Diferent regularization mechanisms, such as the limitation on the sparsity of network layers, or the Frobenius norm of the Jacobian matrix of the encoder activations with respect to the input, lead to diferent flavors of AEs, such as sparse, contractive AEs [11] and others.
In this paper we examine whether automatic feature extraction using autoencoders can help with cognitive load inference from wireless data. We use the wireless signal phase data collected from 23 participants performing the cognitive load elicitation experiments described in Section 2.2. Each participant was assigned a randomly generated identification string (ident), which is used instead of personally identifiable information.
We segment the wireless signal phase data into 30-second time frames (the length of the relaxation period during the Wi-Mind study). We examine several aspects of detecting cognitive load for which we try to infer the user’s state during a time frame: • Busy vs Relaxed (BvR): whether a user is busy or relaxed (solving a task vs being instructed to relax); • Low vs High (LvH): whether a busy user is solving a low or high dificulty task; • Low vs Medium vs High (LvMvH): whether a busy user is solving a low, medium or high dificulty task.
For the BvR problem we use one relax time frame and one busy time frame for each task (before preprocessing), because each period of relaxation only last as long as one time frame. While for the LvH and LvMvH problems we use 10 consecutive time frames from each task (before preprocessing). For each of these problems we first use all task types (FA, GC, HP, NC, PT, SX), then we explore how our models perform when using only one task type at a time. For classification purposes we use a range of machine learning models: • Simple Dense (SD): a simple neural network with 1 dense hidden layer made using Keras; • Long Short-Term Memory (LSTM): an LSTM based classifier made using Keras, based on a classifier made in the original Wi-Mind paper (an LSTM is a type of neural network especially useful for time series data); • k-Nearest Neighbors (kNN): a k-nearest neighbors algorithm from the scikit-learn library; • Support Vector Machine (SVM): a support vector machine based classifier from the scikitlearn library; • Naive Bayes (NB): a naive Bayesian classifier from the scikit-learn library; • Random Forest (RF): a random forest based classifier from the scikit-learn library; • XGBoost (XGB): the XGBoost classifier from the XGBoost python package. Using these classifiers we experiment with features extracted by experts (taken from the WiMind paper) and features extracted by AEs. There are 8 feature groups we used as input for classification, 4 of which are not based on AEs and 4 of which are based on AEs: • Phase (PHS): unwrapped phase data; • Breathing (BR): 12 features related to a person’s breathing extracted from the phase data; • Heartbeat (HB): 10 features related to a person’s heartbeats, extracted from the phase data; • Combined (CMB): concatenated BR and HB features (22 features total), • Undercomplete (under.): encoded representation of data as extracted by the undercomplete AE. This AE has no explicit regularization term, but contains a restricted number of nodes in the hidden layer; • Sparse (sparse): encoded representation of data as extracted by the sparse AE. Here, the loss function penalizes activations within a layer; • Deep (deep): encoded representation of data as extracted by the deep AE; • Contractive (contr.): encoded representation of data as extracted by the contractive AE.
Here, the loss function penalizes large derivatives of the hidden layer activations with respect to the input training examples (i.e. the AE should be more robust to variations in the input data).
The PHS, BR and HB features are based on the Wi-Mind paper. Each AE was set to extract 30 features (chosen empirically to be slightly larger than the number of CMB features and due to 30 seconds being the length of each time frame). Altogether there were 7 task types, 3 classification problems, 7 machine learning models and 8 feature groups, which brings the total number of combinations to 1176. For each combination of test ident, classifier and feature group used, we saved the classification accuracy (CA) to a dictionary. Once we have calculated the CA of each combination, we save the dictionary containing the CAs to a file. This was done using data from each task type separately, and then using data from all task types at once (7 times in total).
For each ident there is raw phase data with timestamps describing when a volunteer was solving each task (or relaxing), how much time was spent on that task and how they performed on that task. Each instance of a class is based on a 30-second time frame, which means that any time an instance of a class was shorter than 30 seconds, it is instead skipped. Additionally, there is a meta-feature which denotes whether there is significant noise present in a certain time frame. Taking the noise meta-feature into account, more instances may be skipped.
Table 1 and 2, compare the number of instances before and after noise filtering. The prefix “before” means that there were no instances removed due to the noise meta-feature. The prefix “after” means that the noise meta-feature was used to remove noisy instances. The sufixes determine the class of that row.
For each problem we evaluated all combinations on each task type separately and then all task types together. Reported are the mean accuracies of the leave-one-out approach averaged over all participants.
For the BvR problem we aim to classify whether the user is resting or solving a task. Before solving each task, participants in the study were instructed to relax for a period of time. The data from that period of relaxation was used to define the relaxed class. In contrast, the data obtained while the user was solving any task (regardless of the task type and dificulty) was used to define the busy class.
The results are summarized in Table 3. Some notable highlights for the BvR problem include: • The highest overall accuracy 0.87 was achieved with the XGB classifier using the BR feature group using the NC task type; • The highest overall accuracy using all task types is 0.76 with the RF classifier and BR feature group; • The highest accuracy using only AE based feature groups is 0.74 with the NB classifier and Deep feature group using the NC task type; • The highest accuracy using only AE based feature groups and all task types is 0.66 with the RF classifier and Deep feature group.
For the LvH problem we aim to classify whether the user is solving a task with low or high dificulty, as each task type was performed at three diferent dificulty levels (low, medium, high) by each of the participants. For this problem we only use data gathered while the participants were solving a task with low dificulty or high dificulty. Table 4 summarizes the results. Some notable highlights for the LvH problem include: • The highest overall accuracy 0.67 was achieved with the NB classifier and Contractive feature group on the FA task type; • The highest overall accuracy for all task types is 0.59 with the RF classifier and BR feature group; • The highest accuracy using only AE-based feature groups is the same as the highest overall accuracy with the same classifier, feature group, and task type; • The highest accuracy using only AE-based feature groups and all task types is 0.57 with the SVM classifier and Undercomplete feature group.
For the LvMvH problem we aim to infer whether the user is solving a task with low, medium or high dificulty. Table 5 summarizes the results. Some notable highlights for the LvMvH problem include: • The highest overall accuracy 0.49 was achieved with the SD classifier and PHS feature group on the PT task type; • The highest overall accuracy using all task types is 0.40 with the SVM classifier and CMB feature group; • The highest accuracy using only AE-based feature groups is 0.47 with the NB classifier and Sparse feature group on the PT task type; • The highest accuracy using only AE-based feature groups and all task types is 0.37 with the LSTM classifier and Contractive feature group.
We finally graphically summarize the results for task-invariant classifiers in Figure 1 and across diferent task types in Figure 2. We see that AE-based solution in general tend to perform worse than solutions based on features extracted from the existing domain expertise published in the literature. Nevertheless, we do see that in certain situations (e.g. certain task type, certain problem definition), AE-based solutions tend to be on a par with the expert feature-based solutions.
Comparison of highest mean accuracies for each task
type. No sufix means that all feature groups were taken into account, while the “_AE” sufix means only
AE-based features were taken into account.
In the previous section we constructed a large number of machine learning pipelines for cognitive
load inference. The dataset we used was previously analysed in [
Confident that our work delivers highly-optimised classification pipelines, we assess the autoencoders’ ability to extract features beyond those already known and reported in the related literature. The results from Section 4 show that in most cases handcrafted features (breathingand heart rate-related features) are suficient, and features extracted from autoencoders do not ofer improvements. However, for certain problem-task type combinations autoencoder-based features can ofer the highest CA. For the BvR problem, for the SX task type, AE based features achieve the highest CA of 67%. For the LvH problem, in case of the FA task type, AE based features achieve the highest CA of 67%. Finally, for the LvMvH problem, for the FA task type, AE based features achieve the highest CA of 46%. Only for the LvH and FA task type, the highest CA using AE based features is also the overall highest CA for any task type in that problem.
The findings of our research are surprising. In theory, any function could be modeled with a deep learning model, therefore an AE should be able to find features that are at least as good as the handcrafted ones. Yet, in most cases, despite a rather detailed hyperparameter tuning, we failed to match the performance of handcrafted features. We postulate that the following could be the causes: • Insuficient training data. On average, AEs contain significantly more parameters than classical machine learning algorithms. Consequently, a larger training set is needed. In Section 3.1 we see that for the BvR problem there are less than 400 data instances, while for the LvH and LvMvH problems, each class has more than 900 instances of data. Looking at the tables with results in Section 4, we see that the discrepancies between the highest mean accuracies using only AE feature groups and the highest mean accuracies using all feature groups are the highest for the BvR problem. • Noise in the data. Wireless signals are notoriously noisy. While the handcrafted features may be robust to certain noise artefacts, AEs could mistakenly “hook” on intermittent noise, which could then lead to over-fitted classifiers. Indeed, the contractive AE, which should be the most immune to noise, appears to be performing slightly better than other AE flavors. • Suboptimal neural architectures. In this work we used fixed simple AE architectures.
Whether architectures with a diferent number of layers, diferent layers widths, and other parameters would perform better is an open question.
Riding on the wave of impressive results in computer vision, speech recognition, and other domains, deep learning was quickly adopted by human-computer interaction (HCI) researchers. Yet, while there are no doubts about the dominance of deep learning for problems such as visual object recognition1, its suitability for various HCI problems is not that clear.
1In 2012 a CNN called AlexNet outperformed all other approaches submitted to the ImageNet competition by a large margin, and neural networks have dominated the said competition ever since.
In this paper we pitted deep learning-based automatic feature extraction against features extracted using conventional expert knowledge on the problem of human cognitive load inference from wireless ranging signals. Our analysis encompassed combinations of four diferent autoencoders for automatic feature extraction, seven diferent classifiers, as well as four diferent expert-feature groups, all tested on three diferent cognitive load inference problem flavours and six diferent cognitive tasks performed by 23 volunteers.
The thorough examination hints that deep learning is by no means a panacea. While autoencoder-based feature extraction achieves comparable results as the best expert-based feature extraction for certain tasks, such as SX, it is often outperformed by expert-based extraction and learning. We believe that the particularities of the problem and the dataset, including a relatively modest number of training datapoints, a high level of noise in the data, as well as the lack of a wider neural network architecture search lead to such results. Based on our experience, we caution against an overly optimistic use of deep learning for less-examined problems in the area of human-computer interaction.
This paper is based on the primary author’s bachelor’s thesis [12], with the other two authors
acting as the mentor and co-mentor for that thesis. We would also like to thank the authors of
Wi-Mind [