Close friends can recognise each other from subtle behaviour cues: from the way one walks, the proficiency with which they complete certain tasks, posture, and other aspects. With an ever-increasing number of sensors built into everyday objects, computers should be capable of identifying users from their behaviour reflected in the sensor-sampled signals. While behaviour-based authentication systems have recently been demonstrated, their utility is still questionable. More specifically, whether and to which extent diferent sensing modalities capture long-term persistent user behavioural traits represents an open question. In this paper we tackle this issue by analysing the informativeness and temporal invariance of data collected for the purpose of user identification in an Internet-of-Things ofice-like environment. We discover that the most informative sensors are not necessarily reflecting the most stable behavioural traits, which may have consequences for the future development of sensor-based authentication systems.
The weaknesses of one-of authentication methods, such as passwords [
The emergence of Internet of Things (IoT) brought approximately 30 billion sensor-enabled
devices to a wide range of environments [
Although multimodal IoT-based authentication has been demonstrated, it is still not clear whether and to what extent diferent sensing modalities contribute to the end goal of user identification. Answering this question would sharpen the focus, so that sensors reflect very general kinds of behaviour could be discarded and more informative sensors would be deployed instead. Furthermore, behaviour-based authentication relies on (machine learning) models of individual behaviour that have to be constructed beforehand. The models enable the actual authentication only if the sensor data collected at the test time matches the data used for the model training. Large discrepancies among the test and the training datasets stemming from the variability of patterns registered by the sensors would render the system unusable. The question of whether the behaviour reflected in a certain sensor modality is “stable” or not is yet another open research question.
In this paper we tackle the above issues and study the informativeness and temporal persistence of IoT-based sensor data when it comes to user authentication. Our work is based on the analysis of a previously compiled dataset containing real-world sensor data collected while 20 users performed three diferent tasks two times each. The richness of the dataset (six diferent sensing modalities) enables us to answer the above research questions and bring the following specific contributions to the area: • We calculate two metrics of feature informativeness brought by diferent IoT sensors and identify the most informative (combinations of) sensors for the task of behaviour-based authentication; • We examine stability (temporal invariability) of diferent sensing modalities and identify sensor which can perform robust behaviour-based authentication across diferent usage sessions; • We propose and discuss alternative sensing modalities that could address the deficiencies of the sensors used in the analysed dataset and potentially increase the reliability of behaviour-based authentication.
The analysis performed in this paper points towards limited informativeness and a rather
low stability of any single modality used for sensor-based authentication. Indeed, this has been
hinted before, as multimodal continuous authentication has been shown to be far more accurate
than any one-of authentication method relying on IoT sensors [
Behavioural biometrics based authentication focuses on employing user’s various behavioural patterns to validate one’s identity. Modern approaches mostly rely on either sensor-rich handheld and wearable devices, such as smartphones and smartwatches, or on exploiting sensor-equipped IoT environments.
Most common techniques on smart devices focus on either the user’s interaction with the
screen, which includes the analysis of navigation gestures, known as touch dynamics [
Rather than burdening the user to never forget her authentication device, IoT
environmentsbased authentication focuses on identifying the user without any concious actions from her,
such as carrying a device, or even providing her fingerprint or remembering her password.
Moving sensors from the user to the environment, IMUs still prove to be an efective modality
for authentication [
The issue with most behavioural biometrics is the monomodality of their approaches, for
example gait dynamics work well when a person is walking, while the moment she sits down, it is
impossible to recognise her based on the way she walks. To overcome this limitation, multi modal
approaches, that ensemble diferent sensing modalities into a more complete authentication
system, started emerging. An overview of modality usage in biometric authentication by
Ryu et al. [15] suggests most commonly used modalities in such systems include all of the
above modalities, frequently tied to physiological biometrics, such as fingerprints and facial
recognition. We follow the multi modal doctrine by collecting and publishing a dataset of
diferent environmental sensors [
There are not many publicly available multi-modal IoT datasets that extend beyond data from
smartphones and wearables. Therefore, we utilise a dataset we collected in our previous
research [
We ask the participants to perform each of the devised tasks twice. The first task includes copying a body of text from a cardboard card to a PC and send it via email and it functions as a keyboard typing exercise, the second task requires the participants to look up for a weather forecast of a place of their choosing and suggest a couple tourist attractions to visit, based on the forecast, which should stimulate diferent PC usage patterns, and the third task sends the participants on a “treasure hunt”, which includes navigating through a series of boxes including instructions on how to proceed and are placed around the environment. In the final box the instructions state to produce a graph from the data that are provided alongside the instructions.
The sensor we employ include; a) an accelerometer and b) a gyroscope, purposed to infer user’s keyboard typing patterns, c) four force sensors, positioned on the corners of a plate that is placed under the mouse and keyboard, purposed to infer user’s posture behind the computer, d) a PC monitor tool collecting the information about CPU, memory and network usage, purposed to infer user’s computer usage patterns, and e) six infrared sensors, positioned around the environment, purposed to infer user’s ofice navigation patterns. The complete dataset contains 115M raw datapoints over a span of fiteen hours and forty minutes.
With the dataset acquired, we first apply some basic filtering techniques to exclude
participants who either failed to follow the instructions, or we experienced technical dificulties with
the data collection process. We retain fiteen users that successfully completed all six (2 ×3)
tasks and do not have any missing sensor data. We process the remaining data by calculating
time- and in case of accelerometer, gyroscope and force sensors also frequency domain features.
A complete list of generated features is presented in [
In the following sections we utilise either the complete dataset (Section 4), or separate it on task level, i.e. taking both sessions of a given task (Section 5).
To better understand the inner workings of a multi-modal, environmental IoT authentication system, we calculate and analyse the informativeness of our generated feature set. That will enable us to better understand how to devise such systems in the future, especially in terms of selecting best performing sensing modalities, and replacing ones that do not provide suficient user inference power. We estimate the informativeness of each feature based on two diferent informativeness metrics; relative mutual information and gini importance.
Our first informativeness evaluation metric is the Relative Mutual Information as defined in [
where ( ) is the user entropy and ( | ) is the conditional user entropy of a given feature.We plot thirteen best scored features in Figure 2. Each bar represents a feature score. The key follows a simple pattern: xy_zzz, where x equals to a given sensor (a – accelerometer, g – gyroscope, f – force sensor, m – memory usage), y (optional) equals to an axis (x, y, z) for accelerometer and gyroscope, or a specific force sensor (a, b, c, d), while zzz equals to the generated feature (me – mean, min – minimum, max – maximum, mcr – mean crossing rate, sd – standard deviation).
( ) − ( | )
( ) Feature ax_me gz_me ay_me a_me gx_me fa_me m_min fd_me m_me m_max gy_me ay_mcr fd_sd fb_me az_me 0.6 ion 0.5 t a rom 0.4 f n I lua 0.3 t u eM 0.2 v it lea 0.1 R
0
We observe a prevalence of time domain features, with the mean values of diferent senors being the most informative feature in most occurrences. Regarding the sensor modalities, accelerometer provides best scores, followed by the gyroscope, force sensors, and memory usage. A similar trend extends beyond the displayed scores, as most frequency domain features, as well as data gathered from the infrared sensor are found as the least informative aspects of the gathered dataset.
The other metric we utilise to evaluate the informativeness of generated features is the normalised gini importance. It is calculated by building a random forest machine learning model and averaging each feature’s contribution to the decrease of impurity over trees, generated by the model. We choose the information gain as the decrease of impurity criteria, with the other available criteria – gini index, yielding similar results. The importance scores are normalised, meaning that the scores of all features sum up to one. We note that the gini importance score tends to favor either numerical features, or categorical features with high number of values [16]. All of our generated features are numerical, hence considered equally by this method.
Top fiteen gini importance scores are displayed in Figure 3. We reuse the key to note features in the plotted graph. Similarly to relative mutual information, features from the accelerometer and gyroscope are ranked highest, followed by the force sensors and memory usage patterns. Furthermore we observe the dominance of time domain features.
0.1 0.08 e c tran 0.06 o p Im 0.04 ii n G 0.02 0
Feature ax_me ay_me gx_me gz_me a_me fd_me gy_me m_me ay_mcr fa_me m_max az_me az_mcr a_mcr m_min
One major hurdle in behaviour based authentication are the varying user behaviour patterns
that may be efected by the user’s mood, cognitive load, and comfort level. Our work so far is no
exception and the drop of accuracy in user inference when taking into account multiple sessions
is significant. For instance, in [
We base our analysis on a two-dimensional linear discriminant analysis (LDA) projection, which is a supervised dimensionality reduction technique. To capture potential inter-sensor dependencies, we take all possible combinations of sensor pairs, as well as all possible combinations of user triplets. With the five diferent sensor modalities, and fiteen diferent users, we end up with (52) ∗ (135) = 4550 diferent projections.
acc_gyr acc_frc acc_pc acc_ir gyr_frc gyr_pc gyr_ir frc_pc frc_ir pc_if
In Figure 4 we display an example of LDA projections for three random users and all diferent sensor combinations. In each subfigure, circles represent projected LDA points, while pentagons are their centroids. Each colour equals to a single user session, where the following colour pairs correspond to the same user: (green, purple), (brown, light blue), and (yellow, dark blue). The titles represent two sensors that were taken into account in the specific projection, divided by an underscore (acc – accelerometer, gyr – gyroscope, frc – force sensors, pc – pc monitor, and ir – infrared sensors). All subfigures are in the same scale, hence we are able to directly compare the clustering capability of each sensor modality pair.
Even to the naked eye, the ability to successfully cluster data of diferent users, varies with diferent sensor modality pairs. With either accelerometer or gyroscope included, the users clusters are clearly separable one from another, while in the absence of these two modalities, we observe clusters starting to blend.
The convenience of visual representation of data with a 2D LDA enables us to quickly estimate result trends. However, to empirically evaluate clustering capabilities of diferent sensor modality pairs, we utilise the silhouette score, which is defined for each point as: () = ( − )/ ( , ) where is the mean intra-cluster distance to a given point, and the mean distance to the points of the nearest cluster our given point is not a part of. Each individual value ranges from −1 to 1, with the latter representing the best possible value, while being closer to the former signifies a mis-clustered point. Mean of all silhouette scores represents a mean silhouette score of a sample i.e. data of three users and a sensor modality pair. To obtain the per-sensor modality pair score, we average all mean silhouette scores of a given sensor modality pair.
In Figure 5 we observe the final silhouette scores. In line with our assumptions from visually observing the LDA clusters, the accelerometer and gyroscope sensor modalities yield good results, with the combination of the two being the best overall. More surprisingly, the force sensor modality, which was scored much lower in informativeness scores (Section 4), is almost on a par with their silhouette scores. On the other side of the spectrum, including the infrared, and to a lesser extent the pc monitor sensor modalities, lowers the silhouette score, as those modalities do not contribute much to the user inference.
1 0.8 e r cso 0.6 e t t e uo 0.4 h li S 0.2 0
Sensor modalities accelerometer & gyroscope accelerometer & force accelerometer & pcmonitor accelerometer & infrared gyroscope & force gyroscope & pcmonitor gyroscope & infrared force & pcmonitor force & infrared pcmonitor & infrared
removing the mean and scaling to standard deviation = − Finally, our main goal of this section is to asses the consistency of diferent sensor modalities in inter-session user behaviour. We evaluate this by measuring the centroid distance of the session clusters that represent the same user performing the same task twice (two sessions). These are the distances between neighbouring pentagons in Figure 4. Each projection is standardized by before calculating the centroid distances.
Once more the accelerometer and gyroscope perform well. However, the best obtained (shortest) average centroid distance is achieved by pairing the acceleromenter with the force sensors, while the pairing with the gyroscope is not far behind. This result matches our findings with the mean silhouette score analysis and exhibits that while force sensors are not best suited for recognising users (as shown in Section 4), they have merit while it comes to inferring user identity accross sessions. We elaborate on this phenomenon in the Discussion section. 0.6 0.5 e
Numerous alternatives to password-based authentication have been proposed by the research
community over the last two decades [
In this paper we analysed a behaviour-based authentication technique that harnesses IoT sensors placed in an ofice-like environment. The mutlitude of sensor modalities used in a corresponding machine learning model poses a question of how and to what extent does a particular modality contribute to user identification. For this, we conducted an informativeness analysis. We find out that the relative mutual information, commonly used in behaviour biometrics-based authentication, does not clearly identify the most informative sensors. Instead, gini importance metric tends to clearly point out to sensors that are “closer” to a user, in our case accelerometer and gyroscope placed on a keyboard, as sensors that are likely to enable discernability among individuals.
Human behaviour tends to vary over time. Typing speed, for instance, might be modulated by a person’s knowledge of the typed text, tiredness, emotions, and other factors. Whether the particular aspect of human behaviour captured by a sensor is going to persist in the same manner over time is a question we aimed to answer in the second part of the paper. Here we first presented the LDA projections to visually compare users’ behaviour across the same tasks reexecuted at diferent points in time. Then, we calculated the silhouette scores to assess whether a particular modality is likely to be informative over a longer period of time. Surprisingly, we ifnd that a sensor that was not itself highly informative for user authentication – the force sensor – introduces stability not observed with some more informative sensors, such as the gyroscope. We further confirm this by calculating the centroid distances among clusters formed upon data stemming from diferent combinations of sensors. Centroids corresponding to the same person should not “move” across sessions, if the sensor-reflected behaviour is persistent. We find that including the force sensor in the mix indeed leads to smaller centroid displacements.
Why is behaviour reflect in the force sensor persistent? In the setup we analysed the force sensors were placed under the front panel of a desk the users were working at. The sensors, thus, likely capture the behaviour related to a user’s general posture at a desk – the way one leans on the desk and the way arms are held during the typing. We postulate that unlike some other modalities, such as the typing speed, the force sensor reflects behaviour that does not change much between two sessions in the described experiments. While people may type faster or slower depending on whether they are familiar with a task or not (and indeed, we noticed that in the second repetition of the same task they often tend to type faster), they are unlikely to change their posture due to their perception of the task.
Our identification of a rather crude force sensor as a pillar of stability in behaviour-based authentication calls for the reconsideration of sensor types used for identification in IoT environments. Recently, wireless sensing has demonstrated impressive results: millimeter wave radars have been used to construct detailed 3D meshes of human users [18] and the same radars have been used for gesture recognition [19]. Guided by the above results and the fact that the sensing range can be tuned according to the distances and the details of interest, in our future work we aim to include wireless sensing in order to capture general postures of users and harness these for authentication.
In this paper we analysed real-world sensor data collected from an ofice-like IoT environment where 20 users conducted three diferent tasks in two sessions each. We focused on identifying the most informative sensors for behaviour-based authentication. We show that while sensors that the users interact the most carry the highest identification potential, the behaviour they reflect is not necessarily the most stable. Instead, we observe that the sensors reflecting more intrinsic properties of users, such as their posture, tend to exhibit a higher temporal invariance. Consequently, our work stresses the need for additional sensors to be included, should mutlimodal IoT sensing become a viable authentication method in the future. Method for Wearable IoT Devices, IEEE Internet of Things Journal 6 (2019) 820–830. doi:1 0 . 1 1 0 9 / J I O T . 2 0 1 8 . 2 8 6 0 5 9 2 . [13] C. Feng, J. I. E. Xiong, L. Chang, F. Wang, J. U. Wang, D. Fang, RF-Identity : Non-Intrusive
Person Identification Based on Commodity RFID Devices 5 (2021) 1–23. [14] P. Zhao, C. X. Lu, J. Wang, C. Chen, W. Wang, N. Trigoni, A. Markham, Human tracking and identification through a millimeter wave radar, Ad Hoc Networks 116 (2021) 102475.
URL: https://doi.org/10.1016/j.adhoc.2021.102475. doi:1 0 . 1 0 1 6 / j . a d h o c . 2 0 2 1 . 1 0 2 4 7 5 . [15] R. Ryu, S. Yeom, S. H. Kim, D. Herbert, Continuous Multimodal Biometric Authentication Schemes: A Systematic Review, IEEE Access 9 (2021) 34541–34557. doi:1 0 . 1 1 0 9 / A C C E S S . 2 0 2 1 . 3 0 6 1 5 8 9 . [16] S. Nembrini, I. R. König, M. N. Wright, The revival of the Gini importance?, Bioinformatics 34 (2018) 3711–3718. doi:1 0 . 1 0 9 3 / b i o i n f o r m a t i c s / b t y 3 7 3 . [17] S. Sugrim, C. Liu, J. Lindqvist, Recruit until it fails: Exploring performance limits for identification systems, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3 (2019) 1–26. [18] H. Xue, Y. Ju, C. Miao, Y. Wang, S. Wang, A. Zhang, L. Su, mmmesh: towards 3d real-time dynamic human mesh construction using millimeter-wave, in: Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services, 2021, pp. 269–282. [19] S. Palipana, D. Salami, L. A. Leiva, S. Sigg, Pantomime: Mid-air gesture recognition with sparse millimeter-wave radar point clouds, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5 (2021) 1–27.