-Touch Dynamics is the behavioral biometric trait that regards how the user interacts with devices equipped with touch displays, the dynamic patterns drawn through the swipe movement can be used to identify the user who is accessing the smartphone. In this paper we investigated whether the same data could be used also to recognize some emotional states. To this aim, an Android App was designed to simulate the unlock patterns and collect data needed to calculate numeric features that have been used not only for identification purposes but also to classify three negative affective states: anxiety, stress and depression. Results obtained so far are encouraging and indicate that Random Forest is capable to reach good classification accuracy both on touch numerical features and on negative emotional states classification also exploiting behavior information such the hand and the finger used in the execution.
Security systems use biometric traits to establish the identity of
a person based on their characteristic features which can be
difficult to be counterfeited and cannot be lost or forgotten.
Biometric traits can be Physiological in the case a direct measure
of a human body part can be performed (e.g. iris, fingerprint,
etc.) or Behavioral in the case in which an action performed by
the user is measured (e.g. handwritten signature, walk, etc.).
Behavioral biometrics also involve a cognitive aspect because
actions performed are learned over time and can change
depending on environmental, psycho-emotional and physical
conditions [
The biometric trait taken into consideration in this work is the Touch Dynamic referred as behavioral trait related to how the user interacts with a touch screen of a device (e.g. smartphone, tablet). Exploiting touch dynamics in this context, means investigating on features like the pressure applied on the screen, finger-display touch area, the speed of swipes, the variation of the device sensors (e.g. accelerometer), and so on.
One of the advantages of exploiting this biometric trait is total transparency since the user no needs to use unfamiliar devices or wear sensors, or interact differently from his habits, but the data are recorded and analyzed automatically in a natural way. As a side effect of the identification/verification task, due to the intrinsic capabilities of a behavioral biometric, emotional states of the user can also be revealed. In this work negative emotional states (anxiety, depression and stress) have been considered and collected by a specific questionnaire. Results obtained so far indicate that Random Forest is capable to reach good classification accuracy both on touch numerical features and on negative emotional states also exploiting behavior information such the hand and the finger used.
The work is organized as follows: in Section II related works and literature are presented; Section III introduces the negative affective states. In Section IV the smartphone App is described. The dataset and the numeric features are depicted in Section V while Section VI presents the experimental phase and, finally, Section VII contains conclusions and future work directions.
An important part of studies when considering user behaviors on
devices concerns the password entry and how approaching
them. Draffin et al. [
III. THE DASS-42 QUESTIONNAIRE AND AFFECTIVE STATES Stress, anxiety and depression are responses to the challenges of everyday life and results useful to detect and prevent them before they impact on individual health and daily actions: if a device detects a negative emotional state, an Intelligent System can warn user with appropriate solutions, for example by lightening its work schedule or organizing the environment and the interface in a more relaxing way with suitable icons, colors, sounds and brightness. The Stress is a general adaptation syndrome designed to re-establish a new internal balance following changes in internal balance at the humoral, organic and biological levels; in the physical response it shows tachycardia, muscle contraction and other factors typical of the "fight or flight" response. The Anxiety is a state characterized by intense feeling of concern and fear, often unfounded, related to a specific environmental stimulus associated with a failed response of adaptation and is often accompanied by palpitations, shortness of breath and tremor with response of "fight or flight". Depression is a disorder characterized by mood episodes accompanied by low self-esteem and loss of interest in normally pleasant activities. It is a debilitating disease that involves both the affective and cognitive spheres affecting work, sleep and physical health with a strong impact on style and quality life.
The DASS-42 questionnaire is called ‘Depression, Anxiety and
Stress Scales’ [
To date there is no available public dataset related to touch tasks and emotional states. To the aim of data collection, an application has been specifically developed.
The application is called EmoTouch Lock (EMOtion and TOUCHdynamics in a LOCKscreen) since it was designed to simulate unlock patterns on an Android smartphone device collecting data on user touch behaviors.
At the first usage, the user is required to provide age and gender info, successively 4 tabs are displayed: Survey and Easy/Medium/Difficult Pattern. The first one allows access to the DASS-42 questionnaire while the three ‘pattern tabs’ offer a touch sequence to be executed (Fig.1) with different difficulty due to the number of swipes to be executed.
When a specific task is proposed by the system, for example the Easy one, a pop-up appears (Fig.2, left) containing a tutorial showing how to complete the sequence without leaving the finger from the touch screen. Next, a screen with the 9 points to be linked following the proposed pattern is presented (Fig. 2, right).
User data are recorded from the first touch until the finger is lifted; whether the inserted pattern is correct or not the data are sent to the server.
The data and the raw values acquired by the sensors must be transformed and adapted to be used effectively in machine learning classifiers and models.
The first stage of is swipe extraction. A swipe is a touch interaction with no sensible curvature. Figure 3 provides an example of the execution of the ‘Medium’ difficulty task. Considering the pattern execution sequence (from A to D), black dots are the sampled coordinates projections of the user touch while red dots highlight instants where the touch changes direction: a swipe is the sequence of dots sampled until a direction change (the ideal red line in the figure). To determine the red points in a pattern execution, the angular values calculated with respect to the horizontal axis between a sequence of three black points has been considered and empirically matched against a threshold of 135 degrees.
A set of numerical features have been extracted from the swipe as reported in Table I. In other words, 14 features characterize each single swipe. Among the others, ‘line deviation’ and the ‘swipe direction’ are the most important. The former represents a change in the touch movement while the latter considers the deviation of the movement from the ‘ideal’ trajectories requested from the task, indicating how the user designed a swipe in a linear manner. Numerical features calculated from the drawn swipes in each task. Some features (Table I) have been calculated in different ways based on the points contained in a swipe as follows: ● PP: the feature has been calculated point-to-point, i.e. all the points are considered in pairs (point 1-point 2, point 2-point 3, point 3-point 4, etc.) ● SP: the first point of the pair is always the starting point of the sequence (start point - point 1, start point - point 2, start point - point 3, etc.) ● EP: unlike SP, indicates that the end point is taken into account rather than the starting point
A.
In order to perform experiments, 40 distinct subjects (university students, mean age of 22 years, 6 females and 34 males) have been asked to execute the swipe tasks, with a minimum of 5 patterns for each difficulty level, for a total of at least 15 patterns per users. The only constraint was that a new pattern could be executed only after (at least) 10 minutes after completion of the previous one while the subjects used the app in an uncontrolled real environment, recording data on the train, walking, sitting and so on; moreover, there was not a limit on the interaction mood since subjects were able to complete the patterns using the most comfortable hand and fingers (however, such information have been requested at the end of each task execution). Due to the freedom given to participants, not all of them reached the same number of recorded patterns and so, after a data cleaning operation, to maximize the possibility of comparisons it was decided to divide them into three experimental groups, also overlapped to consider the subject skills progression over time. So that, the entire dataset has been balanced into the following:
B.
At the initial stage, a set of classifiers has been adopted to evaluate classification accuracy at single swipe level (i.e. not at entire pattern level). The following have been considered: ● J48: algorithm for the generation of a C4.5 decision tree, pruned and not pruned ● Support Vector Machine: a model that assigns one of two classes separated through support examples ●
● Bayesian Network: a probabilistic model that exploits variables and their conditional dependencies ● Naive Bayes: requires knowledge of the a priori and conditional probabilities related to the problem.
Classification results of swipes with features on the whole dataset. Accuracy
Classification results, considering the whole dataset about the swipe features goodness, are in Table II while a 10-fold cross validation mode has been employed to reduce the impact of the variance while choosing the training and test examples. The best performance has been obtained with the Random Forest: 76% of accuracy.
Random Forest has been used for further tests and results related to negative emotional states classification with 10-fold cross validation are reported in Table III where 15 users has been randomly selected with 184 swipe samples each. it can be observed that the recognition accuracy is near 70% and all results improve when exploiting the further behavioral feature set. The best recognized emotional state is the Stress with 73.6% and 78% respectively while the Depression is the one that takes the most advantage from new features (+5%). Finally, the Random Forest classifier has been employed to perform the identity verification tests, checking if it is possible to associate each single swipe to the user who produced it in order to highlight its characterization through this behavioral biometric trait. In other words, understand if it is possible to have identity verification as well as emotional state recognition at unlock time. In this case due to the reduced dimension, for each of the three sub-dataset a leave-one-out setup has been adopted: in this way, if the dataset contains n swipe feature vectors, there are used n-1 for the training and the remaining one for the test for n times. As evaluation metric the Equal Error Rate (EER) has been chosen: it indicates how the proportion of false acceptances is equal to the proportion of false rejections (the lower the EER value, the higher the accuracy of the biometric system is).
From Table IV emerges that best-recognized subjects are those having the more number of attempts for each task execution (dataset C): although they are smaller in number than the other groups, can be considered their greater characterization due to increasingly specific data, along with a progression in the mastery of the execution skills. Results are noteworthy, with the highest error of only 23% on dataset A and the best classification results on dataset C where error decreases about 50%.
This study introduces the Touch Dynamics method as biometric trait able to recognize a users’ emotional states as information useful to characterize and distinguish users as well as to adapt contents and behavior of smartphones. A set of tasks related to touch un-lock patterns have been exploited to extract numerical features on a smartphone devices demonstrating that that touch swipes are useful for recognizing the identity with an EER of 12% for users with a huge practice using the Random Forest. The exploit of emotional states has been performed adopting the DASS-42 questionnaire whose results will be further inquired in future studies; in this case, classification accuracy is around 70%. Starting from this work several improvements are possible, such as adding more features, classification methods and swipe patterns to test how users improve their performances. Equally necessary is to have the users balanced between male and females along with a feature selection study to inquire which are the features with the major informative contribution. Furthermore, the proposed methodology could be adapted to new contexts, exploiting swipes made by users while moving around the device interface or in other applications.
This work is supported by the Italian Ministry of Education, University and Research within the PRIN2017 - BullyBuster project - A framework for bullying and cyberbullying action detection by computer vision and artificial intelligence methods and algorithms. CUP: H94I19000230006.