So far, most research investigating the predictability of human behavior, such as mobility and social interactions, has focused mainly on the exploitation of sensor data. However, sensor data can be dificult to capture the subjective motivations behind the individuals' behavior. Understanding personal context (e.g., where one is and what they are doing) can greatly increase predictability. The main limitation is that human input is often missing or inaccurate. The goal of this paper is to identify factors that influence the quality of responses when users are asked about their current context. We find that two key factors influence the quality of responses: user reaction time and completion time. These factors correlate with various exogenous causes (e.g., situational context, time of day) and endogenous causes (e.g., procrastination attitude, mood). In turn, we study how these two factors impact the quality of responses.
Smartphones facilitate pervasive and continuous interaction between humans and machines
by enabling timely notifications and data collection, while simultaneously gathering sensor
data such as Bluetooth, GPS, and accelerometer. This creates opportunities for smartphone
applications to learn how to ask the right questions at the right moment and in the right
context, while also verifying the accuracy and consistency of user input. Previous research, as
demonstrated in [
However, an outstanding issue remains with user-provided answers as they often fail to
meet the necessary quality standards. There are many possible reasons for this, as described
in Furnham’s work [
The goal of this paper is to provide an in-depth study of the parameters that influence the quality of answers. By quality, we refer to a a low number of mistakes. Diferent individuals possess diverse understandings of questions and context, which leads to variations in the quality of their answers. In this study, we discovered that the correctness of an answer is directly negatively afected by the reaction time, which, in turn, depends on the characteristics of the respondent and the situational context. Thus, controlling the reaction time becomes crucial for establishing meaningful hybrid machine-artificial intelligence.
The reasons for respondents making mistakes when providing answers can be categorized into
four main causes: (i) the context in which the answer is provided (e.g., being alone, at university,
using social media, on Monday) [
Initially, we hypothesized that longer reaction times could negatively afect answer quality due to memory recall issues. The longer the time elapsed since the request, the greater the risk of memory-related problems and, consequently, making mistakes in the answer. Additionally, we hypothesized that longer completion times could also have a negative influence on answer quality. Furthermore, the combined efect of reaction and completion times could further amplify their negative impact on answer quality.
According to our hypothesis, the primary challenge of our study is to determine how variations in reaction time and completion time contribute to errors in answers. However, solving this problem is complex due to the causal relationship that exists between reaction and completion time in a temporal sequence. To address this issue, we have chosen a causal model [16, 17], which establishes a hierarchy that considers the temporal order and logical derivation of variables along the pathway connecting explanatory features and objectives. In this paper, we assess answer correctness by examining the temporal consistency between the respondent’s indicated location at home and the GPS position of the phone at the time of notification.
To achieve the goal of this paper, we analyzed the SmartUnitn 2 database [18], which designed an experiment that lasted four weeks and involved students from the University of Trento. Throughout the experiment, the iLog application [19] was used to ask a set of questions diachronically, enabling the selective asking of multiple questions for situational context, which defined in [20, 21]. And then collecting sensor data from mobile phones.
We study the correctness of the user answers by building a causal model able to capture both the direct and indirect efects of the various features. We compute the correctness of answers by comparing the distance from home, as computed by the device via the GPS information, any time the respondent declares she is at home. Notice that each participant was asked to provide their home address, we then used google maps were compute their home GPS coordinates; Google maps were used to calculate the GPS coordinates of the house.
Features M1: Home phone distance (meter) Completion time Reaction time GPS accuracy Constant M2: Unanswered Questions (count) Reaction time Constant M3: Reaction time (minutes) User Characteristics: Procrastination User Characteristics: Mood Event context: Activity Study time Study time2 Social context: Alone vs not alone Constant M4: Completion time (second) Pending notification (count) Question delivery delay time (sec.) Event context: Activity Social context: Alone vs not alone Study time Study time2 User Characteristics: Procrastination Constant 0.2401*** 0.4725** 0.3439** 7.3587 0.0252*** -0.1165* 1.0325*** 1.4515*** 0.5991*** 1.6248*** -0.3105*** -13.5724*** 18.7450*** -0.4689*** -0.0012** -0.2697*** -2.521*** -0.1818*** 0.0203*** 0.0766*** 13.7792*** 0.0863*** 0.0238** 0.0237** 0.0457 0.8809*** -0.0691 0.1216*** 0.0168* 0.0530*** 0.1127*** -0.0716*** -0.1170*** 0.3596*** -0.0946*** -0.0212* -0.1784*** -2.521*** -0.0844*** 0.0357*** 0.0644*** 1.7036***
The results are reported in Table 1. Horizontally we have a path model compose to four models (M1-M4). The variable written in italic near the name of the model is the dependent variable, the variables listed below are the input variables. In both models, we report the correlation (i.e., “Coef ") between input and dependent variable, for each model in M1-M4. Out of the two models, the first is a Multilevel Structural Equation Model (ML-SEM) [ 22] , and a Structural Equation Model (SEM) [23]. Both models (MSEM, SEM) support the idea of a chain of event on answer quality. In fact, the input variables are (mostly strongly) relevant and the ift indices show that the model fits the observed data very well (RMSEA= 0.022; SRMR=0.013; chi2=95.41 (20); 2 = 0.127). Moreover, the model explains 13.0% of fluctuation in answer quality.
In M1, completion and reaction time correlate positively with “Home ⇔ phone distance" controlling by the smartphone accuracy. Their interaction has a series of negative consequence on answer quality. In fact, a longer reaction time has two efects. The first a direct efect on increase the distance between “Home ⇔ phone”. The second indirect, thru the increase of the “pending notifications" (M2), which, in turn, induce a shorter completion time which in turn decreases the error. This is not a contradictory result, but the twofold way in which an error can appear.
Our results proof that subjective annotations present a certain degree of error due to both exogenous and endogenous factors (M3, M4) afecting the quality of responses. Context history, cognitive ability, attention, efort, motivation, burden, procrastination, mood, and technical problems can play a role in terms of raising the probability of stopping the interaction with the machine, of not compliance with the interaction protocol, of a decrease in the level of attention and, consequently, they can cause a decrease of the accuracy of answers.
This paper has investigated the efects of various factors on the quality of answers, in terms of missed answers as well as their correctness. The key results of this paper are of two kinds. The first kind is that in the future the researcher’s attention should be placed on several factors related to: (a) controlling the situational and temporal context to find the best moment for administering a notification; (b) focusing on the human-machine interaction not only on the layout of the apps, but on the structure and order of the response alternatives, the ease of filling in, and finally on the support of the machine to help respond so as to reduce the response time and improve its quality. The second kind of results is related to the cognitive and more generally psychosocial traits of the respondents. It is clear that not all subjects are cooperative and follow the research protocols carefully. In the future, it will be a matter of finding what and how cognitive factors act diferently and how to extrapolate their data and replace missing data from the few and fragmented data provided. The future work will focus on how to minimize the reaction time by establishing the best moment for asking a question.
The research by Ivano is funded by the European Union’s Horizon 2020 FET Proactive project “WeNet – The Internet of us”, grant agreement No 823783. The work by Haonan receives funding from the China Scholarships Council (No.202107820038). [13] J. A. Krosnick, Response strategies for coping with the cognitive demands of attitude measures in surveys, Applied cognitive psychology 5 (1991) 213–236. [14] P. Lynn, O. Kaminska, The impact of mobile phones on survey measurement error, Public
Opinion Quarterly 77 (2013) 586–605. [15] B. Read, Respondent burden in a mobile app: Evidence from a shopping receipt scanning study, in: Survey Research Methods, volume 13, European Survey Research Association, 2019, pp. 45–71. [16] P. W. Holland, Statistics and causal inference, Journal of the American statistical Association 81 (1986) 945–960. [17] J. J. Hox, et al., Multilevel regression and multilevel structural equation modeling, The
Oxford handbook of quantitative methods 2 (2013) 281–294. [18] I. Bison, F. Giunchiglia, M. Zeni, E. Bignotti, M. Busso, R. Chenu-Abente, Trento 2018 - an extended pilot on the daily routines of university students, DataSet soon to be available at https://ri.internetofus.eu, 2021. DataScientia dataset descriptors. [19] M. Zeni, I. Zaihrayeu, F. Giunchiglia, Multi-device activity logging, in: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication., ???? [20] F. Giunchiglia, Contextual reasoning, Epistemologia, special issue: I Linguaggi e le
Macchine 16 (1993) 345–364. [21] F. Giunchiglia, E. Bignotti, M. Zeni, Personal context modelling and annotation, in: 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), IEEE, 2017, pp. 117–122. [22] S. Rabe-Hesketh, A. Skrondal, A. Pickles, Generalized multilevel structural equation modeling, Psychometrika 69 (2004) 167–190. [23] S. Rabe-Hesketh, A. Skrondal, X. Zheng, Multilevel structural equation modeling, in: Handbook of latent variable and related models, Elsevier, 2007, pp. 209–227.