Despite the eforts of specialists to provide an IoT product, due to unforeseen challenges, these environments require engagement from the end users. In the simplest case, for this personalization, they use simple recipes like if-then. Some of the most important providers of this service include IFTTT, Zapier, and Microsoft flow. Despite conducting studies with questionnaire methods to compare the usability of the systems related to these service providers, no study has yet objectively investigated their usability. The purpose of this study will be to use the fNIRS technique to assess the objective usability of the mobile systems of these service providers.
The Internet of Things (IoT) connects the digital and physical worlds, making the world around
us smarter and more responsive [
In general, due to the number of objects connected to each other on the one hand and the
variety of people using the systems and their expectations on the other hand, it is not possible to
consider all the problems and challenges of the end users when introducing a new product [
However, end users can influence IoT systems that depend on them through simple actions
and commands. The entry of the end user into the cycle of providing simple commands to IoT
systems is also called experience personalization [
Despite the numerous advantages of IoT-based technologies, the fundamental problem
remains to tailor their behavior to meet the highly contextualized, unique, and frequently changing
demands of users. One of the methods proposed in the literature to overcome this problem is
simple if-then commands. In this case, when a trigger occurs, a specific action that has been
given to the system occurs [
These commands, which are called Trigger-Action rules, have been investigated in diferent
contexts such as personalizing humanoid robot behaviors and smart homes [
The usability of these software products is critical. The cognitive approach is a method of
evaluating the usability of systems that focuses on understanding users’ mental processes while
using the system. This approach is based on the concept that users interact with the system
through their cognitive processes, such as perception, attention, memory, and reasoning, and
that understanding these processes can help design a more usable system [
To use the cognitive approach for assessing the usability of a system, one needs to identify the users’ goals and the activities performed when using the system. Next, the cognitive processes involved in these activities must be analyzed to determine whether the system supports or hinders these processes.
There are several tools and techniques used in the cognitive approach to assess the usability of systems. In this work, we propose the use of fNIRS (functional near infrared spectroscopy) for the objective evaluation of User eXperience and usability of these services.
The providers IFTTT, Zapier, and Microsoft Flow have been compared from diferent points of view [12][13]. In recent years, the User eXperience and usability of these services have gained attention [14]. Schrepp et al. surveyed the usability of three web automation service providers with the participation of 82 users. They did a real scenario with these products. The results showed that IFTTT had the highest scores in all 6 dimensions of the User Experience Questionnaire (UEQ) [15]. Most studies use subjective methods to analyze the usability of these IoT-based products.
Objective analysis of a product’s usability is a scientific field developed in the last 20 years. In the early stages of using neuroscience devices to study User eXperience and usability, we can see the importance of electroencephalogram (EEG) [16][17]. With the advent of new devices such as fNIRS, the use of this new neurophysiological approach to study user experience and usability is proposed [18][19].
In a recent study which integrates Augmented Reality into IoT, researchers use both subjective and objective measures to study the usability. Following a neuroscientific approach, they used an fNIRS device. The scenario was about two information search modes, an AR-based, and web-based. The results showed that information search using AR was more eficient and had a lower cognitive load than information search with a website. The findings of the usability testing show that AR, as an emerging retail technology, may significantly enhance consumer experiences and raise buy intent [20].
In a recent study, authors tried to capture objective data about the User eXperience of gamers and non-gamers based on fNIRS [21]. The aim of this study was about quantifying users’ cognitive parameters about the game dificulty level. Three brain regions that have the responsibility for perceptual UX are selected for data gathering. 40 university students participated in this study and a multi-channel near-infrared system was utilized to record the brain activity. The 32-channel fNIRS device had a sampling frequency of 10 Hz and its wavelengths are 760 and 850 nm (21 light sources and 20 detectors). The authors concluded that a combination of the flow state scale (subjective questionnaire) and fNIRS is a feasible approach to assess players’ UX.
Considering the importance of objective studies to survey the usability of interactive products and the lack of a published study on this matter in these IoT-based services, we propose an objective technique to study the usability of these services. Based on the results obtained from this study, we can have a better judgment of end users’ relationship with these technologies.
In this section, we describe the experimental design of the study.
Forty participants (M=20, F=20) will be considered. They will be homogeneously selected based on several confounding parameters such as education level, programming experience and eye health status. The reason for this homogeneous selection is that these confounding variables do not distort (bias) participants’ perception of the final variable, so the usability of the products. People participating in the study will fill out the informed consent form along with the initial explanation.
For comparing the usability of these three software products, fNIRS will be used. The use of this technique in usability studies does not have much history and only a few articles have been published in recent years [22][23].
By using the fNIRS device we can observe the hemodynamic response, which refers to changes in blood flow and oxygenation levels in response to neural activity. During cognitive processes, such as problem-solving, attention, or memory tasks, there is an increased demand for oxygenated blood in specific regions of the brain. fNIRS can detect these changes by measuring the concentrations of oxygenated and deoxygenated hemoglobin. Typically, an increase in oxygenated hemoglobin and a decrease in deoxygenated hemoglobin are observed in brain areas involved in the cognitive task. Using an fNIRS device, during a cognitive task, an increase in oxygenated hemoglobin and a decrease in deoxygenated hemoglobin would indicate increased neural activity and oxygen consumption in the corresponding brain regions. Therefore, during software use, we expect to observe changes in the levels of oxygenated and deoxygenated hemoglobin, and we want to quantify such changes in terms of the proportion of deoxyhemoglobin/oxyhemoglobin. This technique will be used to conduct a comparative experimental study on the use of the three software products IFTTT, Zapier, and Microsoft Flow.
The device used in this study will have 64 channels. To collect data, the 10-20 international system technique will be used [24]. Considering that usability/UX perception’s region of interest (ROI) is related to the frontal pole area (FPA), dorsolateral prefrontal cortex (DLPFC), and ventral lateral prefrontal cortex (VLPFC), additional data will be removed during preprocessing [22]. Through changes in the amount of hemoglobin without oxygen compared to hemoglobin with oxygen, the perception of participants with respect to those three software products will be calculated. 3.3. Tasks Due to the importance of some crucial aspects in objective usability such as eficiency, efectiveness, and learnability, we develop a scenario that can yield the results of these aspects. Our scenario is based on personalizing a simple IoT system. In our scenario, the installed version of three applications related to the three service providers is presented to participants. They will encounter a simple tangible IoT system. This system has an intelligent light, a Twitter/Facebook account, an intelligent voice command device (Google Nest), and the software which is mentioned before.
The participants should do some simple tasks about opening the software and provide some simple rules in the software to connect the light to the Twitter and Facebook accounts. Then they should command by voice to the IoT system and do some predefined tasks.
First user will be informed about the required procedure. Then they will use the system based on a predefined scenario. The scenario is constant in all participants. For each software, they have the first 5 minutes for familiarization. Then they will use each software for 15 minutes.
All of these applications will be installed from Google Play. Also, all of the participants will use the software in a Google Pixel 7 pro.
During the total procedures of using the software and the IoT system, they will be brain scanned by fNIRS.
People will have enough rest between using diferent software. At the end of using each software, System Usability Scale (SUS) and User Experience Questionnaire (UEQ) questionnaires will be filled for each one.
In this work, we have focused on the importance of a cognitive approach to objective usability assessment. With this methodology, we expect (i) that the understanding and evaluation of the ease of use, eficiency, and satisfaction of using the software may vary based on gender. (ii) A positive and significant correlation will be observed between the results of subjective usability (questionnaires) and the results of cognitive parameters (based on brain activity), that is, the results obtained from subjective measures of usability and objective measures of cognitive parameters, such as brain activity, are related to each other. For example, if a software program is perceived as easy to use by the user, this may be reflected in increased activity in certain areas of the brain associated with attention, memory, and problem-solving; (iii) a significant diference between the usability of these software products will be observed. Some software may be more user-friendly, eficient, and efective than others. This diference in usability may arise due to factors such as the design of the software, the features and functionalities ofered, and the user’s prior experience and knowledge of the software. Compared with the other devices for physiological and behavioral measures, such as fMRI and EEG, that can also provide accurate and ”objective” assessments, fNIRS can be more efectively used in real human-technology interaction scenario. Indeed, the fNIRS is more portable (in comparison to fMRI) and less sensitive to any excessive noise (in comparison to EEG). platform, Human Factors: The Journal of the Human Factors and Ergonomics Society 63 (2020) 001872082092042. doi:1 0 . 1 1 7 7 / 0 0 1 8 7 2 0 8 2 0 9 2 0 4 2 9 . [11] A. Thorpe, K. Nesbitt, A. Eidels, Assessing game interface workload and usability: A cognitive science perspective, in: Proceedings of the Australasian Computer Science Week Multiconference, ACSW ’19, Association for Computing Machinery, New York, NY, USA, 2019. doi:1 0 . 1 1 4 5 / 3 2 9 0 6 8 8 . 3 2 9 0 7 4 9 . [12] A. Rahmati, E. Fernandes, J. Jung, A. Prakash, Ifttt vs. zapier: A comparative study of trigger-action programming frameworks (2017). [13] B. Cosar, A comparative analysis of zapier and ifttt through the lens of enterprise integration (2021) 54. [14] K. Ito, H. Kogami, S. Kang, R. Yoshizaki, Y. Sakurai, K. Nakano, D. Yoshioka, J. Wu, M. Fujisaki-Sueda-Sakai, I. Sugarawa, T. Miura, K. Yabu, T. Ifukube, Configuring interface for iot: Natural user interface study on senior adults in japan, in: LifeTech 2021 - 2021 IEEE 3rd Global Conference on Life Sciences and Technologies, Institute of Electrical and Electronics Engineers Inc., 2021, pp. 200–203. doi:1 0 . 1 1 0 9 / L i f e T e c h 5 2 1 1 1 . 2 0 2 1 . 9 3 9 1 9 4 3 . [15] M. Schrepp, A. Hinderks, J. Thomaschewski, Applying the user experience questionnaire (ueq) in diferent evaluation scenarios, 2014, pp. 383–392. doi: 1 0 . 1 0 0 7 / 9 7 8 - 3 - 3 1 9 - 0 7 6 6 8 - 3 _ 3 7 . [16] J. Chai, Y. Ge, Y. Liu, W. Li, L. Zhou, L. Yao, X. Sun, Application of frontal eeg asymmetry to user experience research, in: 11th International Conference on Engineering Psychology and Cognitive Ergonomics - Volume 8532, Springer-Verlag, Berlin, Heidelberg, 2014, p. 234–243. doi:1 0 . 1 0 0 7 / 9 7 8 - 3 - 3 1 9 - 0 7 5 1 5 - 0 _ 2 4 . [17] H. Masaki, M. Ohira, H. Uwano, K.-i. Matsumoto, A quantitative evaluation on the software use experience with electroencephalogram, in: A. Marcus (Ed.), Design, User Experience, and Usability. Theory, Methods, Tools and Practice, Springer Berlin Heidelberg, Berlin, Heidelberg, 2011, pp. 469–477. [18] K. Pollmann, M. Vukelić, N. Birbaumer, M. Peissner, W. Bauer, S. Kim, fnirs as a method to capture the emotional user experience: A feasibility study, in: M. Kurosu (Ed.), HumanComputer Interaction. Novel User Experiences, Springer International Publishing, Cham, 2016, pp. 37–47. [19] K. Lukanov, H. A. Maior, M. L. Wilson, Using fnirs in usability testing: Understanding the efect of web form layout on mental workload, in: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, Association for Computing Machinery, New York, NY, USA, 2016, p. 4011–4016. doi:1 0 . 1 1 4 5 / 2 8 5 8 0 3 6 . 2 8 5 8 2 3 6 . [20] J. Wu, D. Zhang, T. Liu, H. Yang, Y. Wang, H. Yao, S. Zhao, Usability evaluation of augmented reality: A neuro- information-systems study, Journal of Visualized Experiments (2022) e64667. doi:1 0 . 3 7 9 1 / 6 4 6 6 7 . [21] D. Yu, S. Wang, F. Song, Y. Liu, S. Zhang, Y. Wang, X. Xie, Z. Zhang, Research on user experience of the video game dificulty based on flow theory and fnirs, Behaviour & Information Technology 42 (2023) 789–805. doi:1 0 . 1 0 8 0 / 0 1 4 4 9 2 9 X . 2 0 2 2 . 2 0 4 3 4 4 2 . [22] J. Wu, D. Zhang, T. Liu, H. Yang, Y. Wang, H. Yao, S. Zhao, Usability evaluation of augmented reality: A neuro- information-systems study, Journal of Visualized Experiments (2022) e64667. doi:1 0 . 3 7 9 1 / 6 4 6 6 7 . [23] Y. N. Ortega-Gijon, C. Mezura-Godoy, Usability evaluation of brain computer interfaces: Analysis of methods and tools, in: 2019 IEEE International Conference on Engineering Veracruz (ICEV), volume I, 2019, pp. 1–8. doi:1 0 . 1 1 0 9 / I C E V . 2 0 1 9 . 8 9 2 0 4 0 9 . [24] R. Katmah, F. Al-Shargie, U. Tariq, F. Babiloni, F. Al-Mughairbi, H. Al-Nashash, Mental stress analysis using the power spectrum of fnirs signals, in: 2022 International Conference on Electrical and Computing Technologies and Applications (ICECTA), 2022, pp. 216–219. doi:1 0 . 1 1 0 9 / I C E C T A 5 7 1 4 8 . 2 0 2 2 . 9 9 9 0 2 3 7 .