Human Computer Interaction Aspects of Low-Power Wide Area Networks for Wearable Applications Charalampos Orfanidis Martin Jacobsson Xenofon Fafoutis chaorf@dtu.dk martin.jacobsson@sth.kth.se xefa@dtu.dk Technical University of Denmark KTH Royal Institute of Technology Technical University of Denmark Copenhagen, Denmark Stockholm, Sweden Copenhagen, Denmark ABSTRACT the characteristics of LPWAN. Consequently, these applications The advent of Low-Power Wide Area Networks has enabled sig- became very popular since the application domain is not that broad. nificant developments of the IoT ecosystem. Long range commu- After LPWANs were established in IoT, almost bound with a cer- nication using low power is now feasible and offers connectivity tain set of applications (e.g. smart cities), researchers and industry to remote areas where cellular network is not available. Therefore, representatives started using it for other scenarios as well, such as new application scenarios have emerged, such as smart cities, smart wearable systems [24] or activity recognition [8]. As mentioned metering and more, which are attracting a lot of attention from above, the offered data rates are low and cannot meet the require- both research and industry. Beside the aforementioned popular ments of several applications. Nevertheless, certain applications scenarios, Low-Power Wide Area Networks have started to be used which do not require high data traffic (e.g. elderly monitoring [27]), in wearable systems scenarios as well. In this position paper, we could take advantage of the long range communication feature. In pose some questions regarding the Human Computer Interaction smart cities and other environments where LPWAN gateways are aspects of Low-Power Wide Area Networks which will help them abundant, a wearable system can be used as a standalone device integrate in Ubiquitous Computing applications. We illustrate by since there is coverage to a large urban environment. For instance, a wearable system, which is based on an foot gesture interface, a an elderly fall monitoring application is not depended on a smart- Low-Power Wide Area Network, and an Neural Network classifier. phone, a short range gateway or a GSM modem. In these cases, The discussion is based on the state of art of foot interfaces and using LPWAN may improve the user experience. highlights open issues and challenges. In this paper we raise some questions about LPWAN from a Human Computer Interaction (HCI) perspective. The increased CCS CONCEPTS time-on-air due to low data rates, the larger interaction range en- abled by the longer range communication coverage and other HCI • Computer systems organization → Embedded and cyber- aspects will be affected because of the nature of LPWAN. This physical systems; • Human-centered computing → Gestural would result in a different user experience of wearable systems and input; Human computer interaction (HCI). other human-centered applications which needs to be examined KEYWORDS properly. To this end, we present a wearable which combines a foot gesture interface, an LPWAN and a Neural Network (NN) classi- LPWAN; HCI; Foot gesture; Wearable systems; IoT fier to contextualize the discussion and illustrate some challenges around foot interfaces. We argue that similar questions would arise 1 INTRODUCTION in scenarios with other interfaces where LPWAN is used. The application scenario domain of the Internet of Things (IoT) This paper is organized as follows. First, we introduce the wear- has been enriched since the arrival of Low Power Wide Area Net- able prototype and the application scenario which it is designed works (LPWANs). LPWAN enables battery powered long range for. In the next section, a discussion follows about the HCI charac- communication [14], with a lifetime 2 to 4 years depending on the teristics of LPWAN focusing on foot interfaces. Then, the related configuration parameters [10]. Another characteristic that make work section presents the state of the art in foot interfaces and we LPWAN an attractive option for IoT is the fact that it is robust and compare the differences with the wearable we introduced and we able to tolerate high level of interference [3, 15]. However, LPWANs conclude in final the section. are only able to achieve low data rates (kilobits per second) which might be a limiting factor in some applications. Channel utilization 2 A LONG RANGE EMERGENCY SYSTEM regulations are also applied to avoid overcrowded environments. Hence, there is a specific amount of applications which can use this This section provides an overview of the application scenario we wireless technology, such as smart cities [12], smart agriculture [7], focus on and the motivation behind it. Moreover, it introduces a smart metering [2], and other applications whose requirements fit brief technical description of the prototype to shape the context in which the discussion takes place afterwards. Copyright 2021 for this paper by its authors. Use permitted under Creative Commons The scenario that we focus on is the following: a user is doing License Attribution 4.0 International (CC BY 4.0). an outdoor activity (i.e. walking, jogging) and feels threatened by a possible perpetrator. In that case the user is willing to broadcast a message asking for help, but at the same time this action has to be discreet and not being noticed from the possible perpetrator. Hence, we design a long range emergency system, which includes a foot CHIIoT 1, February 17, 2021, Delft, The Netherlands Orfanidis, et al. interface for capturing a foot gesture in case of emergency, a NN 3 LONG RANGE COMMUNICATION IN FOOT classifier to distinguish the gestures from other activities and an INTERFACES LPWAN to transmit the message. It is very important to distinguish Foot interfaces were investigated from the early start of HCI estab- accurately the gestures from activities because the user might be in lishment. The decreasing size of electronics and cost have made this danger and the foot interface is based on force sensors below the input modality more attractive. The combination of foot interface shoe sole, which are giving very raw data. Thus, the presence of with a low-power long range communication is bringing some new a classifier that enhances the accuracy is essential. An advantage characteristics which might result in a different user experience. In of using LPWAN, is that it offers long range coverage (in smart this section we try to outline how the new features from an LPWAN cities environments), and therefore the user does not need to carry might affect the user experience of a foot interface. a smartphone or being dependent on any short range gateway, One the factors which is different when LPWAN is used in foot which is the case for many wearables designed for outdoor or sport interfaces is the Interaction Range compared with interfaces using activities. Bluetooth Low Energy (BLE), IEEE 802.15.4 or shorter range wire- During the design, we selected low cost consumer electronics less communication technologies. In that case a foot interface can which can operate on batteries in order to be suitable for the IoT be used within the coverage of smart environment where several ecosystem. The wearable system we propose can be divided in two LPWAN gateways are available. We speculate that the increased parts, the hardware prototype and the NN classifier used to identify interaction range will improve user experience as the user will feel gestures. less dependent on smartphone or a desktop computer. But at the To realize the prototype, we use a normal shoe and we deploy moment it is unclear if and how much the increased range factor two force sensors below the shoe sole. One is deployed at the toe tip can affect the user experience of a foot interface. and the other at the heel as depicted in Figure 1. The force sensors Another factor that may affect the user experience, and is related are connected to the LPWAN Microcontroller Unit (MCU), which to the interaction range, is the Interaction With Other Devices. Foot is glued to the side of the shoe with a small power-bank. The force interfaces traditionally interact with other devices such as mobile sensors have a surface of 38 mm × 38 mm in square shape. The devices [25], desktop computers [21], public displays [19] and oth- LPWAN device consists of an ESP32 MCU and a RFM95 LoRa [20] ers. In the case where a foot interface interacts directly with a cloud modem. service, like our application scenario, the output is also taking place at the wearable. Therefore, there are certain HCI aspects about the output that are required to be explored. The feedback or output in foot interfaces can be classified in visual, auditory, haptic and thermal and has been investigated through several applications [22]. An application using an LPWAN may have a delay to the output due to the long time-on-air values imposed from the physical char- acteristics of the LPWAN technology. The user experience might be affected from this drawback and thus it should be investigated further. First how a delay in the output may degrade the user expe- rience in this context. Second, if there is any way to overcome this drawback. For instance, the time-on-air on some LPWANs varies a lot depending the configuration parameters. Which are the optimal (a) (b) (c) parameters to have tolerant delay in the context of the focused scenario? Figure 1: The prototype was based on a regular shoe includ- If we focus on the long range emergency scenario we introduced, ing two force sensors, one at the the tip of the shoe (a), one where discreteness is a crucial requirement and assume that the at the heel (b), and both are connected to an IoT node (c). outputs will take place on the wearable, the discreetness of the outputs should be evaluated as well. Fukahori et al. [6] introduced a foot interface for foot plantar-based inputs with force sensors Mobile and embedded devices have benefited a lot from the attached on socks and evaluated if the foot gestures are observable development of NN during the last years [9]. The NN developments in a public space. A similar evaluation should be carried out to have lead to to scientific breakthroughs and has shaped the norm investigate how discreet are the available outputs. Obviously some in pattern recognition and other features offered from NNs in IoT types of outputs are less discreet by default (auditory, visual) be- and activity monitoring. In our approach, given the constrained cause they are directly observable, but some others, like vibrations, resources of the MCU, we use an NN classifier with two hidden make more sense to be evaluated. Moreover, if the context is more layers that operates by forwarding information in one direction broad and we just consider a foot interface where the long range through each layer in the network. The selection of the Machine communication allows it to operate as a standalone device and the Learning (ML) model was made after considering other models outputs take place on the interface, all the available outputs should and evaluating the trade-off between accuracy and implementation be evaluated because the user experience might be affected. complexity to fit on an ESP32 MCU, since the model is implemented on board. A more detailed description about the implementation of the prototype and its performance is described in [16]. Human Computer Interaction aspects of Low-Power Wide Area Networks for Wearable Applications CHIIoT 1, February 17, 2021, Delft, The Netherlands 3.1 Challenges on the top of a shoe, pointing to the wearer and a single board One of the issues that originates from the long range communi- computer. The rationale behind this approach is to capture a set of cation characteristic of LPWAN is to conduct a proper evaluation novel hand-gestures which can be associated with several scenarios involving real experiments, because of the long distances and lack like answering the phone, activate silence mode to the smartphone of testbeds. A proper evaluation of the wearable we propose would and many others. be to distribute it to a number of individuals and use it for a long- Fan et al. in [4] study how often people want to use a foot gesture term period. Furthermore stabilizing the background variables in interface when both their hands are occupied and what kind of an urban environment is rather difficult and using an alternative smartphone related tasks they would like to perform. Afterwards environment (e.g. lab location) is not capturing the real scenarios they develop footsketch, a foot gesture recognition app for smart- we might desire for some cases. phones. Footsketch uses acceloremeter data and a Dynamic Tree One more issue is that even though there are several gateways in Warping algorithm to distinguish different foot gestures. After at- urban environments there is still the chance of going out of range taching the smartphone on the leg to evaluate the performance they and lose connectivity. Most of the times researchers set their own found that for some cases, one can save over 70% of the time over a gateways to perform experiments but that can be time demanding gesture compare with a traditional touch gesture on a smartphone and also restricted in terms of coverage area. display. Felberbaum et al. in [5] present a study to analyze and A problem which has to do with the methodology of HCI is elicit users’ perception of foot gestures when they are taking place that several times it is included video footage of the user using the on a horizontal surface. The authors examine three different user interface to obtain timings or assess other characteristics of user conditions: standing in front of a display, sitting down in front of a experience. Following such a method when using long range com- desktop display and standing on a projected surface. Furthermore, munication is more challenging due to the fact that the evaluation a metric is introduced to quantify how a gesture is preferable to an might take place in-the-wild where footage infrastructure is not action. Maragliulo et al. in [13], develop a foot gesture recognition possible to be installed or it might be illegal. system based on two electromyography (EMG) sensors, deployed at the lower knee. The system in combination with an SVM is able to identify a certain number of trained foot gestures. The interface 4 RELATED WORK is evaluated through use cases aiming at playing musical instru- This section covers other approaches with foot gesture interfaces ments which require equipment when the hands are occupied. A used in various applications. Unfortunately we were not able to find foot interface to induce a certain walking cycle is presented in [23]. any approach using LPWAN to carry out a comparison with the The authors target navigation scenarios where the user might not system we introduce. Therefore, we try to focus on the Interaction consider the environmental circumstances and develop a proto- Range and the Interaction With Other Devices parameters of the type which obtains the walking cycle through pressure sensors mentioned approaches. and vibration motors to influence the specific walking cycle. An There are several attempts to investigate research questions approach which is one of the closest to the one which is presented around foot gesture interfaces. For instance, one of the first attempts in this paper is presented in [6], where Fukahori et al. design a to design a foot interface is described in [17], where a set of tiles foot interface based on a sock with force sensors. The interface is with force sensors can be combined in different shapes on the recognizing a set of subtle gestures with the support of a ML model. floor. The main applications of interest is music and dance control, The differences with our approach is that the interaction range is medicine and sports science but also control in computer games. shorter due to used the wireless technology (IEEE 802.15.4) and the The communication protocols for sensor networks were not very ML model runs on the host computer and not on board. advanced at the time and the authors use a wired protocol where All the aforementioned approaches have an interaction range the tiles communicate with each other until they rich a sink node below 300 meter approximately and the main devices which in- which is connected to a computer. Footsee [26] is a foot interface teract is a mobile device or a desktop computer. The long range based on a sensor pad to be used as control for video games. The communication offered by an LPWAN is able to deliver a different sensor pad consists of a grid of 160 by 64 pressure sensors and it is user experience and affect the aforementioned parameters. able to depict full body motions after an offline training process. A multimodal hand and foot gesture interface for handheld de- vices is presented in [11]. The interface is evaluated through a 5 CONCLUSION football game on a smartphone where the user is controlling an In this paper, we presented a position paper where we argue for augmented ball with foot and hand gestures on the smartphone dis- a further investigation of the HCI aspects of LPWAN. The latter play. The results show that a multimodal game is more interesting have been very popular in application scenarios like smart cities and fun than a monomodal one which was used in the evaluation. but when they are used in more human-centered applications there Another approach is demonstrated in [18], where hand and foot are still several questions to be answered. Therefore, we introduce a gestures are combined to be utilized in multiple tasks on tabletop foot gesture interface implemented in a regular shoe with low-cost systems. The authors identify which foot gestures can be combined consumer electronics supported by a NN classifier and an LPWAN. with hand gestures compared to the combined interface with single We focus on a scenario where a user is doing an outdoor activity hand gesture and found that they require the same time while the and feels threatened so she/he uses the foot interface to send an combined one could speed up multitasking for some cases. Shoe- emergency message for help in a discreet manner. We highlight a Sense [1] is a wearable which consists of a depth camera attached set of questions and challenges which will assist to explore the user CHIIoT 1, February 17, 2021, Delft, The Netherlands Orfanidis, et al. experience further when an LPWAN is present on an interface like [18] Nuttapol Sangsuriyachot and Masanori Sugimoto. 2012. Novel Interaction Tech- the foot gesture interface we present. niques Based on a Combination of Hand and Foot Gestures in Tabletop Envi- ronments. In Proceedings of the 10th Asia Pacific Conference on Computer Human Interaction (Matsue-city, Shimane, Japan) (APCHI ’12). Association for Computing Machinery, New York, NY, USA, 21–28. https://doi.org/10.1145/2350046.2350053 [19] William Saunders and Daniel Vogel. 2015. The Performance of Indirect Foot Pointing Using Discrete Taps and Kicks While Standing. In Proceedings of the 41st REFERENCES Graphics Interface Conference (Halifax, Nova Scotia, Canada) (GI ’15). Canadian [1] Gilles Bailly, Jörg Müller, Michael Rohs, Daniel Wigdor, and Sven Kratz. 2012. Information Processing Society, CAN, 265–272. ShoeSense: A New Perspective on Gestural Interaction and Wearable Applications. [20] Semtech. 2020. What is LoRa®? https://www.semtech.com/lora/what-is-lora. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Accessed: 2020-12-21. (Austin, Texas, USA) (CHI ’12). Association for Computing Machinery, New York, [21] Eduardo Velloso, Jason Alexander, Andreas Bulling, and Hans Gellersen. 2015. NY, USA, 1239–1248. https://doi.org/10.1145/2207676.2208576 Interactions Under the Desk: A Characterisation of Foot Movements for Input [2] D. F. Carvalho, A. Depari, P. Ferrari, A. Flammini, S. Rinaldi, and E. Sisinni. 2018. in a Seated Position. In Human-Computer Interaction – INTERACT 2015, Julio On the feasibility of mobile sensing and tracking applications based on LPWAN. Abascal, Simone Barbosa, Mirko Fetter, Tom Gross, Philippe Palanque, and Marco In 2018 IEEE Sensors Applications Symposium (SAS). 1–6. https://doi.org/10.1109/ Winckler (Eds.). Springer International Publishing, Cham, 384–401. SAS.2018.8336765 [22] Eduardo Velloso, Dominik Schmidt, Jason Alexander, Hans Gellersen, and An- [3] Rashad Eletreby, Diana Zhang, Swarun Kumar, and Osman Yağan. 2017. Em- dreas Bulling. 2015. The Feet in Human–Computer Interaction: A Survey of powering Low-Power Wide Area Networks in Urban Settings. In Proceedings of Foot-Based Interaction. ACM Comput. Surv. 48, 2, Article 21 (Sept. 2015), 35 pages. the Conference of the ACM Special Interest Group on Data Communication (Los https://doi.org/10.1145/2816455 Angeles, CA, USA) (SIGCOMM ’17). Association for Computing Machinery, New [23] Junji Watanabe, Hideyuki Ando, and Taro Maeda. 2005. Shoe-Shaped Interface York, NY, USA, 309–321. https://doi.org/10.1145/3098822.3098845 for Inducing a Walking Cycle. In Proceedings of the 2005 International Conference [4] Mingming Fan, Yizheng Ding, Fang Shen, Yuhui You, and Zhi Yu. 2017. An on Augmented Tele-Existence (Christchurch, New Zealand) (ICAT ’05). Association Empirical Study of Foot Gestures for Hands-Occupied Mobile Interaction. In for Computing Machinery, New York, NY, USA, 30–34. https://doi.org/10.1145/ Proceedings of the 2017 ACM International Symposium on Wearable Computers 1152399.1152406 (Maui, Hawaii) (ISWC ’17). Association for Computing Machinery, New York, NY, [24] F. Wu, C. Rüdiger, J. Redouté, and M. R. Yuce. 2018. WE-Safe: A wearable IoT USA, 172–173. https://doi.org/10.1145/3123021.3123043 sensor node for safety applications via LoRa. In 2018 IEEE 4th World Forum [5] Yasmin Felberbaum and Joel Lanir. 2018. Better Understanding of Foot Gestures: on Internet of Things (WF-IoT). 144–148. https://doi.org/10.1109/WF-IoT.2018. An Elicitation Study. In Proceedings of the 2018 CHI Conference on Human Factors in 8355234 Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing [25] T. Yamamoto, M. Tsukamoto, and T. Yoshihisa. 2008. Foot-Step Input Method for Machinery, New York, NY, USA, Article 334, 12 pages. https://doi.org/10.1145/ Operating Information Devices While Jogging. In 2008 International Symposium 3173574.3173908 on Applications and the Internet. 173–176. https://doi.org/10.1109/SAINT.2008.97 [6] Koumei Fukahori, Daisuke Sakamoto, and Takeo Igarashi. 2015. Exploring Subtle [26] KangKang Yin and Dinesh K. Pai. 2003. FootSee: An Interactive Animation Foot Plantar-Based Gestures with Sock-Placed Pressure Sensors. In Proceedings of System. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Computer Animation (San Diego, California) (SCA ’03). Eurographics Association, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, Goslar, DEU, 329–338. NY, USA, 3019–3028. https://doi.org/10.1145/2702123.2702308 [27] Jia-Xin Yue, Zhong Wang, Qi Yang, and Xiao-Bin Zheng. 2018. Design and [7] S. Heble, A. Kumar, K. V. V. D. Prasad, S. Samirana, P. Rajalakshmi, and U. B. Implementation of Intelligent Monitoring System for the Elderly on Android Desai. 2018. A low power IoT network for smart agriculture. In 2018 IEEE 4th Phone. In Proceedings of the International Conference on Information Technology World Forum on Internet of Things (WF-IoT). 609–614. https://doi.org/10.1109/WF- and Electrical Engineering 2018 (Xiamen, Fujian, China) (ICITEE ’18). Association IoT.2018.8355152 for Computing Machinery, New York, NY, USA, Article 63, 8 pages. https: [8] Tahera Hossain, Yusuke Doi, Tahia Tazin, Md Atiqur Rahman Ahad, and Sozo //doi.org/10.1145/3148453.3306302 Inoue. 2018. Study of LoRaWAN Technology for Activity Recognition. In Pro- ceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers (Singapore, Singapore) (UbiComp ’18). Association for Computing Machinery, New York, NY, USA, 1449–1453. https://doi.org/10.1145/3267305.3267510 [9] N. D. Lane and P. Warden. 2018. The Deep (Learning) Transformation of Mobile and Embedded Computing. Computer 51, 5 (2018), 12–16. [10] Jansen C. Liando, Amalinda Gamage, Agustinus W. Tengourtius, and Mo Li. 2019. Known and Unknown Facts of LoRa: Experiences from a Large-Scale Measurement Study. ACM Trans. Sen. Netw. 15, 2, Article 16 (Feb. 2019), 35 pages. [11] Zhihan Lv, Alaa Halawani, Shengzhong Feng, Haibo Li, and Shafiq Ur Réhman. 2014. Multimodal Hand and Foot Gesture Interaction for Handheld Devices. ACM Trans. Multimedia Comput. Commun. Appl. 11, 1s, Article 10 (Oct. 2014), 19 pages. https://doi.org/10.1145/2645860 [12] D. Magrin, M. Centenaro, and L. Vangelista. 2017. Performance evaluation of LoRa networks in a smart city scenario. In 2017 IEEE International Conference on Communications (ICC). 1–7. https://doi.org/10.1109/ICC.2017.7996384 [13] S. Maragliulo, P. F. A. Lopes, L. B. Osório, A. T. De Almeida, and M. Tavakoli. 2019. Foot Gesture Recognition Through Dual Channel Wearable EMG System. IEEE Sensors Journal 19, 22 (2019), 10187–10197. [14] Kais Mekki, Eddy Bajic, Frederic Chaxel, and Fernand Meyer. 2019. A comparative study of LPWAN technologies for large-scale IoT deployment. ICT Express 5, 1 (2019), 1 – 7. https://doi.org/10.1016/j.icte.2017.12.005 [15] C. Orfanidis, L. M. Feeney, M. Jacobsson, and P. Gunningberg. 2017. Investi- gating interference between LoRa and IEEE 802.15.4g networks. In 2017 IEEE 13th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob). 1–8. https://doi.org/10.1109/WiMOB.2017.8115772 [16] C. Orfanidis, R.B.H. Hassen, A. Kwiek, X. Fafoutis, and M.Jacobsson. 2021. A Discreet Wearable Long-Range Emergency System Based on Embedded Machine Learning. In 2021 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). [17] Bruce Richardson, Krispin Leydon, Mikael Fernstrom, and Joseph A. Paradiso. 2004. Z-Tiles: Building Blocks for Modular, Pressure-Sensing Floorspaces. In CHI ’04 Extended Abstracts on Human Factors in Computing Systems (Vienna, Austria) (CHI EA ’04). Association for Computing Machinery, New York, NY, USA, 1529–1532. https://doi.org/10.1145/985921.986107