Human Computer Interaction Aspects of Low-Power Wide Area Networks for Wearable Applications

Abstract

The advent of Low-Power Wide Area Networks has enabled significant developments of the IoT ecosystem. Long range communication using low power is now feasible and ofers connectivity to remote areas where cellular network is not available. Therefore, new application scenarios have emerged, such as smart cities, smart metering and more, which are attracting a lot of attention from both research and industry. Beside the aforementioned popular scenarios, Low-Power Wide Area Networks have started to be used in wearable systems scenarios as well. In this position paper, we pose some questions regarding the Human Computer Interaction aspects of Low-Power Wide Area Networks which will help them integrate in Ubiquitous Computing applications. We illustrate by a wearable system, which is based on an foot gesture interface, a Low-Power Wide Area Network, and an Neural Network classifier. The discussion is based on the state of art of foot interfaces and highlights open issues and challenges.

CCS CONCEPTS

• Computer systems organization → Embedded and cyberphysical systems; • Human-centered computing → Gestural input; Human computer interaction (HCI).

INTRODUCTION

The application scenario domain of the Internet of Things (IoT) has been enriched since the arrival of Low Power Wide Area Networks (LPWANs). LPWAN enables battery powered long range communication [ 14 ], with a lifetime 2 to 4 years depending on the configuration parameters [ 10 ]. Another characteristic that make LPWAN an attractive option for IoT is the fact that it is robust and able to tolerate high level of interference [ 3, 15 ]. However, LPWANs are only able to achieve low data rates (kilobits per second) which might be a limiting factor in some applications. Channel utilization regulations are also applied to avoid overcrowded environments. Hence, there is a specific amount of applications which can use this wireless technology, such as smart cities [ 12 ], smart agriculture [ 7 ], smart metering [ 2 ], and other applications whose requirements fit Copyright 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). the characteristics of LPWAN. Consequently, these applications became very popular since the application domain is not that broad.

After LPWANs were established in IoT, almost bound with a certain set of applications (e.g. smart cities), researchers and industry representatives started using it for other scenarios as well, such as wearable systems [ 24 ] or activity recognition [ 8 ]. As mentioned above, the ofered data rates are low and cannot meet the requirements of several applications. Nevertheless, certain applications which do not require high data trafic (e.g. elderly monitoring [ 27 ]), could take advantage of the long range communication feature. In smart cities and other environments where LPWAN gateways are abundant, a wearable system can be used as a standalone device since there is coverage to a large urban environment. For instance, an elderly fall monitoring application is not depended on a smartphone, a short range gateway or a GSM modem. In these cases, using LPWAN may improve the user experience.

In this paper we raise some questions about LPWAN from a Human Computer Interaction (HCI) perspective. The increased time-on-air due to low data rates, the larger interaction range enabled by the longer range communication coverage and other HCI aspects will be afected because of the nature of LPWAN. This would result in a diferent user experience of wearable systems and other human-centered applications which needs to be examined properly. To this end, we present a wearable which combines a foot gesture interface, an LPWAN and a Neural Network (NN) classiifer to contextualize the discussion and illustrate some challenges around foot interfaces. We argue that similar questions would arise in scenarios with other interfaces where LPWAN is used.

This paper is organized as follows. First, we introduce the wearable prototype and the application scenario which it is designed for. In the next section, a discussion follows about the HCI characteristics of LPWAN focusing on foot interfaces. Then, the related work section presents the state of the art in foot interfaces and we compare the diferences with the wearable we introduced and we conclude in final the section. 2

A LONG RANGE EMERGENCY SYSTEM

This section provides an overview of the application scenario we focus on and the motivation behind it. Moreover, it introduces a brief technical description of the prototype to shape the context in which the discussion takes place afterwards.

The scenario that we focus on is the following: a user is doing an outdoor activity (i.e. walking, jogging) and feels threatened by a possible perpetrator. In that case the user is willing to broadcast a message asking for help, but at the same time this action has to be discreet and not being noticed from the possible perpetrator. Hence, we design a long range emergency system, which includes a foot interface for capturing a foot gesture in case of emergency, a NN classifier to distinguish the gestures from other activities and an LPWAN to transmit the message. It is very important to distinguish accurately the gestures from activities because the user might be in danger and the foot interface is based on force sensors below the shoe sole, which are giving very raw data. Thus, the presence of a classifier that enhances the accuracy is essential. An advantage of using LPWAN, is that it ofers long range coverage (in smart cities environments), and therefore the user does not need to carry a smartphone or being dependent on any short range gateway, which is the case for many wearables designed for outdoor or sport activities.

During the design, we selected low cost consumer electronics which can operate on batteries in order to be suitable for the IoT ecosystem. The wearable system we propose can be divided in two parts, the hardware prototype and the NN classifier used to identify gestures.

To realize the prototype, we use a normal shoe and we deploy two force sensors below the shoe sole. One is deployed at the toe tip and the other at the heel as depicted in Figure 1. The force sensors are connected to the LPWAN Microcontroller Unit (MCU), which is glued to the side of the shoe with a small power-bank. The force sensors have a surface of 38 mm × 38 mm in square shape. The LPWAN device consists of an ESP32 MCU and a RFM95 LoRa [ 20 ] modem.

(a) (b) (c)

Mobile and embedded devices have benefited a lot from the development of NN during the last years [ 9 ]. The NN developments have lead to to scientific breakthroughs and has shaped the norm in pattern recognition and other features ofered from NNs in IoT and activity monitoring. In our approach, given the constrained resources of the MCU, we use an NN classifier with two hidden layers that operates by forwarding information in one direction through each layer in the network. The selection of the Machine Learning (ML) model was made after considering other models and evaluating the trade-of between accuracy and implementation complexity to fit on an ESP32 MCU, since the model is implemented on board. A more detailed description about the implementation of the prototype and its performance is described in [ 16 ]. 3

LONG RANGE COMMUNICATION IN FOOT

INTERFACES

Foot interfaces were investigated from the early start of HCI establishment. The decreasing size of electronics and cost have made this input modality more attractive. The combination of foot interface with a low-power long range communication is bringing some new characteristics which might result in a diferent user experience. In this section we try to outline how the new features from an LPWAN might afect the user experience of a foot interface.

One the factors which is diferent when LPWAN is used in foot interfaces is the Interaction Range compared with interfaces using Bluetooth Low Energy (BLE), IEEE 802.15.4 or shorter range wireless communication technologies. In that case a foot interface can be used within the coverage of smart environment where several LPWAN gateways are available. We speculate that the increased interaction range will improve user experience as the user will feel less dependent on smartphone or a desktop computer. But at the moment it is unclear if and how much the increased range factor can afect the user experience of a foot interface.

Another factor that may afect the user experience, and is related to the interaction range, is the Interaction With Other Devices. Foot interfaces traditionally interact with other devices such as mobile devices [ 25 ], desktop computers [ 21 ], public displays [ 19 ] and others. In the case where a foot interface interacts directly with a cloud service, like our application scenario, the output is also taking place at the wearable. Therefore, there are certain HCI aspects about the output that are required to be explored. The feedback or output in foot interfaces can be classified in visual, auditory, haptic and thermal and has been investigated through several applications [ 22 ]. An application using an LPWAN may have a delay to the output due to the long time-on-air values imposed from the physical characteristics of the LPWAN technology. The user experience might be afected from this drawback and thus it should be investigated further. First how a delay in the output may degrade the user experience in this context. Second, if there is any way to overcome this drawback. For instance, the time-on-air on some LPWANs varies a lot depending the configuration parameters. Which are the optimal parameters to have tolerant delay in the context of the focused scenario?

If we focus on the long range emergency scenario we introduced, where discreteness is a crucial requirement and assume that the outputs will take place on the wearable, the discreetness of the outputs should be evaluated as well. Fukahori et al. [ 6 ] introduced a foot interface for foot plantar-based inputs with force sensors attached on socks and evaluated if the foot gestures are observable in a public space. A similar evaluation should be carried out to investigate how discreet are the available outputs. Obviously some types of outputs are less discreet by default (auditory, visual) because they are directly observable, but some others, like vibrations, make more sense to be evaluated. Moreover, if the context is more broad and we just consider a foot interface where the long range communication allows it to operate as a standalone device and the outputs take place on the interface, all the available outputs should be evaluated because the user experience might be afected.

Human Computer Interaction aspects of Low-Power Wide Area Networks for Wearable Applications 3.1

Challenges

One of the issues that originates from the long range communication characteristic of LPWAN is to conduct a proper evaluation involving real experiments, because of the long distances and lack of testbeds. A proper evaluation of the wearable we propose would be to distribute it to a number of individuals and use it for a longterm period. Furthermore stabilizing the background variables in an urban environment is rather dificult and using an alternative environment (e.g. lab location) is not capturing the real scenarios we might desire for some cases.

One more issue is that even though there are several gateways in urban environments there is still the chance of going out of range and lose connectivity. Most of the times researchers set their own gateways to perform experiments but that can be time demanding and also restricted in terms of coverage area.

A problem which has to do with the methodology of HCI is that several times it is included video footage of the user using the interface to obtain timings or assess other characteristics of user experience. Following such a method when using long range communication is more challenging due to the fact that the evaluation might take place in-the-wild where footage infrastructure is not possible to be installed or it might be illegal. 4

RELATED WORK

This section covers other approaches with foot gesture interfaces used in various applications. Unfortunately we were not able to find any approach using LPWAN to carry out a comparison with the system we introduce. Therefore, we try to focus on the Interaction Range and the Interaction With Other Devices parameters of the mentioned approaches.

There are several attempts to investigate research questions around foot gesture interfaces. For instance, one of the first attempts to design a foot interface is described in [ 17 ], where a set of tiles with force sensors can be combined in diferent shapes on the lfoor. The main applications of interest is music and dance control, medicine and sports science but also control in computer games. The communication protocols for sensor networks were not very advanced at the time and the authors use a wired protocol where the tiles communicate with each other until they rich a sink node which is connected to a computer. Footsee [ 26 ] is a foot interface based on a sensor pad to be used as control for video games. The sensor pad consists of a grid of 160 by 64 pressure sensors and it is able to depict full body motions after an ofline training process.

A multimodal hand and foot gesture interface for handheld devices is presented in [ 11 ]. The interface is evaluated through a football game on a smartphone where the user is controlling an augmented ball with foot and hand gestures on the smartphone display. The results show that a multimodal game is more interesting and fun than a monomodal one which was used in the evaluation. Another approach is demonstrated in [ 18 ], where hand and foot gestures are combined to be utilized in multiple tasks on tabletop systems. The authors identify which foot gestures can be combined with hand gestures compared to the combined interface with single hand gesture and found that they require the same time while the combined one could speed up multitasking for some cases. ShoeSense [ 1 ] is a wearable which consists of a depth camera attached on the top of a shoe, pointing to the wearer and a single board computer. The rationale behind this approach is to capture a set of novel hand-gestures which can be associated with several scenarios like answering the phone, activate silence mode to the smartphone and many others.

Fan et al. in [ 4 ] study how often people want to use a foot gesture interface when both their hands are occupied and what kind of smartphone related tasks they would like to perform. Afterwards they develop footsketch, a foot gesture recognition app for smartphones. Footsketch uses acceloremeter data and a Dynamic Tree Warping algorithm to distinguish diferent foot gestures. After attaching the smartphone on the leg to evaluate the performance they found that for some cases, one can save over 70% of the time over a gesture compare with a traditional touch gesture on a smartphone display. Felberbaum et al. in [ 5 ] present a study to analyze and elicit users’ perception of foot gestures when they are taking place on a horizontal surface. The authors examine three diferent user conditions: standing in front of a display, sitting down in front of a desktop display and standing on a projected surface. Furthermore, a metric is introduced to quantify how a gesture is preferable to an action. Maragliulo et al. in [ 13 ], develop a foot gesture recognition system based on two electromyography (EMG) sensors, deployed at the lower knee. The system in combination with an SVM is able to identify a certain number of trained foot gestures. The interface is evaluated through use cases aiming at playing musical instruments which require equipment when the hands are occupied. A foot interface to induce a certain walking cycle is presented in [ 23 ]. The authors target navigation scenarios where the user might not consider the environmental circumstances and develop a prototype which obtains the walking cycle through pressure sensors and vibration motors to influence the specific walking cycle. An approach which is one of the closest to the one which is presented in this paper is presented in [ 6 ], where Fukahori et al. design a foot interface based on a sock with force sensors. The interface is recognizing a set of subtle gestures with the support of a ML model. The diferences with our approach is that the interaction range is shorter due to used the wireless technology (IEEE 802.15.4) and the ML model runs on the host computer and not on board.

All the aforementioned approaches have an interaction range below 300 meter approximately and the main devices which interact is a mobile device or a desktop computer. The long range communication ofered by an LPWAN is able to deliver a diferent user experience and afect the aforementioned parameters. 5

CONCLUSION

In this paper, we presented a position paper where we argue for a further investigation of the HCI aspects of LPWAN. The latter have been very popular in application scenarios like smart cities but when they are used in more human-centered applications there are still several questions to be answered. Therefore, we introduce a foot gesture interface implemented in a regular shoe with low-cost consumer electronics supported by a NN classifier and an LPWAN. We focus on a scenario where a user is doing an outdoor activity and feels threatened so she/he uses the foot interface to send an emergency message for help in a discreet manner. We highlight a set of questions and challenges which will assist to explore the user experience further when an LPWAN is present on an interface like the foot gesture interface we present.

References

[1] Gilles Bailly , Jörg Müller, Michael Rohs, Daniel Wigdor, and Sven Kratz . 2012 . ShoeSense: A New Perspective on Gestural Interaction and Wearable Applications . In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) ( CHI '12) . Association for Computing Machinery , New York, NY, USA, 1239 - 1248 . https://doi.org/10.1145/2207676.2208576

[2] D. F. Carvalho , A. Depari , P. Ferrari , A. Flammini , S. Rinaldi , and E. Sisinni . 2018 . On the feasibility of mobile sensing and tracking applications based on LPWAN . In 2018 IEEE Sensors Applications Symposium (SAS) . 1-6 . https://doi.org/10.1109/ SAS. 2018 .8336765

[3] Rashad Eletreby , Diana Zhang, Swarun Kumar, and Osman Yağan . 2017 . Empowering Low-Power Wide Area Networks in Urban Settings . In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (Los Angeles, CA, USA) ( SIGCOMM '17) . Association for Computing Machinery , New York, NY, USA, 309 - 321 . https://doi.org/10.1145/3098822.3098845

[4] Mingming Fan , Yizheng Ding, Fang Shen, Yuhui You , and Zhi Yu . 2017 . An Empirical Study of Foot Gestures for Hands-Occupied Mobile Interaction . In Proceedings of the 2017 ACM International Symposium on Wearable Computers (Maui, Hawaii) (ISWC '17) . Association for Computing Machinery , New York, NY, USA, 172 - 173 . https://doi.org/10.1145/3123021.3123043

[5] Yasmin Felberbaum and Joel Lanir . 2018 . Better Understanding of Foot Gestures: An Elicitation Study . In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18) . Association for Computing Machinery , New York, NY, USA, Article 334 , 12 pages. https://doi.org/10.1145/ 3173574.3173908

[6] Koumei Fukahori , Daisuke Sakamoto, and Takeo Igarashi . 2015 . Exploring Subtle Foot Plantar-Based Gestures with Sock-Placed Pressure Sensors . In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15) . Association for Computing Machinery , New York, NY, USA, 3019 - 3028 . https://doi.org/10.1145/2702123.2702308

[7] S. Heble , A. Kumar , K. V. V. D. Prasad , S. Samirana , P. Rajalakshmi , and U. B. Desai . 2018 . A low power IoT network for smart agriculture . In 2018 IEEE 4th World Forum on Internet of Things (WF-IoT) . 609 - 614 . https://doi.org/10.1109/WFIoT. 2018 .8355152

[8] Tahera Hossain , Yusuke Doi, Tahia Tazin, Md Atiqur Rahman Ahad, and Sozo Inoue . 2018 . Study of LoRaWAN Technology for Activity Recognition . In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers (Singapore , Singapore) ( UbiComp '18) . Association for Computing Machinery , New York, NY, USA, 1449 - 1453 . https://doi.org/10.1145/3267305.3267510

[9] N. D. Lane and P. Warden . 2018 . The Deep (Learning) Transformation of Mobile and Embedded Computing . Computer 51 , 5 ( 2018 ), 12 - 16 .

[10] Jansen C. Liando , Amalinda Gamage, Agustinus W. Tengourtius, and Mo Li . 2019 . Known and Unknown Facts of LoRa: Experiences from a Large-Scale Measurement Study . ACM Trans. Sen. Netw . 15 , 2 , Article 16 ( Feb . 2019 ), 35 pages.

[11] Zhihan Lv , Alaa Halawani, Shengzhong Feng, Haibo Li , and Shafiq Ur Réhman. 2014 . Multimodal Hand and Foot Gesture Interaction for Handheld Devices . ACM Trans. Multimedia Comput. Commun. Appl . 11 , 1s , Article 10 (Oct. 2014 ), 19 pages. https://doi.org/10.1145/2645860

[12] D. Magrin , M. Centenaro , and L. Vangelista . 2017 . Performance evaluation of LoRa networks in a smart city scenario . In 2017 IEEE International Conference on Communications (ICC) . 1-7 . https://doi.org/10.1109/ICC. 2017 .7996384

[13] S. Maragliulo , P. F. A. Lopes , L. B. Osório , A. T. De Almeida , and M. Tavakoli . 2019 . Foot Gesture Recognition Through Dual Channel Wearable EMG System . IEEE Sensors Journal 19 , 22 ( 2019 ), 10187 - 10197 .

[14] Kais Mekki , Eddy Bajic, Frederic Chaxel, and Fernand Meyer. 2019 . A comparative study of LPWAN technologies for large-scale IoT deployment . ICT Express 5 , 1 ( 2019 ), 1 - 7 . https://doi.org/10.1016/j.icte. 2017 . 12 .005

[15] C. Orfanidis , L. M. Feeney , M. Jacobsson , and P. Gunningberg . 2017 . Investigating interference between LoRa and IEEE 802. 15 . 4g networks . In 2017 IEEE 13th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob) . 1 - 8 . https://doi.org/10.1109/WiMOB. 2017 .8115772

[16] C. Orfanidis , R.B.H. Hassen , A. Kwiek , X. Fafoutis , and M. Jacobsson . 2021 . A Discreet Wearable Long-Range Emergency System Based on Embedded Machine Learning . In 2021 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops).

[17] Bruce Richardson , Krispin Leydon, Mikael Fernstrom, and Joseph A. Paradiso . 2004 . Z-Tiles: Building Blocks for Modular, Pressure-Sensing Floorspaces . In CHI '04 Extended Abstracts on Human Factors in Computing Systems (Vienna, Austria) ( CHI EA '04) . Association for Computing Machinery , New York, NY, USA, 1529 - 1532 . https://doi.org/10.1145/985921.986107

[18] Nuttapol Sangsuriyachot and Masanori Sugimoto . 2012 . Novel Interaction Techniques Based on a Combination of Hand and Foot Gestures in Tabletop Environments . In Proceedings of the 10th Asia Pacific Conference on Computer Human Interaction ( Matsue-city, Shimane , Japan) ( APCHI '12) . Association for Computing Machinery , New York, NY, USA, 21 - 28 . https://doi.org/10.1145/2350046.2350053

[19] William Saunders and Daniel Vogel . 2015 . The Performance of Indirect Foot Pointing Using Discrete Taps and Kicks While Standing . In Proceedings of the 41st Graphics Interface Conference (Halifax, Nova Scotia, Canada) ( GI '15) . Canadian Information Processing Society , CAN, 265 - 272 .

[20] Semtech . 2020 . What is LoRa®? https://www.semtech.com/lora/what-is-lora. Accessed: 2020-12-21.

[21] Eduardo Velloso , Jason Alexander, Andreas Bulling, and Hans Gellersen . 2015 . Interactions Under the Desk: A Characterisation of Foot Movements for Input in a Seated Position . In Human-Computer Interaction - INTERACT 2015 , Julio Abascal , Simone Barbosa, Mirko Fetter, Tom Gross, Philippe Palanque , and Marco Winckler (Eds.). Springer International Publishing, Cham, 384 - 401 .

[22] Eduardo Velloso , Dominik Schmidt, Jason Alexander, Hans Gellersen, and Andreas Bulling . 2015 . The Feet in Human-Computer Interaction: A Survey of Foot-Based Interaction . ACM Comput. Surv . 48 , 2 , Article 21 ( Sept . 2015 ), 35 pages. https://doi.org/10.1145/2816455

[23] Junji Watanabe , Hideyuki Ando, and Taro Maeda . 2005 . Shoe-Shaped Interface for Inducing a Walking Cycle . In Proceedings of the 2005 International Conference on Augmented Tele-Existence (Christchurch, New Zealand) (ICAT '05) . Association for Computing Machinery , New York, NY, USA, 30 - 34 . https://doi.org/10.1145/ 1152399.1152406

[24] F. Wu , C. Rüdiger , J. Redouté , and M. R. Yuce . 2018 . WE-Safe: A wearable IoT sensor node for safety applications via LoRa . In 2018 IEEE 4th World Forum on Internet of Things (WF-IoT) . 144 - 148 . https://doi.org/10.1109/WF-IoT. 2018 . 8355234

[25] T. Yamamoto , M. Tsukamoto , and T. Yoshihisa . 2008 . Foot-Step Input Method for Operating Information Devices While Jogging . In 2008 International Symposium on Applications and the Internet . 173 - 176 . https://doi.org/10.1109/SAINT. 2008 .97

[26] KangKang Yin and Dinesh K. Pai . 2003 . FootSee: An Interactive Animation System . In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (San Diego, California) ( SCA '03) . Eurographics Association, Goslar, DEU, 329 - 338 .

[27] Jia-Xin Yue , Zhong Wang, Qi Yang , and Xiao-Bin Zheng . 2018 . Design and Implementation of Intelligent Monitoring System for the Elderly on Android Phone . In Proceedings of the International Conference on Information Technology and Electrical Engineering 2018 (Xiamen, Fujian, China) ( ICITEE '18) . Association for Computing Machinery , New York, NY, USA, Article 63 , 8 pages. https: //doi.org/10.1145/3148453.3306302