The exponential growth of the technology industry over the past years has led to the development of a new paradigm. Pervasive Computing, which seeks to integrate computational capabilities into everyday objects so that they can communicate efficiently and perform useful tasks in a way that minimises the need for the end-users to interact with computers. Along with this paradigm, new technologies, such as wireless body area sensor network (WBASN) and robotics, have been rapidly growing. Such innovations can be used as health and wellness enablers. This paper introduces a system to support the health and wellbeing of people in both in-door and out-door environments. The system consists of an app, a robot and a remote server. The app and the robot can be connected to a wristband to monitor movements, physical activities, and heart rates. Furthermore, the app provides functions for users to record and monitor their calories intakes, and completed workouts. The robot is equipped with speech capabilities which, integrated with the emotion recognition algorithm, provide supportive feedback to the user. The proposed system represents an early step towards automated care in everyday life which opens the doors to many new scenarios, such as for elderly people who need help but live independently or also for people who would like to improve their lifestyles.
In 1991 Mark Weiser introduced his vision about the evolution of computer in
the next century and the possibility of using computers and other devices in
our lives without the user being aware of it [
Pervasive computing is typically associated with the race to miniaturise
hardware with some degree of connectivity, intelligence and mobility [
This paper proposes a solution to support the health and well-being of people in daily living, integrating data from wearable sensors with a robot. The system is composed of three heterogeneous sub-systems: a mobile application, a robotics system and a remote server. The mobile application allows keeping track of the user’s everyday activities, such as food consumed and workouts performed, along with his extrapolated health details from a wristband. The mobile application allows also, to check the user trends by the user himself or by a designated parent. The robotic system is designed as an assistance agent capable of speaking, recognising voice commands and collecting fitness data from the user. The robotic system aims to recognise the user’s emotion and consequently start a conversation. The remote server is the main source of storage for the system. It stores data from both the mobile application and the robotic system, subsequently, it performs data processing and reasoning. The collaboration between these three sub-systems enables the development of a system that is capable of collecting information from the user via multiple sources of information, offering more accuracy and reliability. The multiple sources of knowledge allow the system to operate in both indoor and outdoor environment, since they are not mandatory together, allowing a greater degree of independence for the user. The objective is to design a system that can be used by most people without any hindrance, keeping costs to a minimum and making the system as simple as possible.
This paper is structured as follow. Related works are described in Section 2. The architecture and implementation of the system are presented in Section 3. The tests are described and evaluated in Section 4, followed by the findings
SenseBot 3 discussed in Section 5. The paper is concluded by a summary and future work in Section 6. 2
Thanks to the various improvements in the area of robotics and the
miniaturisation of wearable devices, different works can be found in the literature, that
combine these two technologies to improve healthcare. Huang et al. [
Gorˇsiˇc et al. [
Novel robotics and cloud-assisted healthcare system (ROCHAS) was developed
by Chen et al. [
Ma et al. [
This section will describe in detail the system architecture, dealing with the communication between the different subsystems. 4
The proposed system is composed of three main components: remote server
(henceforth called PyServer), mobile application (henceforth called MyFit ), and
robot infrastructure (henceforth called PyBot). These three components work
together to gather information from a person, store the information and take
actions to help the person to be healthy. As can be seen in Fig. 1, the
communication between these components is led by the HTTP protocol, and the MyFit
and PyBot exploit a Bluetooth Low Energy (BLE) protocol to interface with the
wearable sensor. The choice of the HTTP protocol is driven by the need for a
protocol that can be suitable on multiple and heterogeneous devices in different
locations. On top of the HTTP protocol are created different Representational
State Transfer (REST) API architectures, as described below, to encourage the
use of an understandable language for all the components, JSON (JavaScript
Object Notation), which is an open standard file format, and data interchange
format, that uses human-readable text to store and transmit data objects
consisting of attribute–value pairs and array data types (or any other serializable
value) [
MyFit is a mobile application that can help the overall system to acquire new data from the user during the day. The app is divided into two versions: the fit version and the robot version.
The main purpose of the fit version is to gather information from wristband (steps done, distances travelled, calories burned, heart rates), instead, the main purpose of the robot version is to control remotely the PyBot. Leaving aside the
SenseBot 5 differences in the main purpose, the other functionalities of the app are shared for both the versions. The entire navigational path of MyFit is shown in Fig. 2
The app can be accessed only after authentication, then the user, when starts the app, has to sign-in, if already register, otherwise he can sign-up and automatically he is sign-in. After the authentication step if the app run is the fit version the user has to choose the wristband from the list of available Bluetooth devices and then it is redirected to the main view where are shown all his fitness data, otherwise, if the app run is the robot version it is redirected to the main view where it can control the PyBot. Fig. 3 shows the different views that the user sees according to the version.
(a) fit version
(b) robot version
Fig. 3: MyFit: main views according to the versions 6
Once the user is in the main view, he can move among other views through the left navigation drawer, activated by the button on the top left. The possible views are: – daily calories: this view allows the user to see and to add the food eaten with the numbers of calorie, in the current day or the past days according to the meal. – daily workouts: this view allows the user to see and to add the activities done with the amount of time spent, in the current day or the past days. – statistics: this view allows the user to see the trend of its fitness data in the last 10 days. – family: this view allows the user to see its family members1, add a new one scanning his QR-code, be a family member of another user generating the QR-code and letting him scan it.
MyFit is developed for Android Devices, the devices supported are all the devices that run a version of Android between Android 7.0 and Android 10. It is developed in JAVA and XML. 3.2
The physical design of the robot platform allows to interact in a friendly
way with the user, to recognise the user’s emotions, and to recognise the fitness
activity made by the user. The robot is based on a design made by [
SenseBot 7 with two wheels are combined to allow the robot to move, allowing flexibility on which type of surfaces the robot can travel, thus restricting it to flat surfaces. Since the robot can move, it has incorporated a distance sensor to prevent it from colliding with objects on the front. The DC motors and the distance sensor are managed by the GoPiGo3 board. The robot has to communicate with the user in a friendly way, to reduce the gap between human and machine, for this reason, are integrated a screen that visualises information, a speaker, and a microphone. The robot has to, also, recognise facial emotion of the user, to allow this a Raspberry Pi Camera module is mounted on the front of it.
The robot platform can interact with the user and take decisions thanks to a python program. The program is built of four processes that run parallel on the machine: digital assistant, API manager, fit manager, emotion manager. Digital Assistant This process is mainly responsible for the conversation with the user. The digital assistant can recognise and speak four different languages (English, Italian, Chinese, Spanish) but only one at a time, this can be set at the stat-up of PyBot.
When the PyBot starts up the digital assistant begins recording sound.
Through the SpeechRecognition library [
The examples of recognised commands and the respective responses are: – hello: it says ”hello” – how are you: it says ”I’m fine and you?” – follow me: it says ”Let’s go” and starts walking forward – turn right : it says ”turn right” and turns right – turn left : it says ”turn left” and turns left – go back : it says ”ok I’ll go backward” and go backward – play: it plays random music Api Manager This process is mainly responsible for providing external APIs. The APIs allows the user to access the PyBot resources from a different location without the need to have the PyBot nearby. The APIs handle different resources: – movements: movements performed by the PyBot – stream: camera stream of the PyBot
Fit Manager This process is mainly responsible for the gathering of data from the wristband. The process can acquire from the user: steps done, distances travelled, calorie burned, heart rates.
The process is based on the library provided by [
Emotion Manager This process is mainly responsible for the recognition and handling of user emotions. The process captures every 10 seconds a frame from the camera module and then sends the image captured to the PyServer, which aims to recognise the emotion and send back the result. The way of emotion recognition by PyServer is described in the next subsection. The emotion manager when receives the result from the PyServer, analysis the result, and according to the emotion obtained it starts a conversation with the user. 3.3
The PyServer is designed to act as the remote server of the system proposed. It is responsible for the storing of the information generated by both the MyFit
SenseBot 9 and the PyBot, as well as for the operations of emotion recognition. As mentioned at the beginning of this section the architecture used is the REST API which can use an interchangeable language to manipulate the HTTP protocol and to provide resources.
The APIs handle different resources: – users: users registered in the system – foods: foods eaten by users – activities: training activities done by users – steps: steps done by users – distances: distances travelled by users – calories: calories burned by users – heart rates: heart rates of the users
The PyServer is designed to be secure, provided an authentication strategy, mandatory to access resources, based on a token, JWT. The token allows identifying users and their role, to restrict the resources according to which actor is logged in.
To allow emotion recognition a pre-trained CNN, developed by Serengil [
The flow of the activities to predict emotions is shown in Fig. 6.
When the emotion recognition resource is called the server expects to receive
an image as input. When the image is loaded, the conversion of its colour scheme
10
is applied, which is changed in grayscale. The segment where the face is located
is obtained, if present, by using an OpenCV function [
The PyServer is developed in Python, the APIs are developed using the Flask framework. For the storage is used a relational database to provide a well-defined structure with multiple relationships. 4
To evaluate the functionality offered by the system, several usability tests had been defined to be submitted to different people to evaluate the ease of use of the system and the correct functioning, but due to the current critical world situation, caused by the spread of a disease on a large scale, the tests could not be completed.
Three macro groups of tests have been identified, one in which the MyFit app is tested, one in which the PyBot is tested, and one in which the facial detection features are tested. For each test, the user had to be asked to complete some activities to test the usability and correctness of the system. The experiments were conducted in the laboratory. Participants were given a set of tasks to test MyFit and PyBot. The following results were recorded: – Time to complete a task per user – Number and type of error per task – Number of errors per unit time per user – Number of users completing a task successfully
After carrying out the tests a questionnaire on how the users feel about using the product, by asking them to rate it along with a number of scales, after interacting with it.
The facial detection test was carried by one person only, due to the covid19. The person involved is a male of 23 years old. The test was set up in a room of around 10 sqm with no artificial light with few background noises. The person was asked to stand still in front of the PyBot for 4 minutes, two minutes looking at it and two minutes with their backs facing it. The test is repeated four times at different distances: 50 cm, 100 cm, 150 cm, 200 cm. The aims are to evaluate the performance of the underline face detection infrastructure when the user is in front of the robot in both looking and not looking scenarios.
The data obtained are classified into two classes, face recognised (1) and face not recognised (0), so it will be:
SenseBot 11 – True Positive (TP): the person is looking at the PyBot and the face is recognised – False Positive (FP): the person is not looking at the PyBot and the face is recognised – True Negative (TN): the person is not looking at the PyBot and the face is not recognised – False Negative (FN): the person is looking at the PyBot and the face is not recognised
The confusion matrices from the result of each experiment are shown in table 1.
not present
present not present
present not present
present not present present 12
T P + T N
ACC = T P + F P + T N + F N
T P P REC = T P + F P
T P SN = T P + F N
T N
SP = T N + F P
Results show that the underlying infrastructure is robust and can also be used for further emotion detection, which is not covered in this paper. However, this must be considered only as a preliminary study as the experiments were conducted on one person and the images collected are fairly few. 5
The system aims at gathering the user’s health data without destroying what is his daily life. Using different heterogeneous systems has allowed reaching a good compromise and it has allowed monitoring the user’s activities even when the user decides to leave home. In addition, the use of a robotic system to communicate with the user allows the distance between the user and the system to be minimised making the user feel more comfortable.
To better understand what the advantages and disadvantages of this system are, it is good to analyse the three proposed sub-systems differently. 5.1
The MyFit is responsible for gathering the major number of information from the user. After considerable and prolonged use, several advantages have emerged. The most important is that the user as soon as he accesses the application starts a background service that collects information from the wristband without the user having to interact with it. The only limitation of this service is that the app must be run in the background even if closed, otherwise, the service will not work until the app is reopened. Another advantage is that the app provides the
SenseBot 13 user an all in one place to record his daily habits, usually a user has to install different applications. In addition, the user can check his trends, or even more useful can check the trends of a family member. The main issue raised up is the network reliability for the data storage, because there is an algorithm that sends the data to the server and stores in local the information added by the user, but in case of network connection leak, there is no way to understand if the data was been sent to the server or not and if so retry. MyFit can be connected to a wristband but is now only compatible with Xiaomi Mi Band 3, so it can think of expanding compatibility to many more wristbands. 5.2
The PyBot is responsible for acquiring information from the user, but user
interaction is even more useful. PyBot can offer several advantages to the user.
The most important thing is that if the user is not happy to wear a wristband,
the data can be acquired by the PyBot, leaving the user free of wearable devices,
but at the same time under the control of the robot. It is also important that
PyBot can be used to encourage the user, for example on a bad day, by
providing support through conversation or playing music. During the development
different design issues are raised:
– Energy issue: the PyBot has several sensors connected to itself, so the
average energy consumption is significant. Due to its mobility, it cannot be
charged for long and often, otherwise, it becomes useless. Optimising energy
consumption is, therefore, the main question that must be solved
– Network reliability: the PyBot uses different services over the internet, the
emotion recognition provided by the remote server, the text-to-speech and
the speech-to-text provided by a third part. Therefore, is necessary to ensure
a reliable connection or provide an algorithm that can compute in local.
– Quality of the modules: the PyBot integrates different modules, fig 4. The
use and the reliability of these modules is a key concept to allow the PyBot
to work well. Problems relating to modules are identified during tests. The
camera used is not capable of operating in a low light environment and the
resolution is poor, so it can lead to user emotion confusion due to low image
quality in everyday use. The microphone used takes on a lot of ambient
noise, so it can lead to wrong speech recognition. The distance sensor is not
able to identify all the obstacles in front of it. The distance sensor utilises
ultra-sonic waves that are useful because they are not influenced by object
light, colour or transparency, but they are not reflected by soft materials so
that the robot sometimes fails to identify the person in front of it.
– Emotion recognition: the emotion recognition is one of the most complex
machine learning problem because the emotions of people can affect
differently the face of each person, so it is very difficult to identify a pattern. To
improve the reliability of emotion recognition is possible to integrate with
face images also audio recording and analysis of body movements [
To solve the design-related issues, just assume the components are easily replaceable thanks to the system’s modularity, which can lead to great improvements according to needs. The PyServer is the main source of storage. Since a huge amount of extensive data of the user is collected the privacy invasion will be very serious. After considerable and prolonged usage, it can be established that the system used for authentication, token, enables to avoid possible holes in the system while protecting user data. Furthermore, all data collected are stored following the GDPR guidelines. The other function of PyServer besides storage is to recognise emotions, the server-side emotion recognition can be modified to take advantage of new sources of knowledge according to the previous subsection on PyBot. In summary, a good level of data collection has been achieved which can lead to different new scenarios. One possible scenario is to create a dataset containing the fitness activities, the foods eaten, the training activities, the emotions belong to the users, related to their health, in order to create a machine learning algorithm that can predict the health status of a user collecting only this information. 6
A heterogeneous framework for controlling and improving health and well-being is proposed in this paper. A smartphone app, a remote server, a robot and a wristband make up the whole system. The mobile app facilitates the processing of information from the wristband, allowing, also, the user to record the foods consumed and the activities performed. The robot is capable of gathering information from the wristband without user intervention and is capable of understanding the user’s emotional state to assist it, additionally the robot’s abilities to talk and listen lead to reduce the gap between robot and human. A limitation of the study is the lack of a usability study, due to the closure of university campuses caused by the spread of Covid-19. In fact, to improve the performance obtained one of the most important future works will be to organise test sessions in a controlled environment to consolidate the work done. In conclusion, it can be noticed that the mutual exclusivity between the app and the robot let the user a greater degree of freedom maintaining a good level of information collection and thanks to the use of the PyBot it is possible to make the user feel safer and more peaceful. Furthermore, the possibility of having a remote view of family members can be used in an easy way to monitor their habits without being too invasive. The proposed system represents an early step towards automated care in everyday life which opens the doors to many new scenarios. The system could be used as an encouragement for people who are reluctant to play sports or other physical activity, motivating them to increase
SenseBot 15 their participation on days when they are more sedentary, to maintain their good health and to prevent obesity. The system could be used as a help for elderly people, who wish to maintain their autonomy but are required to seek thirdparty assistance.
Future work will be carried out to incorporate other wristbands with the MyFit app, to improve the facial detection and emotion detection algorithms, and to undertake a large scale of data collection and evaluate the system. 7
This research is partially supported by the Beitto-Ulster collaboration programme.
D’Arco et al.