Nowadays wearable devices are part of our everyday lives. More and more devices that we carry every day are becoming smart and gain more and more features. One of them is the humble watch, which has been telling the time and other useful basic information for ages. Since this inconspicuous device is worn almost all the time, it can be successfully used for activity recognition. We propose a system that uses a smartwatch as a primary movement data collection device that connects to a local data hub, which acts as a data gateway. The local hub's role is to upload the data into the cloud, a MongoDB database in realtime. To recognize activities, this system will process the obtained data ofline, using the ML.net machine learning framework. This proposed activity recognition system can be easily expanded due to the local data hub and provides a non-intrusive method of collecting user's data with only a smartwatch. Combined with the machine learning data processing mechanism, this ensures a flexible activity recognition system that has a good recognition rate.
Wearable devices are becoming everyday objects, assisting us in many fields
including healthcare [
Our objective is to implement and test a data recognition system that uses a smartwatch as the main data source. The sensor data is then uploaded to the cloud so that the ofline data processing phase can begin. The main diference between the proposed system and other similar systems is using the ML.NET framework for the processing phase to obtain a portable and low footprint activity recognition component that is .NET compatible.
Activity recognition is done using various sensors [
Our proposed system consists in a smartwatch that is used for data acquisition and a couple of software applications that are used to retrieve, upload data into the cloud, and process the obtained data-set ofline. Uploading data into the cloud is done using a middle-ware local data hub, that receives the data from the watch and uploads it directly to the cloud. The local data hub, a raspberry pi device, is used to overcome any networking issues and latency. The high level architecture of the proposed system, showing the main components and their interaction, is shown in Figure 1.
The system can be used for activity recognition as a base system or can be part of a more complex activity recognition system, synchronizing and correlating data with other systems. Having another device, a smartwatch that is worn by the user on his wrist, helps significantly in providing correlation data and additional information regarding the user’s current activity. The local data hub simplifies any future integration with other systems and allows for local data processing. It provides a simple method for the system to be expanded. Currently the proposed system is used standalone, without being integrated into a larger system, and only for testing.
The chosen activity detection device is the Samsung Gear S3 smartwatch. This
was chosen due to its low price, small form factor(46mm x 49mm), sensors, and
additional features. Even though this is a commercially available product, due to its
built-in sensors, it is capable of monitoring user activity [
The data from the chosen smartwatch, the Samsung Gear S3, was gathered using a custom Tizen app written in C#. This Tizen .NET application has access to the underlying sensor layer and its values and runs on the watch.There are multiple advantages of using the C# programming language, besides rapid application development, as we benefit from the Common Language Infrastructure standards and a managed runtime. This application handles the data gathering process from the smartwatch device and uses a WebSocket to transfer this data to the cloud. This process uses a middle-ware component, a simple local data hub, that handles the cloud uploading process.The obtained data is being also displayed on the watch face as shown in Figure 2 while the watch application is running.
The local data hub is used as a temporary data storage and includes the cloud upload functionality. The data from the smartwatch is received using a WebSocket and then it is relayed to a cloud-based database. The cloud database we used for testing is a MongoDB database since it’s document orientated and allows for rapid scaling. The local hub application runs on a Raspberry Pi computer and is comprised currently of a basic data routing logic, taking data from the WebSocket, storing the data in memory and uploading it immediately in the MongoDB database.
To identify the activity that the user was executing, based on the movement data collected from the smartwatch, we need to process this data. For the data processing flow, we choose a .net machine learning framework, ML.Net. With this framework, we have access to complex pipelines for machine learning processing. Due to the emergence of .net Core, which can run on multiple operating systems, this library can be used on a wide number of platforms. One of the most important aspects of choosing this framework was the fact that is open source, allowing the developer to see or even alter the source code.
The gathered and processed data is comprised of accelerometer and gyroscope data for the three axes: x, y and z. After the data is loaded into the cloud, it is currently ofline processed using the ML.NET framework to analyze the activity types.
There are two major steps for the data processing phase: training the machine learning algorithm and using the algorithm to detect the activity type.
The machine learning training process was done using the ML.net framework, using a very small training time, 20 seconds. If required, this training time can be increased in order to allow more training time for each of the available algorithms that are used by the framework. After the training completes, we obtain a comparative list of algorithms used and their accuracy. The top five models are shown in Figure 3 alongside the validation error, loss for each algorithm.
From the above-mentioned list, the best algorithm is selected to be used and the accuracy is displayed as the result alongside the actual training time used by this particular algorithm. The best algorithm, as shown in Figure 4, is FastTreeRegression with a coeficient of determination of 0.9392.
The chosen FastTreeRegression algorithm is an implementation of the MART gradient boosting algorithm and yielded good results on the used data-set. This machine learning algorithm was trained using a small data-set of about 9000 records and tested with an additional data-set of about 3000 records.
Once the training step is complete, the machine learning component can be used to detect the activity type for the received movement data.
This paper presents the implementation and testing of a smartwatch based activity detection system that process the data ofline. The usage of a commercially available product proved to be a good choice, as it allows user activity monitoring without many additional body worn devices or sensors.
Using the ML.net framework provides a powerful sensor data analysis tool that can be rapidly trained and used in real-time applications. It can be greatly expanded with custom code and processing to further increase the data recognition rate. Also to increase the current recognition rate, a pre-processing step should be implemented as well.
Thus using a smartwatch based activity recognition system proved to be a great versatile and small footprint implementation that can easily be expanded or modified for various applications.