This paper describes a smart ecosystem able to continuously monitor patients' health condition, at home or at work or during recreational activities, by exploiting a creative blend of Medical Wearables, Intelligent Agents (IA) and Complex Event Processing (CEP). With the help of a smart application, that links together the Wearables and the power of IA and CEP, patients will be continuously and actively supervised during their daily activities. This can even save their lives, in case sudden or gradual issues should occur. Thanks to our system, patients with non-severe though potentially unstable chronic diseases will no longer overburden first-aid services. This is also useful for containing the spread of COVID-19. Specifically, in this paper we focus on automated vitals monitoring, electrocardiogram (ECG) analysis, and psoriasis detection. Experimental results carried out on real patients show how promising our approach is.
The Coronavirus pandemic has highlighted telehealth as a crucial component of modern and sustainable treatment. In fact, telemedicine’s primary purpose is to virtually erase the distance between patient and physician, as well as to reduce time and expenses involved in healthcare access. Consequently, in this context here we propose an approach which exploits in a synergistic way wearable devices (such as portable ECG devices and pulse oximeters), Intelligent Agents (IA), and Complex Event Processing (CEP) techniques, to create an integrated framework able to follow the patient wherever (s)he is.
In the literature, even if some works may appear similar to ours [
The remainder of the paper is organized as follows. Section 2 describes the system architecture, section 3 shows an overview of the experimental results, and section 4 concludes the paper and outlines future work.
Our system consists of six main components: User and Doctor Interfaces - Android App, Database and Web interface, Hardware, Intelligent Agents, Complex Event Processing and Image Processing.
Figure 1 depicts the entire workflow of our system. Specifically, to interact with the system, ifrst of all the patient (possibly supported by a caregiver) downloads the Android application on his/her smartphone. After downloading the app, (s)he can begin using it, inserting his/her personal information, such as name and age. Then, (s)he can perform the manual data acquisition, by entering the information listed in the so called activity within the green box in Figure 1.
After inserting all the manual data, and after wearing suitable devices if needed, the patient can proceed to the actual usage of the system.
In all situations, if, after analyzing the data in connection to the sensations and wellbeing impressions possibly provided by the patient, the patient turns out to be healthy, then our system does nothing, except for providing reassurance and suitable advice (e.g., “stay seated for some minutes, breath deeply and try to relax”), and reminding the patient of the medicines to take. If minor issues are detected, then the system alerts the user using Android notifications. If severe issues are detected, then the system alerts the doctor, and, if the situation is critical, it alerts the first aid services.
Oxygen saturation is obtained via the Arduino device. The device is connected via Bluetooth, so the data are directly sent to the server through the application.
The manual data and the oxygen saturation value are then sent to the DALI Intelligent Agents, which proceed to searching for and identifying possible health issues. The agent coding has been performed following the advice of medical doctors.
After obtaining oxygen saturation, the patient can start the ECG acquisition process, by connecting the related ECG device. After connecting, wearing the device, and acquiring the data, those are automatically sent to the server for processing.
The ECG data are then elaborated, and short-term events, if present, are detected; the data are subsequently sent to the ISEQL engine, which is in charge of constantly checking for the presence of health issues that cannot be seen by instantaneous analysis, but can be instead detected via long-term observation, elaboration and aggregation of events (this by performing “Complex Event Processing”).
Additionally, the patient can also send pictures of his/her face and/or hand to be checked by a doctor, in case (s)he wants her/him to better evaluate the situation. In the case of the face picture, the image is directly sent to the doctor, who, after evaluating the patient’s appearance, can decide, if the detected situation does not look good (e.g., excessive paleness with deep dark circles), to alert either the user, providing suitable advice or making an appointment for a face-to-face medical examination, and/or the first aid services. In the case of the hand picture, it is automatically processed by our software, to find signs of hand disease. In particular, as a first case study, we have considered psoriasis as it is a rather frequent pathology.
In our work, we use DALI [
It also acts as an e-health companion, reminding the user to take pills at the correct time, sending a reminder for temperature, blood pressure and weight measurements. The Doctor agent receives messages from the Patient agent, analyzes the communicated problem, e.g., fever or tachycardia, and responds by suggesting a medication to take or, in case of serious danger, it suggests going to the Emergency Room. Events currently detected by our DALI intelligent agents are angina pectoris, tachicardia, bradhicardia, hypoxia, hypertension and fever (Figure 2).
The language we exploit in order to carry out Complex Event Processing tasks in this context is
named ISEQL (it stands for “an Interval-based Surveillance Event Query Language”); some of
the authors of this paper defined it in [
A Short-term Event is an event to be detected within the context of a single ECG measurement.
To detect such events, we must analyze the ECG signal in depth using specific criteria stored in
the knowledge base and complying with the guidelines found in the literature [
Moreover, on top of such events, we define specific additional event models in ISEQL, aimed
at identifying possible anomalies in ECGs, thus extending the literature in the context of their
automatic analysis. More specifically, the events are modeled in ISEQL and associated with
their severities on a scale from 0 (lowest severity) to 3 (highest severity - in such a case, a
medical doctor is automatically alerted). One example is the Too Long QT interval, which could
be potentially very dangerous. Additionally, exploiting its cardinality constraints [
Working on aggregated data at a diferent resolution, such as examining the results of several
ECG measurements taken in specified wider temporal windows, allows a detailed long-term
analysis of the patient’s clinical history. Here, we focus on the detection of diferent categories
of long-term events. In fact, on the one hand we identify some possible long-term anomalies
exploiting ISEQL; for instance, the specific scenario shown in Figure 4 is easily detectable via
ISEQL, exploiting the cardinality constraints [
On the other hand we also implement specific algorithms for detecting the Atrial Fibrillation
(AF) and the Wolf-Parkinson-White (WPW) syndromes [
We defined an image processing algorithm aimed at elaborating and extracting features from images, in order to evaluate the condition of the patient’s hand, and possibly diagnose the presence of psoriasis. In order to correctly elaborate the image of a person’s hand, and make it 1 2 3 4
5 ti6me (day7s) temporal window 8 “visible” to the computer, we exploit one of the most used software for image elaboration, that is named ImageMagick2.
In this paper, we proposed an integrated framework aimed at actively supporting patient monitoring by exploiting an innovative combination of wearables, DALI intelligent agents and Complex Event Processing. The developed framework also includes an efective image processing algorithm for psoriasis detection. Preliminary experiments conducted on real patients confirm the validity of the proposed approach.
Future work will be devoted to extend the system in many ways; in fact, the intelligent agent will cope with many other health issues, the set of short-term and long-term events detectable via ISEQL will be expanded, and the psoriasis detection algorithm will be able to automatically detect other skin diseases.