To measure vital parameters, traditional devices are equipped with sensors that need to be used through contact with the subject's skin. In recent years photoplethysmography has been developed as a contactless method to monitor vital signs. Thanks to this method, di culties concerning the detection of parameters through contact devices can be overcome, especially in elderly subjects. In this work we use remote photoplethysmography to estimate cardiovascular parameters through the use of a contactless device equipped with a re ective mirror and a webcam that captures video frames of people's faces. Besides, we use the clustering technique to automatically estimate the lips colour. Finally, the measured parameters are used as input to fuzzy inference rules integrated into our system, in order to predict cardiovascular risk.
According to a 2015 report3, cardiovascular diseases are one of the leading causes
of death in the world. Therefore, monitoring cardiovascular functions is essential
to prevent the onset of chronic diseases and carry out therapies in an appropriate
manner. In order to detect the risk level of cardiovascular diseases [
Another low-cost non-invasive technique that detects the cardiovascular pulse wave through variations in transmitted or re ected light is photoplethysmography (PPG). Through PPG, it is possible to obtain values such as heart rate,
Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
arterial blood oxygen saturation, blood pressure, cardiac output and autonomic
function [
This paper provides a synthetic description of the ongoing PhD research
activity of the author. A more detailed description can be found in [
Using photopletismography, the video frames are processed in order to derive an estimate of vital parameters such as Heart Rate (HR), Breathing Rate (BR) and Oxygen Saturation in blood (SpO2). As an additional vital sign, lips colour is automatically detected using clustering-based color quantization. Unlike other existing contact-less monitoring solutions, that are oriented only to measure vital parameters, the proposed solution integrates an intelligent component that provides for a support to medical diagnosis of cardiovascular disease. This component uses fuzzy IF-THEN rules to infer a cardiovascular risk level starting from the values of the vital parameters.
The proposed solution is a cheap device that is easy to use, lending itself very well for domestic use as well as for telemedicine applications. 2
This work proposes a solution for real-time estimation of cardiovascular parameters without the use of contact sensors, but with the use of a see-through mirror equipped with a HD camera that acquires the video frames of human faces and processes the signal from facial blood vessels to measure heart rate, breathing rate and blood oxygen saturation values. Our aim is to create a smart personal monitoring device made of a few assembled low-cost HW components. The principal device is a see-through mirror which has a 12" 12" 3mm thick acrylic lm that is partially re ective and partially transparent. A monitor is put in the darker side so that the output of the system can be displayed through the mirror. A Microsoft LifeCam has been used to ensure high images quality and sharpness. It is quite small and is equipped with autofocus and a 1080p HD sensor. This type of camera has been integrated with the see-through mirror to ensure quality video images. Two LED strips, each composed of 18 LED lights, have the following features: 12V, 6.0W, 0.5A and 120 beam angle. They have been placed on both sides of the frame to ensure good lighting during the acquisition phase of the video frames. The HW equipment is completed by a client/server architecture. The client is a Raspberry pi 3 board, which sends frames to the server that processes them to perform the signal analysis. In the current prototypical version of the system, the server is a desktop computer equipped with CPU Intel(R) Core(TM) i5-5200 2.20GHz 64 bit, 4GB RAM and 500GB hard disk.
The software architecture of the system includes a back-end and a
frontend module. The front-end module acquires the video frames through a camera
and sends them to the back-end module. The back-end module runs a face
tracker within the video frames and localizes the region of interest (ROI) useful
to estimate the vital signs. A pretrained frontal face detector is used to detect the
face within video frames, which is available with the library Dlib [
In order to evaluate the health condition, lip color has been used as an additional parameter. Normal people show lips with a pinkish nuance, while altered states or illness may provoke a modi cation of this color. Using image processing techniques applied to a speci c ROI acquired from an image of the patient's face, we automatically detect the lip color. ROI is processed after lips are detected and isolated. Then the dominant colour is classi ed as "regular", "altered" or "purplish". In order to quantify the lip color and detect the dominant color in the ROI, we apply a K-Means clustering with k = 3. Then, by means of K-Means algorithm, we obtain a histogram with the percentages of the three main colors expressed in RGB format. Finally, we convert the value of the dominant color into hexadecimal, associating it to a nominal label with the help of the library "N ameT hatColor"4. We developed a Fuzzy Inference System (FIS) with the help of the physician once all the parameters had been obtained. On the basis of the estimated vital signs, fuzzy rules have been de ned to support the diagnosis of cardiovascular disease by assessing a risk level. The linguistic input variables created are HR, BR, SpO2, LipsColor, which represent the measured vital signs. Furthermore, the output variable called RiskLevel represents the level of risk for cardiovascular disease. Besides, for each linguistic variable, we combined the linguistic terms with the relative fuzzy set. Finally, we obtained a total of 81 rules by de ning a rule for each combination of input and output fuzzy values. 4 http://chir.ag/projects/name-that-color/
We have conducted two experiments. In the rst experiment we involved healthy people, for a total of 25 participants (19 males, 6 females) from 18 to 65 years old and with varying skin colors. In the second experiment, we considered a sample of 10 subjects (5 females and 5 males) from 66 to 96 years old. All subjects were elderly people with cardiovascular diseases undergoing pharmacological treatment. During the experiments, HR and SpO2 values were collected simultaneously from our system and through a Finger tip Pulse Oximeter worn by the subjects. Then measurements from these two di erent sources were used for comparison. We considered the pulse oximeter for comparison because it is based on PPG that is the same principle underlying our device. A comparison with the use of ECG was avoided because it is based on a completely di erent approach.
In each test the subject was sitting in front of the mirror for 1 minute at a
distance of 50cm ca. from the integrated HD camera. In Table 1 we summarize
the statistics computed on the di erence between the pulse oximeter and our
device. The measurements obtained by our device are, in most cases, comparable to
those of the pulse oximeter. It can be seen that the error increases in case of
unhealthy people. This was due to the fact that most of unhealthy people involved
in the tests were elderly subjects who had di culties in standing still in front
of the device because of their neurological problems (for example, Parkinson's
disease [
all subjects healthy subjects unhealthy subjects HR 3:45 2:93 2:87 2:39 4:90 3:74
SpO2 1:83 2:43 1:54 1:76 2:56 3:63
We constructed a dataset by measuring and collecting the four vital signs of 116 people. In this way, the e ectiveness of the fuzzy rule-based system in simulating the decision of the expert has been tested. A second physician, different from the physician responsible for building the fuzzy knowledge base, has been involved to label the dataset, have a di erent expert opinion and make a more reliable validation process. Thanks to his knowledge and experience, the physician rst observed the measured life signs and then assigned them a risk label. The labelled dataset represents the ground truth for the evaluation of the diagnostic results obtained with the developed FIS. We have applied the inference of fuzzy rules to obtain a risk label (Low, Medium, High, or Very High) for each subject of the dataset. Risk class acc tnr tpr ppv npv tp tn fp fn
For each of the four output classes, we evaluated the accuracy together with additional measures that are commonly considered in classi cation tasks. In particular, while analyzing a single class c, we consider accuracy (acc), true positive (tp), true negative (tn), false positive (fp), and false negative (fn).
Moreover we considered: tp Positive Predictive Value: ppv = tp+fp Ratio of correctly classi ed samples w.r.t. those identi ed as pertaining to class c tn Negative Predictive Value: npv = tn+fn . Ratio of correctly classi ed samples w.r.t. those identi ed as not pertaining to class c
tp True Positive Rate: tpr = tp+fn . Ratio of samples correctly classi ed as belonging to class c w.r.t. those actually belonging to class c
tn True Negative Rate: tnr = fp+tn . Ratio of samples correctly classi ed as not belonging to class c w.r.t. those actually not belonging to class c The values of these measures evaluated for each class are reported in Table 2. It can be seen that in general the tnr and npv values are greater than those of tpr and ppv. This means that the fuzzy system is more e ective in determining the non-membership to each class than the sensitivity to each speci c class. This could be related to the unbalancement of the dataset. 4
One of the main research areas in the eld of biomedical engineering is the design of non-invasive and low-cost solutions for monitoring vital parameters. Our aim is to create easy-to-use accurate solutions that can be used both at home and in clinics. With our system, vital signs can be monitored at home in a comfortable way, without the need for additional invasive or even expensive medical devices. Our solution represents an innovative and smart object that can be of extremely useful in the eld of Personal Healthcare. It was intended to be used for daily personal monitoring of vital signs. This system is a proof-of-concept methodology that still needs re nement. Nevertheless, results of experiments have shown that it provides e ective measurements of vital signals as well as a reliable intelligent component based on fuzzy rules, which is able to simulate the expert physician decision. In our future works, we aim to improve our methodology through the acquisition of more data from ill people, and the integration of information such as demographic characteristics and patients and their family's history. We aim to carry out large-scale tests with patients su ering from cardiovascular diseases, through machine learning methods to automatically learn fuzzy rules from data.
The author wish to thank Prof. Giovanna Castellano for supervising his PhD research activity.