Narcolepsy with cataplexy is a severe lifelong disorder characterized, among the others, by the sudden loss of bilateral face muscle tone triggered by emotions (cataplexy). In this extended abstract, we present two methodologies for the automatic analysis of patients' videos able to assist neurologists in diagnosing the disease and/or detecting attacks. Indeed, recent findings demonstrated that the detection of abnormal motor behaviors in video recordings of patients undergoing emotional stimulation is efective in characterizing the disease symptoms. Such motor behaviors (ptosis, mouth opening, head drop) are however to be discovered by neurologists through manual inspection of patients' videos. Automatic content-based video analysis is clearly of immediate help here. Experimental results conducted on real data support the efectiveness of the presented automated techniques.
Narcolepsy with cataplexy is a rare disorder mainly arising in young adults/children
characterized by daytime sleepiness, sudden loss of muscle tone while awake triggered by emotional
stimuli (cataplexy), hallucinations, sleep paralysis, and disturbed nocturnal sleep [
It is evident that a tool able to detect the “correct” facial expression changes (i.e., the disease
symptoms) from video recordings of patients would be able to automatically identify the
presence of the disease. This could be extremely helpful, not only to support neurologists in
diagnosing the disease, but also in monitoring everyday activities in a non-invasive way to
provide early warnings in the event of the insurgence of a crisis. Indeed, it is well known that
the synergistic use of Machine Learning (ML) techniques can help in alleviating the burden
of the medical specialist in analyzing patient data, thus improving diagnostic consistency and
accuracy [
In [
In this extended abstract, we report details on the video analyzer for the automatic detection
of cataplexy symptoms. This component of the CAT-CAD system is built on top of SHIATSU, a
general and extensible framework for video retrieval which is based on the (semi-)automatic
hierarchical semantic annotation of videos exploiting the analysis of their visual content [
After reviewing some background information, we detail the methodologies used to automatically analyze videos: a pattern-based technique able to recognize facial patterns and a novel approach based on convolutional neural networks (Section 2). Finally, we provide results obtained from an extensive experimental evaluation to compare the performance of the two video analysis approaches, using a benchmark containing recordings from real patients (Section 3) and conclude (Section 4).
Narcolepsy with cataplexy usually arises in adolescence or young adulthood, but the diagnosis
is typically established after a long period with a mean delay (across Europe) from symptom
onset to diagnosis of 14 years [
Few scientific studies have considered the video-polygraphic features of cataplexy in adult
age and only recently the motor phenotype of childhood cataplexy has been described
exclusively using video recordings of the attacks evoked by watching funny cartoons [
To the best of our knowledge, CAT-CAD is the first study about the automatic recognition
of cataplexy by exploiting patient video recordings. However, automatic detection of facial
motor phenomena similar to the ones used to diagnose cataplexy is commonly used in other
contexts. For example, the detection of eyelid closure, head pose, or mouth opening is useful
for the automatic recognition of fatigue/drowsiness in vehicle drivers [
The core of the CAT-CAD tool is the real-time analysis of patients’ videos to detect the presence of disease symptoms (i.e., ptosis, head drop, and smile/mouth opening). Two diferent approaches were developed to perform video analysis, with the idea of comparing their relative performance and possibly combining them to achieve the best possible result in recognizing the diferent motor phenomena: • The Pattern-Based approach (Section 2.1) is built on the automatic detection of facial features in video frames. • The Deep Learning approach (Section 2.2) exploits three convolutional neural networks, each trained to detect a specific motor phenomenon.
The first video analyzer to be implemented in CAT-CAD for the detection of cataplexy motor
phenomena exploits facial landmarks, as detected by OpenFace [
The alternative video analysis tool is based on convolutional neural networks (CNNs). The
CNN architecture used in this work is based on the DeXpression Network [
Our CNN architecture consists of three diferent types of blocks:
1. an Input Block, which performs image pre-processing,
2. a Feature Extraction Block, inspired by the architectural principles introduced by GoogleNet [
We have trained three diferent networks, one for each motor phenomenon to be recognized. The three networks share the same architecture, but the learned weights are clearly diferent, due to the use of diferent training classes. It is clear that, for this approach, each frame is analyzed per se, and no considerations on frame sequences, like duration of eyelids closing or of head drop, can be extrapolated, contrary to the pattern-based approach. The three neural networks have been trained for 8 epochs each, for a total time of about 12 hours.
For each video frame, the face of the patient is first detected by means of OpenFace and then cropped. The resulting image is converted to grayscale and downsized to produce images of 320 × 320 pixels. Cropping of the images was necessary in order to provide the CNN with only face details (thus avoiding that the surrounding environment would distract the learning).
The benchmark used for our experimental evaluation consists of a population of patients of
the Outpatient Clinic for Narcolepsy of the University of Bologna, who were assessed for the
presence of cataplexy by way of a neurophysiological and biological diagnosis [
The first (experimental) group of patients includes 14 subjects displaying symptoms of the disease. Training of video analyzers has been performed using an inter-patient separation scheme, where patients have been randomly assigned to non-overlapping training and test sets, by respecting sex and age distribution. In particular, 11 patients have been included in the training set (thus, their entire videos have been used to train each analyzer), while the remaining 3 patients have been exploited to test the accuracy of the tool.
The second group includes 44 diferent subjects that show no sign of the disease. Among those, 14 patients have been selected as a control group so as to follow the same sex and age distribution of the experimental group.
For the deep-learning approach, data augmentation was performed by adding, to each training set frame, seven additional images by performing: (i) 3 rotations with a random angle between − 45∘ and +45∘ , (ii) 3 translations with a random shift between -50 and 50 pixels, and (iii) 1 horizontal flipping . The final training sets consists of 191140 labeled images for ptosis, 61216 labeled images for head-drop, and 108196 labeled images for mouth opening.
To objectively evaluate the performance of our analyzers, each frame can be labeled according to a confusion matrix as correctly and incorrectly recognized for each of the two classes available (in our case, motor phenomenon actually present or absent). The four possible outcomes are (true positive, a frame where the symptom is correctly detected as present), (false negative, symptom wrongly not detected), (true negative, symptom correctly not detected) and (false positive, symptom wrongly detected as present). From the confusion matrix, the performance measures used in our experiments are defined as follows.
Recall/Sensitivity () is defined as the fraction of the frames showing signs of the disease (positives) that are correctly identified: = + . is therefore used to measure the accuracy of a technique in recognizing the presence of the disease.
Specificity () is the fraction of frames not showing the disease (negatives) that are correctly classified: = + . thus expresses the ability of a technique to avoid false alarms (which can lead to expensive/invasive exams).
Precision ( ) is another popular metric, besides and , which are the fundamental prevalenceindependent statistics. is defined as the fraction of correct positively classified frames and assesses the predictive power of the classifier: = + .
Accuracy () measures the fraction of correct decisions, to assess the overall efectiveness of + the algorithm: = +++ .
Balanced Score (1) is a commonly used measure, combining and in a single metric 1 2· computed as their harmonic mean: 1 = 2 1 + 1 = 2· ++ .
For the pattern-based approach, thresholds used for the detection of ptosis, head drop, and
mouth opening were chosen as the ones providing the best classifying performance on the test
set [
• The pattern-based approach lead to significantly superior results with respect to its deep learning counterpart. In particular, for cataplectic subjects the former attains the best performance in 85% of the metrics (17 out of 4 × 5 = 20 performance measures). • When considering specific motor phenomena, the pattern-based approach consistently outperforms the deep learning approach in detecting ptosis, while the latter sports superior measures only for specificity and precision in detecting mouth opening and for recall in head drop detection. • The superior specificity of the pattern-based technique is confirmed in non-cataplectic subjects, with an overall specificity at 98%.
A possible explanation for the inferior performance of the deep learning approach is the fact that such approach cannot discriminate between quick and long eye blinks/head drops, due to the fact that each frame is analyzed individually by the CNN. It is therefore likely that the higher number of false positives is because the CNN wrongly detects “regular” eye blinks or head movements as ptosis or head drop.
For the case of non-cataplectic subjects, it is interesting to note that the performance of the deep learning approach for the overall detection of cataplexy is sensibly worse than those attained for the single motor phenomena. For such patients, false positives for ptosis, head drop, and mouth opening are present in diferent frames. Indeed, due to the absence of positive cases, the set of false positive frames for the overall cataplexy coincides with the union of frames wrongly classified by any specific motor phenomenon detector.
Finally, we include a brief discussion about eficiency of the proposed techniques. On our experimental setup, which involved a commodity (low-end) machine, we were able to extract EAR, CoG and MAR descriptors in real-time for each video frame. Clearly, this is the more time consuming operation for the pattern-based approach, thus it is proven that the whole process of automatic detection can be performed on-line during a single emotional stimulated video recording session. On the other hand, our current implementation of the deep learning approach is only capable to obtain a throughput of 18.5 frame/s, thus being unable to attain real-time performance (recall that the frame rate of videos is 30 frame/s). The reason for this measure is the following: when analyzing a single frame, about 50% of the time is spent in detecting the position of the patient face, about 25% in cropping the image (retaining only the face), and 25% for the classification of the frame by the three neural networks. The bottleneck of the whole computation is clearly the face detection phase, which we implemented using the OpenFace library, instead of using other faster methods (such as the well-known Haar-Cascade iflter). This choice was carried out starting from the consideration that quicker filters often fail to identify the face within the image, especially in videos with excessive head movement, which is the common case for cataplectic subjects.
In this extended abstract, we reported details on the video analyzer of CAT-CAD for the automatic detection of cataplexy symptoms (ptosis, head drop, and smile/mouth opening). Two diferent approaches are introduced for the detection of disease symptoms: the Pattern-Based approach is based on analysis of facial features, using the OpenFace framework, while the Deep Learning approach uses CNNs, as implemented by TensorFlow. An extensive comparative experimental evaluation conducted on a benchmark of real patients recordings demonstrated the accuracy of the proposed techniques. When comparing the efectiveness of the two video analyzers we introduced to detect cataplexy symptoms, the pattern-based approach achieves superior performance. One of the possible explanations for the inferior detection accuracy of the deep learning approach is the fact that 2D CNNs are unable to properly take into account the temporal dimension that correlates subsequent frames in a video. The use of 3D CNNs could be an interesting way to pursue, and we plan to consider their inclusion in CAT-CAD.