<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Visualising Convolutional Neural Network Decisions in Automated Sleep Scoring ?</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Fernando Andreotti</string-name>
          <email>fernando.andreotti@eng.ox.ac.uk</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Huy Phan</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Maarten De Vos</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Institute of Biomedical Engineering, University of Oxford</institution>
          ,
          <addr-line>Oxford</addr-line>
          ,
          <country country="UK">UK</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Current sleep medicine relies on the supervised analysis of polysomnographic recordings, which comprise amongst others electroencephalogram (EEG), electromyogram (EMG), and electrooculogram (EOG) signals. Convolutional neural networks (CNN) provide an interesting framework for automated sleep classi cation, however, the lack of interpretability of its results has hampered CNN's further use in medicine. In this study, we train a CNN using as input Continuous Wavelet transformed EEG, EOG and EMG recordings from a publicly available dataset. The network achieved a 10-fold cross-validation Cohen's Kappa score of = 0:71 0:01. Further, we provide insights on how this network classi es individual epochs of sleep using Guided Gradient-weighted Class Activation Maps (Guided Grad-CAM). The proposed approach is able to produce ne-grained activation maps on time-frequency domain for each signal providing a useful tool for identifying relevant features in CNNs.</p>
      </abstract>
      <kwd-group>
        <kwd>Convolutional Neural Networks</kwd>
        <kwd>Guided Backpropagation</kwd>
        <kwd>Polysomnography</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>
        Sleep is a fundamental biological process critical for the maintenance of physical
and mental health. Associations between sleep disruption and various
morbidities have been often reported, with some parainsomnias preceding serious neural
disorders by many years [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ]. Therefore, sleep monitoring is a matter of utmost
importance. Current clinical praxis heavily relies on the analysis of
polysomnographic (PSG) recordings, which include electroencephalogram (EEG),
electromyogram (EMG), and electrooculogram (EOG) amongst other physiological
signals. These signals are then interpreted based on clinical guidelines, such as
R&amp;K [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ] and the AASM [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ], which divide sleep into a few stages according
to speci c spectral content and characteristic waveform patterns (see Table 1).
Manual scoring following these rules is the gold-standard in sleep medicine.
      </p>
    </sec>
    <sec id="sec-2">
      <title>Stage</title>
      <p>Wake
N1
N2
N3
REM</p>
    </sec>
    <sec id="sec-3">
      <title>Description</title>
      <p>Presence of EEG alpha rhythm (8-12 Hz); EMG with high-amplitude
due to movements; EOG presents eye blinking artifacts, also visible on
EEG around 0:5 2 Hz.</p>
      <p>Attenuated alpha rhythm, presence of theta signal (4-7 Hz) on EEG;
Muscle tone and slow eye movements decrease on EMG and EOG.</p>
      <p>EEG presents K-complexes, i.e. large low frequency negative peaks in
the range &lt; 1:5 Hz, and sleep spindles, i.e. bursts of oscillations in the
sigma band (12-15 Hz).</p>
      <p>Slow wave activity for EEG (0.5-3 Hz), EMG tone is low and eye
movements unusual.</p>
      <p>Rapid eye movements (REM) clearly visible in EOG; low-amplitude
and mixed-frequency activity in EEG; muscle atonia on EMG.</p>
      <p>
        In contrast to manual scoring, automated approaches provide objective means
of classifying sleep stages. Traditional approaches make use of numerous
handengineered features from the physiological signals in combination with classical
machine learning methods, e.g. support vector machines or hidden Markov
models. Throughout the past decade, Convolutional Neural Networks (CNNs) have
been widely applied in elds such as computer vision and audio processing due
to their ability to operate on raw data, not requiring the explicit de nition of
features. In the last couple of years, some early works applying CNNs in the eld
of sleep analysis began to emerge [
        <xref ref-type="bibr" rid="ref1 ref12 ref19 ref22 ref3">22, 19, 3, 1, 12</xref>
        ]. Despite the competitive
performance achieved, due to the high degree of abstraction present in CNN hidden
layers, interpretability is an issue that has hindered its further use in medicine.
      </p>
      <p>
        Vilamala et al. [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ] proposed to visualise CNN decisions in sleep analysis
using sensitivity maps [
        <xref ref-type="bibr" rid="ref13 ref16">13, 16</xref>
        ]. For this purpose, the authors converted each sleep
epoch of single-lead EEG signals into spectrograms by using the Multitaper
Spectral Estimation. In order to convert spectra into RGB images (i.e.
containing three colour channels), the authors arti cially mapped the spectrogram
intensities using an arbitrary colourmap. This was performed so that pre-trained
image detection CNNs could be applied. The network of choice in [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ] was the
well-known VGGNet [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ], which contains 13 convolutional layers followed by
3 fully connected (FC) layers and 138 million parameters. VGGNet was
netuned to the task of sleep scoring using the Physionet Sleep-EDF Database [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ].
Despite presenting an interesting framework, [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ] has a few limitations. First,
the number and diversity of subjects available in the Sleep-EDF is limited to 20
young healthy subjects. Second, using a single EEG lead is restrictive as experts
classify sleep stages based on multichannel and multimodal settings. Third,
despite the abundance and availability of pre-trained image models, these networks
were trained for a very di erent task than the one at hand. As shown in [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ]
the transferability of features decreases as the distance between the base task
and target task increases. Moreover, these networks usually assume three colour
input channels. The workaround applied in [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ] to obtain RGB channels out of
a single spectrum provides no additional information to the network.
      </p>
      <p>
        In this work, we proposed a simple CNN architecture that is trained from
scratch using a large publicly available database. As input to the CNN we
provide EEG, EOG and EMG signals, which are standard in sleep analysis. For
visualising this network's weighs we apply the Guided Gradient-weighted Class
Activation Maps [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ]. This allows a detailed class-speci c view of the network
for each channel used as input, which may provide somnologists with additional
interpretation tools for sleep analysis.
2
      </p>
      <sec id="sec-3-1">
        <title>Data Material</title>
        <p>
          Data was obtained from the Montreal Archive of Sleep Studies database
(MASSDB) [
          <xref ref-type="bibr" rid="ref11">11</xref>
          ], a large publicly available dataset comprising single night PSG
recordings of 200 healthy participants ageing between 18 and 76 years, including 98
males aged 42:7 19:4 years and 102 females aged 38:1 18:9 years. The database
is divided into 5 cohorts all of which were used in this study. To avoid a
saturation on the number of needed channels in a recording setup we restrict our
setup to three channels commonly used in the speci c literature: a single central
EEG lead (C4-A1 or C3-A2, where available), a di erential EOG (ROC-LOC)
and/or EMG (CHIN1-CHIN2).
        </p>
        <p>As the MASS-DB comprises di erent study protocols, annotations using
R&amp;K were converted into AASM guidelines by assigning S3 and S4 stages to
N 3, while fS0; S1; S2g were relabelled as fW; N 1; N 2g, respectively. Three of
the MASS-DB cohorts contained 20s-epochs and were converted into 30s by
including 5s of signal before and after each segment. A total of 228,870 epochs are
available from the MASS-DB, being 13.6% W, 17.6% REM, 8.5% N1, 47.2% N2,
13.3% N3. To avoid biasing the network to either state, we undersampled each
subject recording to the minority class, which results into 59,848 samples.
3</p>
      </sec>
      <sec id="sec-3-2">
        <title>Methods</title>
        <p>
          Various CNN models have been proposed for the task of sleep stage classi cation,
most of which operate on raw single-channels (EEG or EOG). For instance, [
          <xref ref-type="bibr" rid="ref19">19</xref>
          ]
proposed a network with two branches of 4 convolutional layers with distinct
receptive elds aiming to generate feature maps with low and high frequency
content. In [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ], the authors propose a two-layer CNN model where after the
rst 1-dimensional convolution, lters are reshaped (stacked) and processed by
a 2D convolution. In [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ] a spatial ltering technique is applied to multiple EEG,
EOG and EMG channels. Each group of signals is treated in separate CNN
pipelines as images by applying two-layers of 2D convolutions. The study also
con rms that multiple channels and sensors provide an increase in detection
accuracy, similar results were obtained in [
          <xref ref-type="bibr" rid="ref1">1</xref>
          ].
        </p>
        <p>
          Sleep stages are largely de ned based on the spectral content of its signals
(see AASM de nitions in Table 1). Therefore, it is reasonable to make use of
time-frequency transforms as in [
          <xref ref-type="bibr" rid="ref23">23</xref>
          ]. Similarly to [
          <xref ref-type="bibr" rid="ref23">23</xref>
          ], the Short-Time Fourier
Transform (STFT) coupled with a compact CNN network was successfully
applied in [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ]. Time-frequency representations are also bene cial for visualisation
purposes, as one can observe time events at multiple frequency scales. Therefore,
in this study we aim at generating time-frequency transforms for each epoch
and modality of signal (EEG, EOG and EMG) as described in the following
Section 3.1. The proposed network is further explained in Section 3.2.
3.1
        </p>
        <sec id="sec-3-2-1">
          <title>Preprocessing</title>
          <p>
            All recordings were sampled at fs = 100 Hz and divided in 30 s epochs following
the AASM standard [
            <xref ref-type="bibr" rid="ref2">2</xref>
            ] for sleep scoring. Recordings were high-pass ltered using
zero-phase 100th order FIR lters with 0.1 Hz cuto frequency for EEG/EOG
signals and 10 Hz for EMG.
          </p>
          <p>
            As presented in [
            <xref ref-type="bibr" rid="ref21">21</xref>
            ], Wavelet transforms are particularly suitable for analysing
non-stationary signals (such as EEG), whereas STFT assumes periodicity and is
signicantly a ected its window size, which makes it fundamentally impossible to
correctly determine the onset of rapid events such as sleep spindles. Therefore,
based on [
            <xref ref-type="bibr" rid="ref21">21</xref>
            ], we apply the Continuous Wavelet Transform (CWT) with Morlet
basis function as time-frequency transform to each 30 s epoch using Matlab's
cwt() function with default parameters. The resulting scalograms (dimensions
101 (30 fs)) were further reduced by using bilinear interpolation to 30 300
maps. The resulting time-frequency representations were then normalised to the
range [0; 1], which serve as input to the proposed network.
3.2
          </p>
        </sec>
        <sec id="sec-3-2-2">
          <title>Network Architecture</title>
          <p>
            The proposed network is detailed in Fig. 1. As mentioned, the network assumes
as input scalograms for each channel, i.e. input dimensions 30 300 3. The rst
two convolutional layers use valid padding and convolutional strides of (1,2) to
reduce the input dimensions, thus decreasing the total number of parameters
in the following layers. The following block performs convolutions on 3 3, 5 5
and 7 7 windows in a similar fashion as the nave Inception block [
            <xref ref-type="bibr" rid="ref20">20</xref>
            ] and
in [
            <xref ref-type="bibr" rid="ref12">12</xref>
            ] (shown in Fig. 1). The main idea of this block is to learn features at
multiple temporal and frequency resolutions. At last, global average pooling is
performed to reduce the spatial dimensions of each channel. Global pooling is
directly followed by the softmax layer which outputs classes scores. The network
contains a total of 9.0M parameters.
          </p>
          <p>3x3
convolutions</p>
          <p>3x3
convolutions</p>
          <p>7x7
convolutions</p>
          <p>7x7
convolutions
Previous
Layer
5x5
convolutions</p>
          <p>5x5
convolutions</p>
          <p>Filter
concatenation</p>
          <p>2x2
maxpooling
Dropout
(50%)
(a)
Layer Type
Input
Convolution 2D
Batch Normalisation
Activation
Dropout
Convolution 2D
Batch Normalisation
Activation
Dropout
Inception-like Block
Concatenate
MaxPool 2D
Dropout
Global Average Pool
Dense
#Filters Operation Size Output Dim.</p>
          <p>
            - - (30,300,3)
64 (10 10) (21,146,64)
- - (21,146,64)
- ReLU (21,146,64)
- 50% (21,146,64)
128 (10 10) (12,69,128)
- - (12,69,128)
- ReLU (12,69,128)
- 50% (12,69,128)
256 (3 3),(5 5),(7 7) 3 (12,69,256)
- - (12,69,768)
- (2 2) (6,34,768)
- 50% (6,34,768)
- - (768)
5 Softmax 5
(b)
The proposed model was developed in Keras/Tensor ow. Training was performed
in 100 epochs, with batch size 64 and categorical cross entropy as loss function.
Adam[
            <xref ref-type="bibr" rid="ref9">9</xref>
            ] optimiser was used with default parameters (learning rate lr = 10 3,
1 = 0:9 and 2 = 0:999). L2 norm regularisation was applied to the last layer
with value = 10 5. To assess the performance of this network, a 10-fold
crossvalidation procedure was carried out on the balanced MASS-DB. Results are
evaluated in terms of sensitivity (SE), positive predictive value (PPV), accuracy
(ACC), F1-score (F1) and Cohen's Kappa score ( ).
3.4
          </p>
        </sec>
        <sec id="sec-3-2-3">
          <title>Visualising CNNs</title>
          <p>Deep visualisation is an active eld of research. Current methods for
visualising CNN weights with 2-dimensional data input (e.g. images or time-frequency
representations) can be divided into gradient-based and activation maximisation.</p>
          <p>
            Gradient-based methods aim at highlighting pixels of an input image I that
have the most impact on the prediction score Sc for a given class c. In order to
assess the rate of change in Sc with respect to small changes in image intensity,
[
            <xref ref-type="bibr" rid="ref16">16</xref>
            ] proposed backpropagating the partial derivatives @Sc=@I across the network.
Therefore positive values on the gradient indicate an increase on the output score
for that class. As the class score function is highly non-linear function of image I,
rst-order Taylor expansion is used to linearize this function [
            <xref ref-type="bibr" rid="ref16">16</xref>
            ]. Extensions of
the backpropagation method focus on modifying to the original gradient function
and result in qualitative improvements to visualisation [
            <xref ref-type="bibr" rid="ref15">15</xref>
            ]. Example of such
methods include the Deconvolutional Networks [
            <xref ref-type="bibr" rid="ref27">27</xref>
            ], that modi es the backward
pass of ReLU by clipping negative gradients, and Guided Backpropagation [
            <xref ref-type="bibr" rid="ref10 ref18">10,
18</xref>
            ], on which activations are masked during both deconvolution and forward
pass. This group of techniques can be used for visualising the last layer [
            <xref ref-type="bibr" rid="ref16">16</xref>
            ], or
on each of the hidden neurons [
            <xref ref-type="bibr" rid="ref26 ref4">4, 26</xref>
            ]. These approaches are attractive due to their
simplicity, however despite producing ne-grained visualisations, these methods
are not class-discriminative as optimisation process tends to produce images that
are hardly recognisable [
            <xref ref-type="bibr" rid="ref15">15</xref>
            ]. In order to produce more natural-looking images,
some studies suggested biasing optimisation with natural image priors [
            <xref ref-type="bibr" rid="ref10">10</xref>
            ] or
using regularisation techniques [
            <xref ref-type="bibr" rid="ref26">26</xref>
            ].
          </p>
          <p>
            Activation maximisation methods focus on directly visualising the activation
of some speci c layer of a network given an input image. One approach is to
estimate the importance of input pixels by visualising the probability of the
correct class being chosen as a function of a mask occluding parts of the image,
the so-called occlusion/perturbation sensitivity [
            <xref ref-type="bibr" rid="ref23 ref27 ref5">27, 5, 23</xref>
            ]. Another approach is
to focus on the activation of the last layer before any FC layer, where
higherlevel visual concepts are captured. An example of such approach is are Class
Activation Maps (CAMs) [
            <xref ref-type="bibr" rid="ref28">28</xref>
            ]. In order to retain the tensorial shape any
attening operation is substituted by global average pooling followed directly by
the softmax (i.e. disregarding any FC layers). Based on the class scores Sc, wc
k
corresponding weights to class c for unit k, and Akxy = Px;y fk(x; y) activation
map of unit k in the last layer a class-speci c heatmap Lc can be achieved for
the image such as [
            <xref ref-type="bibr" rid="ref28">28</xref>
            ]:
          </p>
          <p>
            Selvaraju et al. [
            <xref ref-type="bibr" rid="ref15">15</xref>
            ] proposed a generalisation of CAM by introducing the
main concept of backpropagation from gradient-based approaches downstream
from any Ak, i.e.
          </p>
          <p>c
LCAM (x; y) =</p>
          <p>X wkcAkxy
k
kc =
where kc is the partial linearization of the network from feature map Ak and
Z1 Px Py represents the global average pooling operation perform over these
feature maps. The Gradient-weighted CAM (Grad-CAM) then performs a weighted
combination of forward activation maps followed by a ReLU:</p>
          <p>c
LGrad CAM (x; y) = ReLU</p>
          <p>X
k
kcAkxy</p>
          <p>
            Grad-CAM enjoys the bene ts from CAMs and produce class-discriminative
localisation of image regions, while imposing fewer restrictions on the network's
architecture. In fact, it can be used in any CNN-based architecture. However,
Grad-CAM only produces an averaged heatmap for the image so that the
channel information is lost. In order to show ne-grained gradient visualisations,
[
            <xref ref-type="bibr" rid="ref15">15</xref>
            ] further proposed combining Guided Backpropagation and Grad-CAM by
(1)
(2)
(3)
point-wise multiplication of both bi-linear interpolated Grad-CAM and Guided
Backpropagation heatmaps.
          </p>
          <p>In this contribution, we aim at better understanding how the network
classi es sleep stages regarding each signal modality (i.e. EEG, EOG and EMG).
For this purpose, after pre-training the proposed CNN presented in Section 3.2,
Guided Grad-CAM is applied to randomly selected segments of the MASS-DB.
We qualitatively evaluate these Wavelet scalograms and heatmaps regarding
their physiological meaning.
4</p>
        </sec>
      </sec>
      <sec id="sec-3-3">
        <title>Results and Discussion</title>
        <p>
          The results for the 10-fold cross-validation procedure using the proposed model
and balanced MASS-DB set are described in Fig. 2. Despite having less than 10%
the number of parameters of VGGNet, the proposed CNN is able of performing
automated sleep staging when trained from scratch, resulting Kappa score of
= 0:71 0:01. As usual in the literature, N1 stage classi cation performed
worst as the state shares characteristics with wakefulness state. Moreover, the
transition from wake to N1 is described as challenging and has been reported
as a major source of inter-rater variability [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ]. Our analysis di er from the one
presented in [
          <xref ref-type="bibr" rid="ref23">23</xref>
          ] in that the database used in this study is larger and the di erent
classes have been balanced. Similar results for the imbalanced MASS-DB were
obtained in [
          <xref ref-type="bibr" rid="ref19">19</xref>
          ].
        </p>
        <p>
          Figures 3 and 4 demonstrate the application of Guided Backpropagation and
Guided Grad-CAM using the proposed trained model to segments of Wake, REM
and N2 stages respectively. These stages were chosen for illustrating the
visualisation method, due to their more distinguishable characteristics (e.g. presence
of spindles/K-complexes on N2). A total of 100 randomly selected epochs from
all cohorts and subjects were visually inspected from which Figs. 3 and 4 were
selected. Overall Guided Grad-CAM produced cleaner heatmaps than Guided
Backpropagation as described in [
          <xref ref-type="bibr" rid="ref15">15</xref>
          ]. More importantly, Guided Grad-CAM
produces channel-speci c maps, which are relevant when analysing multimodal
data such as PSG recordings.
        </p>
        <p>
          The presented results show that relevant CNN feature maps (i.e. weights)
often correspond to regions of interest for particular sleep stages. For instance
Fig. 3(a) presents high sensitivity around EEG alpha bands (i.e. 8-12 Hz), which
is characteristic for both Wake stage. During the wake stage, the patterns for
EOG and EMG seem erratic spreading along various frequency bands (Fig. 3(a))
whereas during REM (Fig. 3(b)) weighs lower frequent EOG bands. Usually the
N2 stage is the mostly recognisable one, comprising K-complexes (negative peaks
followed by positive peaks with duration &gt; 0.5 s) and sleep spindles (bursts of
oscillatory waves in the sigma band, i.e. 11-15 Hz). Comparing Fig. 4(a) and
(b) we notice that the Guided Grad-CAM weighs heavily such patterns. These
conclusions are similar to the ones found in [
          <xref ref-type="bibr" rid="ref23">23</xref>
          ], except we notice coherent
physiological information across the di erent input channels used in this study.
        </p>
        <p>
          These exemplary epochs demonstrate the potential of such visualisation
approaches for further interpreting how CNN weighs di erent sleep patterns and
could greatly enhance the interaction of domain experts [
          <xref ref-type="bibr" rid="ref23">23</xref>
          ]. Nevertheless,
conclusions must be carefully drawn as the method only provide a partial
understanding of the network (e.g. not including FC layers). Moreover, as more
aesthetically pleasing notions of image saliency are sought, di erent
backpropagation heuristics (e.g. regularisation) are applied [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ]. Further limitations of this
method include i) epoch-speci city i.e. output depends on input
representation/epoch; ii) the visualisation tool is not model-agnostic as each network will
inevitably produce di erent outcomes. In this study we applied data from healthy
subjects, future works should focus on comparing these results with feature maps
generated from pathological data that may provide clinically relevant insights.
Additionally, PSG analysis often takes into consideration surrounding epochs.
The proposed CNN model treats each epoch independently, however, it is
bene cial to consider the temporal/transition information contained in this signal.
This can achieved by using Recurrent Neural Networks or soft-attention
techniques.
5
        </p>
      </sec>
      <sec id="sec-3-4">
        <title>Conclusion</title>
        <p>In this contribution we shed a light onto how CNNs are able to distinguish sleep
stages. For this purpose we trained a small CNN network from scratch which
takes as input CWT scalograms from 3 di erent sensors (EEG, EOG and EMG).
Further, we visualise regions of interest for the trained network by applying the
Guided Grad-CAM method. The proposed approach is able to produce
negrained activation maps on time-frequency representations of each individual
signal providing a useful tool for identifying relevant features in CNNs.
)
z
)
z
EEG Scalogram</p>
        <p>EOG Scalogram
alpha rhythm (8-12 Hz) and EOG erratic content in (a). In (b) EEG contains
mixedfrequencies, low frequency EOG is considered relevant but not EMG.
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
)
z
(b) Respective time signal with characteristic spindles and K-complexes.
network seems to detect both K-complexes (low frequency content) and sleep spindels
(11-13 Hz).</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Andreotti</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Phan</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cooray</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lo</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hu</surname>
            ,
            <given-names>M.T.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>De Vos</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>Multichannel Sleep Stage Classi cation and Transfer Learning using Convolutional Neural Networks</article-title>
          .
          <source>In: IEEE Engineering in Medicine and Biology Conference (EMBC)</source>
          (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Berry</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Brooks</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gamaldo</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Harding</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lloyd</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Marcus</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Vaughn</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          :
          <article-title>The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Speci cations</article-title>
          . American Academy of Sleep Medicine (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Chambon</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Galtier</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Arnal</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wainrib</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gramfort</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>A deep learning architecture for temporal sleep stage classi cation using multivariate and multimodal time series</article-title>
          . arXiv:
          <volume>1707</volume>
          .03321 pp.
          <volume>1</volume>
          {
          <issue>14</issue>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Erhan</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bengio</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Courville</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Vincent</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Visualizing higher-layer features of a deep network</article-title>
          .
          <source>Tech. Rep. 1341</source>
          , University of Montreal (
          <year>2009</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Fong</surname>
            ,
            <given-names>R.C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Vedaldi</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Interpretable Explanations of Black Boxes by Meaningful Perturbation</article-title>
          .
          <source>In: IEEE International Conference on Computer Vision</source>
          (ICCV). pp.
          <volume>3449</volume>
          {
          <issue>3457</issue>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Heidi</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Peter</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Josef</surname>
            ,
            <given-names>Z.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Marion</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hans</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Georg</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Esther</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Erna</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Doris</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Silvia</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bernd</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Andrea</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Georg</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>Interrater reliability for sleep scoring according to the rechtscha en &amp; kales and the new aasm standard</article-title>
          .
          <source>J. Sleep Res</source>
          .
          <volume>18</volume>
          (
          <issue>1</issue>
          ),
          <volume>74</volume>
          {
          <fpage>84</fpage>
          (
          <year>2009</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7. Io e, S.,
          <string-name>
            <surname>Szegedy</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          : Batch Normalization:
          <article-title>Accelerating Deep Network Training by Reducing Internal Covariate Shift</article-title>
          . arXiv:
          <volume>1502</volume>
          .03167 (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Kemp</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zwinderman</surname>
            ,
            <given-names>A.H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tuk</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kamphuisen</surname>
            ,
            <given-names>H.A.C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Oberye</surname>
            ,
            <given-names>J.J.L.</given-names>
          </string-name>
          :
          <article-title>Analysis of a sleep-dependent neuronal feedback loop: The slow-wave microcontinuity of the EEG</article-title>
          .
          <source>IEEE Trans. Biomed. Eng</source>
          .
          <volume>47</volume>
          (
          <issue>9</issue>
          ),
          <volume>1185</volume>
          {
          <fpage>1194</fpage>
          (
          <year>2000</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Kingma</surname>
            ,
            <given-names>D.P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ba</surname>
            ,
            <given-names>J.L.</given-names>
          </string-name>
          :
          <article-title>Adam: a Method for Stochastic Optimization</article-title>
          .
          <source>International Conference on Learning Representations</source>
          <year>2015</year>
          pp.
          <volume>1</volume>
          {
          <issue>15</issue>
          (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Mahendran</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Vedaldi</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Understanding Deep Image Representations by Inverting Them</article-title>
          . arXiv:
          <volume>1412</volume>
          .0035 p.
          <year>2014</year>
          (
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <given-names>O</given-names>
            <surname>'Reilly</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            ,
            <surname>Gosselin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            ,
            <surname>Carrier</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            ,
            <surname>Nielsen</surname>
          </string-name>
          ,
          <string-name>
            <surname>T.</surname>
          </string-name>
          :
          <article-title>Montreal Archive of Sleep Studies: an open-access resource for instrument benchmarking and exploratory research</article-title>
          .
          <source>J. Sleep Res</source>
          .
          <volume>23</volume>
          (
          <issue>6</issue>
          ),
          <volume>628</volume>
          {635 (dec
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Phan</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Andreotti</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cooray</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chen</surname>
            ,
            <given-names>O.Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>De Vos</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>DNN Filter Bank Improves 1-Max Pooling CNN for Automatic Sleep Stage Classi cation</article-title>
          .
          <source>In: IEEE Engineering in Medicine and Biology Conference (EMBC)</source>
          (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Rasmussen</surname>
            ,
            <given-names>P.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Madsen</surname>
            ,
            <given-names>K.H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lund</surname>
            ,
            <given-names>T.E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hansen</surname>
            ,
            <given-names>L.K.</given-names>
          </string-name>
          :
          <article-title>Visualization of nonlinear kernel models in neuroimaging by sensitivity maps</article-title>
          .
          <source>NeuroImage</source>
          <volume>55</volume>
          (
          <issue>3</issue>
          ),
          <volume>1120</volume>
          {
          <fpage>1131</fpage>
          (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Rechtscha</surname>
            <given-names>en</given-names>
          </string-name>
          , A.,
          <string-name>
            <surname>Kales</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>A manual of standardized terminology, techniques and scoring system for sleep stages of human subjects</article-title>
          .
          <source>Tech. rep., National Institutes of Health publication, ; no. 204</source>
          (
          <year>1968</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Selvaraju</surname>
            ,
            <given-names>R.R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cogswell</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Das</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Vedantam</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Parikh</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Batra</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization</article-title>
          .
          <source>In: IEEE International Conference on Computer Vision</source>
          (ICCV). pp.
          <volume>618</volume>
          {
          <issue>626</issue>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Simonyan</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Vedaldi</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zisserman</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Deep Inside Convolutional Networks: Visualising Image Classi cation Models and Saliency Maps</article-title>
          . arXiv:
          <volume>1312</volume>
          .6034 pp.
          <volume>1</volume>
          {
          <issue>8</issue>
          (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Simonyan</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zisserman</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Very Deep Convolutional Networks for Large-Scale Image Recognition</article-title>
          .
          <source>arXiv:1409.1556</source>
          (
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Springenberg</surname>
            ,
            <given-names>J.T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dosovitskiy</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Brox</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Riedmiller</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>Striving for Simplicity: The All Convolutional Net</article-title>
          .
          <source>In: International Conference on Learning Representations (ICLR)</source>
          (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Supratak</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dong</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wu</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Guo</surname>
          </string-name>
          , Y.:
          <article-title>DeepSleepNet: A Model for Automatic Sleep Stage Scoring Based on Raw Single-Channel EEG</article-title>
          .
          <source>IEEE Trans. Neural Syst. Rehabil. Eng</source>
          .
          <volume>25</volume>
          (
          <issue>11</issue>
          ),
          <year>1998</year>
          {2008 (nov
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Szegedy</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wei</surname>
            <given-names>Liu</given-names>
          </string-name>
          , Yangqing Jia, Sermanet,
          <string-name>
            <given-names>P.</given-names>
            ,
            <surname>Reed</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            ,
            <surname>Anguelov</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            ,
            <surname>Erhan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            ,
            <surname>Vanhoucke</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            ,
            <surname>Rabinovich</surname>
          </string-name>
          ,
          <string-name>
            <surname>A.</surname>
          </string-name>
          :
          <article-title>Going deeper with convolutions</article-title>
          .
          <source>In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)</source>
          (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <surname>Tsanas</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , Cli ord, G.D.:
          <article-title>Stage-independent, single lead EEG sleep spindle detection using the continuous wavelet transform and local weighted smoothing</article-title>
          .
          <source>Front. Hum. Neurosci</source>
          .
          <volume>9</volume>
          (
          <issue>April</issue>
          ),
          <volume>1</volume>
          {
          <fpage>15</fpage>
          (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          22.
          <string-name>
            <surname>Tsinalis</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Matthews</surname>
            ,
            <given-names>P.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Guo</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zafeiriou</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          :
          <article-title>Automatic Sleep Stage Scoring with Single-Channel EEG Using Convolutional Neural Networks</article-title>
          .
          <source>arXiv:1610</source>
          .1683 p.
          <volume>12</volume>
          (oct
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          23.
          <string-name>
            <surname>Vilamala</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Madsen</surname>
            ,
            <given-names>K.H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hansen</surname>
            ,
            <given-names>L.K.</given-names>
          </string-name>
          :
          <article-title>Deep Convolutional Neural Networks for Interpretable Analysis of EEG Sleep Stage Scoring</article-title>
          .
          <source>arXiv 1710.00633</source>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          24.
          <string-name>
            <surname>Wul</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gatti</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wettstein</surname>
            ,
            <given-names>J.G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Foster</surname>
          </string-name>
          , R.G.:
          <article-title>Sleep and circadian rhythm disruption in psychiatric and neurodegenerative disease</article-title>
          .
          <source>Nat. Rev. Neurosci</source>
          .
          <volume>11</volume>
          (
          <issue>8</issue>
          ),
          <volume>589</volume>
          {
          <fpage>599</fpage>
          (
          <year>2010</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          25.
          <string-name>
            <surname>Yosinski</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Clune</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bengio</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lipson</surname>
          </string-name>
          , H.:
          <article-title>How transferable are features in deep neural networks? In: NIPS (</article-title>
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          26.
          <string-name>
            <surname>Yosinski</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Clune</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Nguyen</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fuchs</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lipson</surname>
          </string-name>
          , H.:
          <article-title>Understanding Neural Networks Through Deep Visualization</article-title>
          . arXiv:
          <volume>1506</volume>
          .06579 (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          27.
          <string-name>
            <surname>Zeiler</surname>
            ,
            <given-names>M.D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fergus</surname>
          </string-name>
          , R.:
          <article-title>Visualizing and Understanding Convolutional Networks</article-title>
          .
          <source>In: European Conference on Computer Vision (ECCV)</source>
          . pp.
          <volume>818</volume>
          {
          <issue>833</issue>
          (
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          28.
          <string-name>
            <surname>Zhou</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Khosla</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lapedriza</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Oliva</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Torralba</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Learning Deep Features for Discriminative Localization</article-title>
          .
          <source>In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)</source>
          . pp.
          <volume>2921</volume>
          {
          <issue>2929</issue>
          (jun
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>