<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Risk Estimator using a Multi-Layer Perceptron Network for Coronary Artery Disease Prevention</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Didi Liliana Popa</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Mihai Lucian Mocanu</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Radu Teodoru Popa</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Universitatea din Craiova, Facultatea de Automatică</institution>
          ,
          <addr-line>Calcultoare și Electronică, Bulevardul Decebal,nr 107, Craiova</addr-line>
          ,
          <country country="RO">România</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>One of the most prevalent heart disease is coronary artery disease (CAD). We propose the use of Deep Learning (DL) Network Multi-Layer Perceptron (MLP) in order to obtain an early cardiovascular risk estimation at 10 year for CAD prevention in patients with the purpose of reduced rate of mistreatment.For this purpose, we designed a protocol for selecting eloquent data. We also designed a method which is using Deep Neural Network sequential model which has multiple inputs and three outputs. Data set are from a private clinic in South- West zone in Romania. Custom data set included a batch of 784 patients with 11 medical characteristics. The result of predicting the MLP network gives us the probability that the patient will develop a severe heart disease in the following 10 years. By deploying a DL network, we were able to provide an unitary risk assessment method of CAD for physicians that allowed the “localization” of the medical European Society of Cardiology guidelines to Romania region.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;Coronary artery disease</kwd>
        <kwd>deep neural network</kwd>
        <kwd>multilayer perceptron network</kwd>
        <kwd>cardiovascular risk estimator</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>The importance of of early diagnosis and
risk stratification of ischemic heart diseases is
given by the fact that cardiovascular diseases is
the leading cause of death in Europe. [eurostat
- causes of death statistics 2019], and in the
same time in the world [World Health
Organization]. Among them, the most
prevalent manifestation is ischemic heart
disease given by coronary atherosclerosis
pathology, which is associated with an
increased mortality and morbidity rate.</p>
      <p>Coronary artery disease is caused by
cholesterol deposits that stick and narrow the
walls of coronary arteries that supply blood to
the heart.</p>
      <p>
        Clinical presentation of ischemic heart
disease includes silent ischemia, stable
coronary artery disease, unstable angina,
myocardial infarction, heart failure, and sudden
cardiac death[
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. In Europe, the
recommendations for treating cardiac diseases
a1re described in the Guidelines of the
European Society of Cardiology
[www.escardio.org].
      </p>
      <p>Those are covering a minimum of
investigations that should be done to patients
with coronary heart disease such as laboratory
examinations (bio-markers, lipid profile,
NTProBNP, D-Dimers), 12-lead
electrocardiogram, the ECG and imaging effort
test, echocardiography, coronarography and
describes the cardiac risk scores that should be
performed, but it leaves to the physician's
discretion how these protocols will be
implemented.</p>
      <p>The diagnosis and cardiovascular risk
assessment of stable coronary artery disease
(SCAD) involves clinical evaluation, including
identifying significant dyslipidemia,
hyperglycaemia or other biochemical risk
factors and specific cardiac investigations such
as stress testing or coronary imaging. These
investigations may be used to confirm the
diagnosis of ischemia in patients with suspected
SCAD, to identify or exclude associated
conditions or precipitating factors, assist in
stratifying risk associated with the disease and
to evaluate the efficacy of treatment</p>
      <p>Conventional risk factors for the
development of SCAD are hypertension,
hypercholesterolemia, diabetes,sedentary
lifestyle, obesity,smoking and a family history.</p>
      <p>
        Taking into consideration the fact that
cardiac diseases have remained the leading
causes of death globally in the last 15 years [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ],
there is need for a better strategy in improving
the diagnostic and treatment.
      </p>
      <p>Artificial Intelligence can help in order to
have an early diagnosis and more accurate and
also can reduce the rate of misdiagnosis. That
leads to a decrease in mortality rate. In order to
achieve this, is necessary to customized
healthcare for each individual patient.</p>
      <p>The cardiac risk scores used in traditional
medicine are calculated on a generalized
population at a very large level, and doesn’t
allow localized medicine with particularities
from each zone. Neural networks can do
customized healthcare, because they learn and
so the cardiac risk scores is improved.</p>
      <p>
        AI refers to those programs that computers
may execute similar to human intelligence ,
learning and solving problems. The neural
network are simulating the way that human
brain is interacting in the learning process.
Deep learning (DNN) is formulated as a
mathematical neural network architecture
consisting of multiple hidden layers with
nonlinear activation.[
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] One architecture of DNN is
Multilayer perceptron (MLP), in which every
element of a previous layer, is connected to
every element of the next layer and has an
activation function at each hidden layer.[
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]
      </p>
      <p>In literature, there are different methods in
medical research for SCAD classification
using different learning and data mining
techniques , like neural network (NN), support
vector machine, random forest, decision tree,
clustering, and Gaussian mixture model and
others.</p>
      <p>The purpose of this model was to obtain an
early diagnosis of CAD with a good accuracy ,
that can be used in clinical practice for
diagnosis of SCAD, using deep learning
methods for combining results of clinical
examination and other attributes recorded from
the patients.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Methods</title>
      <p>The main purpose was to help physicians in
their practice by automatic predicting the
cardiovascular risk for a particular patient, in
other words to determine which patient will
have in the near future (next 10 years) a major
cardiovascular event such as sudden death,
therefore the physicians will have to prescribe a
more aggressive medical treatment.</p>
      <p>We used private data from a private clinic in
South- West zone in Romania. Data used were
obtained between October 2017- September
2019. The patients enrolled received cardiology
consult with electrocardiogram, different blood
tests. The examination was Data were
anonymized and patient consent was obtained.</p>
      <p>Patient consultations, cardiac ultrasounds
and exercise tests were performed by a
cardiologist. Patients had previous blood tests .</p>
      <p>We proposed a MLP network with 4 layers
Deep Neural Network sequential model which
has multiple inputs and three outputs because
our model needs to predict cardiac overall risk
for the patient.</p>
      <p>We decided to use the most accessible deep
network architecture that could fulfill our
requirements.</p>
      <p>Each hidden network layer used an rectifier
function (ReLu) and we used the SoftMax
function in our output layer, because we want a
three output result (low, intermediate and high)
therefore the number of categories in the output
layer is more than two.</p>
      <p>For the purpose of implementing and testing
the MLP network we used a custom data set that
included a batch of 784 patients.</p>
      <p>The patient dataset was made of 8 medical
characteristics:RegistryNumber, PatientName,
PatientAge, Gender, Total Cholesterol,LDL
Cholesterol, Glicemia, BMI, ABI,Mean Blood
Pressure.After analyzing the medical data,we
determined each medical input attribute and
noticed that:</p>
      <p>-some attributes like PatientAge, Glicemia,
BMI and LDL attributes are integers; others are
cathegorical attributes like Gender,
RelativeRisk, Sex, etc.</p>
      <p>-In the test population test we have more
male , over 60 years old. According to eurostat
2016 standardised death rate were higher for
man than for women for nearly all the main
causes of death , including cardiac disease.</p>
      <p>-Some attributes with zero value are
nonexistent values for that patient.</p>
      <p>-The patient data set is small (for learning
purposes) and contains 784 rows with 11
columns.The output/endpoint of the dataset
consisted of 3 distinct</p>
      <p>
        We also implemented a Graphical User
Interface in order to enter the data.
For the implementation of the neural network
that predicts risk and makes medical
recommendations (intensive medical treatment
and invasive cardiac procedures), we used
Spyder content in the Anaconda library, which
can be downloaded free from the Internet. It
requires also to install the Tensorflow, Theano
and Keras libraries in Spyder. Keras is the main
library that implements Multilayer perceptron
network models and it is built on Tensorflow
and Theano, so that these two libraries work in
back-end whenever we execute a program in
Keras[
        <xref ref-type="bibr" rid="ref5">5</xref>
        ].
0.01, decay steps of 10000 and decay rate 0.9
and epsilon value of 0.01.[
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]
In Machine Learning, we always divide
medical data into a training part and a testing
part[
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. So , we train the model on the training
data and on the test data we check the accuracy
of the model. The efficiency of the model is
evaluated when we test the model on the test
data using F1-score per each class, overall
accuracy, macro-average accuracy,
weightedmacro-average accuracy[
        <xref ref-type="bibr" rid="ref10">10</xref>
        ][
        <xref ref-type="bibr" rid="ref11">11</xref>
        ].
      </p>
    </sec>
    <sec id="sec-3">
      <title>3. Results</title>
      <p>Our study collected the data from 784 cases.
By training our Deep Learning Network we
achieved two things:
-we calculated the accuracy of the final risk
estimation
-we computed for a new patient the risk score
based on previous patient historical data by
deploying the trained network.</p>
      <p>We trained our model using a batch size of 10
and 120 epochs.</p>
      <p>Because we are modelling a multi-class
classification problem using a MLP neural
network, we decided to reshape the output
attribute of a vector that contains value (high
risk, intermediate risk and low risk) to a matrix
with a boolean for each value by using hot
coding or creating dummy variables from a
categorical variable.</p>
      <p>For example, in this problem the three class
values are low risk, medium risk and high risk.
We can turn this into a hot-coded binary matrix
for each data instance that would look like this:
Because we used one-hot encoding for our
cardiovascular data set, the output layer creates
3 output values, one for each class. The output
value with the highest value will be taken as the
class provided by the model.</p>
      <p>We used a Softmax activation function in the
output layer. This ensures that the output values
are in the range 0 and 1 and can be used as
predicted probabilities.</p>
      <p>The result of predicting the MLP network will
give us the probability that the patient will
develop a severe heart disease. We will convert
that probability into binary 0 and 1.</p>
      <p>In following step we evaluated the
performance of our MLP network model. We
already have final results and thus we can
classification reports to verify the accuracy of
the model.</p>
      <p>
        To test our model we used 10 fold stratified
cross validation because we had a small dataset
and we wanted to be sure that the results do not
depend on the initialization of weights or on the
order of presentation of training data
vectors[
        <xref ref-type="bibr" rid="ref12">12</xref>
        ][
        <xref ref-type="bibr" rid="ref13">13</xref>
        ].
      </p>
      <p>KFold 1 acc: 63.29%
KFold 2 acc: 77.22%
KFold 3 acc: 75.64%
KFold 4 acc: 79.49%
KFold 5 acc: 74.36%
KFold 6 acc: 72.15%
KFold 7 acc: 78.48%
KFold 9 acc: 79.49%</p>
      <p>KFold 10 acc: 73.08%</p>
      <p>
        We have computed the average accuracy
(ACA) as the percentage of correctly classified
cases during the testing phase[
        <xref ref-type="bibr" rid="ref14">14</xref>
        ]. Besides the
ACA, the standard deviation (SD) of the ACA
and the 95% confidence interval were
computed also[
        <xref ref-type="bibr" rid="ref15">15</xref>
        ].
      </p>
      <p>Table 2
MLP performance indicators</p>
      <sec id="sec-3-1">
        <title>Variable</title>
      </sec>
      <sec id="sec-3-2">
        <title>MLPNetwor</title>
        <p>k</p>
        <p>
          The reported averages in our testing
included precision[
          <xref ref-type="bibr" rid="ref16">16</xref>
          ], recall, F1-score per risk
class (low ,intermediate and high), macro
average (averaging the unweighted mean per
risk class, weighted average (averaging the
support-weighted mean per risk class), and
overall accuracy. Support parameter described
number of patients included in each risk class.
        </p>
        <p>This way we determine of the performance
of our supervised learning algorithm.For
computing these parameters we used all the
instances in a predicted class, compared with
the instances of the”true”class.T hese instances
contained "actual" and "predicted" values.</p>
        <p>We obtain an accuracy for our cardiac DL
network model of 80%, which physicians
consider is an acceptable accuracy.</p>
        <p>Finally our model could be used to predict
the cardiac risk for a new patient using classifier
“predict_classes “ method.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Conclusion</title>
      <p>
        Sometimes, the diagnosis of coronary heart
disease can escape doctors. With the help of AI,
even less experienced or tired doctors will have
a high degree of accurate diagnosis. AI can help
doctors improve the effectiveness of their
treatment. AI is not perfect, but it has promising
results. One of the outcome is that AI
algorithms need a lot of data and time to be
trained. Studies have suggested that the
combination of clinicians and AI skills will
provide patients with higher quality diagnostic
results than experience alone.[
        <xref ref-type="bibr" rid="ref17">17</xref>
        ].Sooner or
later, the development of deep learning
applications will affect every aspect of health
care.[
        <xref ref-type="bibr" rid="ref18">18</xref>
        ].
      </p>
      <p>We consider that artificial intelligence can
customizes healthcare for each patient because
neural networks can learn and so the cardiac
risk scores is improved.</p>
      <p>Therefore using this innovative DL network,
we were able to provide an unitary diagnosis
method for physicians that allowed the
“localization” of the medical ESC guidelines to
Romania region. This way we created an
method to transmit medical knowledge in a
consistent way, therefore physicians will
benefit from both ESC guidelines and “local”
experience because a DL network has the
ability to “learn” from previous medical
patients data in diagnosis of coronary heart
diseases.</p>
      <p>
        We further plan to train our application and
deep neural network with more clinical data,
including ultrasound and cardiac 3D
angiography data[
        <xref ref-type="bibr" rid="ref20">20</xref>
        ]. Also we plan to use more
complex deep neural networks with multiple
layers to test if we can further improve the
overall accuracy of our risk estimator.
      </p>
    </sec>
    <sec id="sec-5">
      <title>5. References</title>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          <article-title>[1] 2019 ESC Guidelines for the diagnosis and management of chronic coronary syndromes:The Task Force for the diagnosis and management of chronic coronary syndromes of the European Society of Cardiology (ESC)</article-title>
          ,
          <source>Juhani Knuuti</source>
          ,
          <year>2019</year>
          ,European Heart Journal, https://doi.org/10.1093/eurheartj/ehz425
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <surname>WHO</surname>
          </string-name>
          <article-title>The top 10 causes of death URL:https://www</article-title>
          .who.int/newsroom/fact-sheets/detail/the-top-10
          <string-name>
            <surname>-</surname>
          </string-name>
          causesof-death
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <surname>LeCun</surname>
            <given-names>Y</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bengio</surname>
            <given-names>Y</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hinton</surname>
            <given-names>G</given-names>
          </string-name>
          (
          <year>2015</year>
          )
          <article-title>Deep learning</article-title>
          .
          <source>Nature</source>
          <volume>521</volume>
          :
          <fpage>436</fpage>
          -
          <lpage>444</lpage>
          . pmid:
          <volume>26017442</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>Jason</given-names>
            <surname>Brownlee</surname>
          </string-name>
          ,
          <article-title>Your First Deep Learning Project in Python with Keras Step-By-</article-title>
          <string-name>
            <surname>Step</surname>
          </string-name>
          ,
          <article-title>Machine learning mastery, 2019</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>Pushkar</given-names>
            <surname>Mandot</surname>
          </string-name>
          ,
          <article-title>Build your First DeepLearning Neural Network Model using Keras in Python</article-title>
          , Medium,
          <year>2017</year>
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>Milad</given-names>
            <surname>Toutounchian</surname>
          </string-name>
          ,
          <article-title>Deep Learning from Scratch and Using Tensorflow in Python</article-title>
          , Medium,
          <year>2019</year>
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>Khyati</given-names>
            <surname>Mahendru</surname>
          </string-name>
          ,
          <article-title>A Detailed Guide to 7 Loss Functions for Machine Learning Algorithms</article-title>
          , Medium,
          <year>2014</year>
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <surname>Belciug</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          , Artificial Intelligence in Cancer: Diagnostic to Tailored Treatment, (Elsevier,
          <year>2020</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <surname>Peat</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Barton</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          , Medical Statistics:
          <article-title>A guide to data analysis and critical appraisal</article-title>
          .
          <source>(Blackwell Publishing</source>
          ,
          <year>2005</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <surname>Thode</surname>
            ,
            <given-names>H.J.</given-names>
          </string-name>
          , Testing for normality. (New York: Marcel Dekker,
          <year>2002</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <surname>Altman</surname>
            ,
            <given-names>D.G.</given-names>
          </string-name>
          ,
          <source>Practical Statistics for Medical Research</source>
          , (Chapman and Hall, New York,
          <year>1991</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>Jason</given-names>
            <surname>Brownlee</surname>
          </string-name>
          ,
          <article-title>A Gentle Introduction to k-fold Cross-Validation, Machine learning mastery, 2018</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <surname>Varma</surname>
          </string-name>
          , Sudhir; Simon, Richard (
          <year>2006</year>
          ).
          <article-title>"Bias in error estimation when using crossvalidation for model selection"</article-title>
          .
          <source>BMC Bioinformatics</source>
          .
          <volume>7</volume>
          : 91. doi:
          <volume>10</volume>
          .1186/
          <fpage>1471</fpage>
          - 2105-7-
          <lpage>91</lpage>
          . PMC 1397873. PMID 16504092.
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <surname>Politis</surname>
          </string-name>
          , Dimitris N.;
          <string-name>
            <surname>Romano</surname>
            ,
            <given-names>Joseph P.</given-names>
          </string-name>
          (
          <year>1994</year>
          ).
          <article-title>"The Stationary Bootstrap"</article-title>
          .
          <source>Journal of the American Statistical Association</source>
          .
          <volume>89</volume>
          (
          <issue>428</issue>
          ):
          <fpage>1303</fpage>
          -
          <lpage>1313</lpage>
          . doi:
          <volume>10</volume>
          .1080/01621459.
          <year>1994</year>
          .
          <volume>10476870</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <surname>Picard</surname>
          </string-name>
          , Richard; Cook,
          <string-name>
            <surname>Dennis</surname>
          </string-name>
          (
          <year>1984</year>
          ).
          <article-title>"Cross-Validation of Regression Models"</article-title>
          .
          <source>Journal of the American Statistical Association</source>
          .
          <volume>79</volume>
          (
          <issue>387</issue>
          ):
          <fpage>575</fpage>
          -
          <lpage>583</lpage>
          . doi:
          <volume>10</volume>
          .2307/2288403. JSTOR 2288403.
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <surname>Chen</surname>
            <given-names>JH</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Asch</surname>
            <given-names>SM</given-names>
          </string-name>
          .
          <article-title>Machine learning and prediction in medicine- beyond the peak of inflated expectations</article-title>
          .
          <source>N. Eng J Med</source>
          <year>2017</year>
          ;
          <volume>376</volume>
          :
          <fpage>2507</fpage>
          -
          <lpage>2509</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <surname>Jason</surname>
            <given-names>Brownlee</given-names>
          </string-name>
          , How to Calculate Precision, Recall, F1, and
          <article-title>More for Deep Learning Models, Machine learning mastery, 2020</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>Popa</given-names>
            <surname>Didi</surname>
          </string-name>
          <string-name>
            <surname>Liliana</surname>
          </string-name>
          , Faiq Baji,
          <article-title>Popa Radu Teodoru -Overview of the Deep Learning in Medical Imaging</article-title>
          ,
          <source>Annals of the Univ. Craiova</source>
          , Series: Automation, Computers,
          <source>Electronics and Mechatronics</source>
          , Vol.
          <volume>14</volume>
          (
          <issue>41</issue>
          ),
          <source>No. 1</source>
          , 2017
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <surname>Willis</surname>
            <given-names>BH</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Riley</surname>
            <given-names>RD</given-names>
          </string-name>
          (
          <year>2017</year>
          ).
          <article-title>"Measuring the statistical validity of summary metaanalysis and meta-regression results for use in clinical practice"</article-title>
          . Statistics in Medicine.
          <volume>36</volume>
          (
          <issue>21</issue>
          ):
          <fpage>3283</fpage>
          -
          <lpage>3301</lpage>
          . doi:
          <volume>10</volume>
          .1002/sim.7372. PMC 5575530. PMID 28620945.
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <surname>Riley</surname>
            <given-names>RD</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ahmed</surname>
            <given-names>I</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Debray</surname>
            <given-names>TP</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Willis</surname>
            <given-names>BH</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Noordzij</surname>
            <given-names>P</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Higgins</surname>
            <given-names>JP</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Deeks</surname>
            <given-names>JJ</given-names>
          </string-name>
          (
          <year>2015</year>
          ).
          <article-title>"Summarising and validating test accuracy results across multiple studies for use in clinical practice"</article-title>
          . Statistics in Medicine.
          <volume>34</volume>
          (
          <issue>13</issue>
          ):
          <fpage>2081</fpage>
          -
          <lpage>2103</lpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>