<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>K. Dolapsis);</journal-title>
      </journal-title-group>
    </journal-meta>
    <article-meta>
      <title-group>
        <article-title>Comparative analysis of LS-SVM and random forest models for sensor-based lameness detection in cattle⋆</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Konstantinos Dolaptsis</string-name>
          <email>k.dolaptsis@certh.gr</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Georgios Tziotzios</string-name>
          <email>g.tziotzios@certh.gr</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Antonios Morellos</string-name>
          <email>a.morellos@certh.gr</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Dimitrios Kateris</string-name>
          <email>d.kateris@certh.gr</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Dionysis Bochtis</string-name>
          <email>d.bochtis@certh.gr</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Institute for Bio-Economy and Agri-Technology (iBO), Centre for Research and Technology-Hellas (CERTH)</institution>
          ,
          <addr-line>6</addr-line>
        </aff>
      </contrib-group>
      <volume>000</volume>
      <fpage>0</fpage>
      <lpage>0001</lpage>
      <abstract>
        <p>Lameness is a significant concern in dairy cattle management, affecting both animal welfare and farm productivity. Despite efforts to mitigate its impact, traditional methods of lameness detection often overlook early signs, leading to delayed intervention and prolonged suffering for affected cows. This challenge underscores the need for more effective and proactive approaches to identifying and managing lameness. This study seeks to create an objective lameness detection methodology using sensor data from cattle limbs. Inertial Measurement Units (IMUs) were attached to cattle in Eastern Macedonia and Thrace, Greece, to record movement data. The collected data were preprocessed to address missing values, and features from both the time and frequency domains were extracted. Key gyroscope and accelerometer features were selected through Neighborhood Components Analysis. These features were then used to train LeastSquares Support Vector Machine (LS-SVM) and Multiclass Random Forest (MRF) models to classify lameness severity into healthy, mild, and severe categories, achieving an overall accuracy of more than 0.90 for both models. MRF has shown a better performance than LS-SVM.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;Lameness detection</kwd>
        <kwd>machine learning</kwd>
        <kwd>inertial measurement unit</kwd>
        <kwd>cattle</kwd>
        <kwd>Multiclass Random Forest</kwd>
        <kwd>Least Squares Support Vector Machine 1</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Lameness in dairy cattle is a major welfare concern and economic burden, leading to decreased
milk production, increased treatment costs, and early culling [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. It causes pain and distress, further
reducing productivity. Early detection is crucial but traditional methods, like visual assessment, are
subjective and inconsistent [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. Automated systems are needed to improve accuracy and enable
timely interventions [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ].
      </p>
      <p>
        Techniques such as human observation and pressure-sensitive walkways have historically been
used to detect lameness, though these methods are labor-intensive and costly [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. Advances in
sensor-based technologies, like inertial measurement units (IMU), have enabled real-time monitoring
of cattle, providing data that can be analyzed to detect lameness earlier [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ].
      </p>
      <p>
        Machine learning (ML) algorithms, particularly support vector machines (SVM) and random
forests (RF) are increasingly applied to analyze IMU data for lameness detection [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. This study
compares LS-SVM and MRF models, using features selected through neighborhood components
analysis (NCA) to enhance classification accuracy [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ].
      </p>
    </sec>
    <sec id="sec-2">
      <title>2. Materials and Methods</title>
      <sec id="sec-2-1">
        <title>2.1. Data Sampling</title>
        <p>Data collection was carried out using the Blue Trident IMU device from (Figure 1). Four devices
were placed on the limbs of each animal. The device features three types of sensors: an accelerometer,
a gyroscope, and a magnetometer, all capturing data across three axes. In this study, only the data
from the accelerometer and gyroscope were used, with a sampling rate of 500 Hz. Consequently, the
dataset consists of six columns for sensor readings, one for timestamps, and another for the animal's
class or state at the time of measurement, which was determined by expert visual observation and
used as the ground truth. Each animal was observed freely for 10 minutes while the IMU sensors
were active.</p>
      </sec>
      <sec id="sec-2-2">
        <title>2.2. Feature Extraction and Selection</title>
        <p>
          Feature extraction is widely used in the analysis of locomotion monitoring data. This approach
involves deriving new features from the raw data to improve the interpretation of the captured
information and to enable more precise analysis compared to using raw data alone. The extracted
features can be categorized into several groups, such as statistical (e.g., mean, standard deviation),
time domain (e.g., minimum and maximum values, signal energy), and frequency domain (e.g.,
dominant frequency, spectral area) [
          <xref ref-type="bibr" rid="ref8">8</xref>
          ], [
          <xref ref-type="bibr" rid="ref9">9</xref>
          ]. For this study, 13 unique features relevant to both
lameness detection and general livestock activity were identified from the literature and extracted
for each sensor axis (see Table 1).
Signal Area
        </p>
        <p>Neighborhood Components Analysis (NCA) was employed to identify the most relevant features
for lameness detection. NCA is a feature selection method that seeks to maximize the accuracy of
classification by learning a linear transformation of the input data. It selects features that improve
the performance of the model by emphasizing those that contribute most to class separation. In this
study, a threshold of 0.5 was applied to select the top-performing features for further analysis.</p>
      </sec>
      <sec id="sec-2-3">
        <title>2.3. Machine Learning Algorithms</title>
        <p>In this study two machine learning algorithms have been employed for the detection of lameness
in the dairy cattle; the Multiclass Random Forest (MRF) and the Least-Squares Support Vector
Machine (LS-SVM).</p>
        <p>
          A Multiclass Random Forest model extends the traditional Random Forest algorithm to address
classification tasks involving more than two classes. This model builds multiple decision trees, each
trained on a random subset of the dataset. Each tree independently classifies the input into one of
the available classes, and the final prediction is made based on the majority vote from all the trees
involved [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ]. Multiclass Random Forest models are versatile and can be used with both small and
large datasets. However, they are particularly effective for large datasets due to their scalability, as
they are designed to handle high-dimensional data and large amounts of information efficiently. [
          <xref ref-type="bibr" rid="ref11">11</xref>
          ].
        </p>
        <p>
          The Least-Squares Support Vector Machine (LS-SVM) is a modified version of the traditional
SVM, designed to solve classification tasks more efficiently by transforming the problem into a set
of linear equations [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ]. LS-SVM is particularly effective for smaller datasets and binary classification
[
          <xref ref-type="bibr" rid="ref12">12</xref>
          ]. However, it may not capture nonlinear patterns as effectively unless an appropriate kernel is
used.
        </p>
      </sec>
      <sec id="sec-2-4">
        <title>2.4. Evaluation Metrics</title>
        <p>For the evaluation of classification algorithms, the methods used were accuracy, precision and
recall, as presented in Equations (1), (2) and (3).
where TP, TN, FP and FN represent the True Positive, True Negative, False Positive and False
Negative samples in the confusion matrix, respectively.</p>
        <p>
          The accuracy estimation method is one of the most common, but it suffers due to its sensitivity
to imbalanced data. Another issue is that two classification algorithms may have the same accuracy
but different performance in terms of the correctness of the decisions they make [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ]. For this reason,
it is recommended not to be the sole method chosen for evaluating such models.
        </p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>3. Results and Discussion</title>
      <sec id="sec-3-1">
        <title>3.1. Features Selection</title>
        <p>The features selection as it was mentioned, was based on the NCA algorithm, as is shown in
Figure 2.</p>
        <p>The selected variables, as they have been decided by NCA algorithm were the mean, the standard
deviation and the maximum value across the y axis of the accelerometer, the zero crossing across
the x axis of the gyroscope and the signal area, across the z axis of the accelerometer. The mean,
standard deviation, and max in the y-axis of the accelerometer measure changes in vertical limb
movement, providing insights into stride length and height. The zero crossing in the x-axis of the
gyroscope may reflect shifts in limb orientation and balance, essential for detecting irregular gait
patterns. The signal area in the z-axis of the accelerometer tracks overall limb movement energy,
which is critical for identifying reduced or exaggerated movement, both of which can indicate
lameness. These features together offer a comprehensive view of movement abnormalities.</p>
      </sec>
      <sec id="sec-3-2">
        <title>3.2. Hyperparameter Tuning</title>
        <p>Hyperparameter selection is a crucial step in the machine learning pipeline, affecting model
performance, training time, and generalization. Techniques like grid search are commonly used for
hyperparameter tuning. The grid search method systematically tests combinations of
hyperparameters, using cross-validation to ensure results aren’t dependent on data splits. The best
hyperparameter set is determined based on performance metrics from a validation set. The list of the
possible hyperparameters for each ML algorithm and their values range is given in Table 2.</p>
        <p>The selected hyperparameters after the grid search method were the following: for the LS-SVM
model, a C of 10, a Gamma of 0.05 and an Epsilon of 0.01 and for the MRF model a number of trees
of 100, a Gini criterion, a maximum depth of 10 and a minimum samples to split a node of 2.</p>
      </sec>
      <sec id="sec-3-3">
        <title>3.3. Model’s Performance</title>
        <p>The performance of the classification models was evaluated using a confusion matrix, which
provides insights into the model's accuracy and ability to distinguish between the different lameness
LS-SVM</p>
        <p>MRF</p>
        <p>C
Gamma (sigma)</p>
        <p>Epsilon
Number of Trees</p>
        <p>Criterion
Maximum Depth</p>
        <p>Minimum
Samples to Split a</p>
        <p>Node</p>
        <p>Short Description
trade-off between model complexity</p>
        <p>and training error.
width of the Gaussian kernel</p>
        <p>tolerance for errors
number of decision trees in the model
function to measure quality of a split
longest path from the root to a leaf</p>
        <p>node
the minimum number of data samples
required for a node to be split into
child nodes</p>
        <sec id="sec-3-3-1">
          <title>Possible Values</title>
          <p>0.5, 1, 10, 20, 50
0.01, 0.05, 0.1, 0.5
0.1, 0.05, 0.01, 0.001
50, 100, 200, 500</p>
          <p>Gini, Entropy
None, 10,20, 50
2, 5, 10, 20
status of the cattle. The data split followed a 70-30% scheme, for training and test, respectively, with
the test set containing 168 values per feature for each class. The following confusion matrices of
Table 3 and performance metrics summarize the results.</p>
          <p>The results from both confusion matrices demonstrate a strong overall ability of the models to
detect varying levels of lameness in cattle, with notable differences in performance between the
LSSVM and Random Forest classifiers. Comparatively, Multiclass Random Forest outperforms LS-SVM
in both precision and recall across each lameness category. For precision, Random Forest achieved
slightly higher values across all categories (0.97, 0.97, and 0.92 for healthy, mild, and severe lameness,
respectively) compared to LS-SVM (0.96, 0.92, and 0.89). In terms of recall, Random Forest also
displays superior results (0.96, 0.90, and 0.99) in comparison to LS-SVM (0.90, 0.87, and 0.98),
underscoring its enhanced effectiveness in identifying actual cases across all categories, especially
in the "healthy" and "mild lameness" classes.</p>
          <p>
            Overall, Random Forest demonstrates a more balanced and reliable classification performance in
this scenario, effectively capturing a greater number of true instances with fewer false alarms. This
improvement can likely be attributed to Random Forest’s ability to capture complex, non-linear
relationships within the data, a characteristic that is often limited in LS-SVM models due to their
reliance on a fixed kernel function [
            <xref ref-type="bibr" rid="ref14 ref15">14,15</xref>
            ].
          </p>
        </sec>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Conclusions</title>
      <p>•
•
•
•</p>
      <p>The use of Inertial Measurement Units (IMUs) combined with machine learning models (MRF
and LS-SVM) was effective in detecting lameness in cattle, achieving over 90% accuracy.
MRF outperformed LS-SVM across all categories. This is likely due to MRF's ability to handle
complex, non-linear relationships.</p>
      <p>Neighborhood Components Analysis (NCA) successfully identified key gyroscope and
accelerometer features, enhancing model performance.</p>
      <p>Overall, sensor-based detection offers an objective and efficient alternative to traditional
visual assessments, with the potential for early intervention and improved animal welfare.</p>
    </sec>
    <sec id="sec-5">
      <title>Acknowledgements</title>
      <p>This research was funded by the Action «Rural Development Programme of Greece 2014-2020»
under the call Measure 16 “Co-operation”, Sub-Measure 16.1 – 16.2» that is co-funded by the
European Regional Development Fund and Region of Eastern Macedonia and Thrace, project
«COWLAM - Lameness identification system at milk- producing cow units in the Region of Eastern
Macedonia and Thrace» (Project code: Μ16ΣΥΝ2-00031).</p>
    </sec>
    <sec id="sec-6">
      <title>Declaration on Generative AI</title>
      <sec id="sec-6-1">
        <title>The author(s) have not employed any Generative AI tools.</title>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>S.</given-names>
            <surname>Lucey</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G. J.</given-names>
            <surname>Rowlands</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A. M.</given-names>
            <surname>Russell</surname>
          </string-name>
          , “
          <article-title>The association between lameness and fertility in dairy cows</article-title>
          .,
          <source>” Vet Rec</source>
          , vol.
          <volume>118</volume>
          , no.
          <issue>23</issue>
          , pp.
          <fpage>628</fpage>
          -
          <lpage>631</lpage>
          ,
          <year>1986</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>H. R.</given-names>
            <surname>Whay</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. C. J.</given-names>
            <surname>Main</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L. E.</given-names>
            <surname>Green</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A. J. F.</given-names>
            <surname>Webster</surname>
          </string-name>
          , “
          <article-title>Assessment of the welfare of dairy caftle using animal‐based measurements: direct observations and investigation of farm records,” Veterinary record</article-title>
          , vol.
          <volume>153</volume>
          , no.
          <issue>7</issue>
          , pp.
          <fpage>197</fpage>
          -
          <lpage>202</lpage>
          ,
          <year>2003</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>J.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Ji</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Zhao</surname>
          </string-name>
          , and G. Liu, “
          <article-title>Development of a wireless measurement system for classifying cow behavior using accelerometer data and location data</article-title>
          ,
          <source>” Appl Eng Agric</source>
          , vol.
          <volume>35</volume>
          , no.
          <issue>2</issue>
          , pp.
          <fpage>135</fpage>
          -
          <lpage>147</lpage>
          ,
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>F. C.</given-names>
            <surname>Flower</surname>
          </string-name>
          and
          <string-name>
            <given-names>D. M.</given-names>
            <surname>Weary</surname>
          </string-name>
          , “
          <article-title>Effect of hoof pathologies on subjective assessments of dairy cow gait</article-title>
          ,
          <source>” J Dairy Sci</source>
          , vol.
          <volume>89</volume>
          , no.
          <issue>1</issue>
          , pp.
          <fpage>139</fpage>
          -
          <lpage>146</lpage>
          ,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>M.</given-names>
            <surname>Alsaaod</surname>
          </string-name>
          and
          <string-name>
            <given-names>W.</given-names>
            <surname>Büscher</surname>
          </string-name>
          , “
          <article-title>Detection of hoof lesions using digital infrared thermography in dairy cows</article-title>
          ,
          <source>” J Dairy Sci</source>
          , vol.
          <volume>95</volume>
          , no.
          <issue>2</issue>
          , pp.
          <fpage>735</fpage>
          -
          <lpage>742</lpage>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>M.</given-names>
            <surname>Taneja</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Byabazaire</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Jalodia</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Davy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Olariu</surname>
          </string-name>
          , and
          <string-name>
            <given-names>P.</given-names>
            <surname>Malone</surname>
          </string-name>
          , “
          <article-title>Machine learning based fog computing assisted data-driven approach for early lameness detection in dairy cattle,” Comput Electron Agric</article-title>
          , vol.
          <volume>171</volume>
          , p.
          <fpage>105286</fpage>
          ,
          <year>2020</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>N.</given-names>
            <surname>Kavya</surname>
          </string-name>
          et al., “
          <article-title>Feature selection using neighborhood component analysis with support vector machine for classification of breast mammograms</article-title>
          ,” in International Conference on Communication,
          <source>Computing and Electronics Systems: Proceedings of ICCCES 2019</source>
          , Springer,
          <year>2020</year>
          , pp.
          <fpage>253</fpage>
          -
          <lpage>260</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>D.</given-names>
            <surname>Kateris</surname>
          </string-name>
          , I. Gravalos,
          <string-name>
            <given-names>T.</given-names>
            <surname>Gialamas</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Xyradakis</surname>
          </string-name>
          , and
          <string-name>
            <given-names>D.</given-names>
            <surname>Moshou</surname>
          </string-name>
          , “
          <article-title>A new approach to fault diagnosis in agricultural tractor mechanical gearbox,”</article-title>
          <source>In Proceedings of the 6th International Conference on Trends in Agricultural Engineering</source>
          ,
          <fpage>7</fpage>
          - 9
          <source>September</source>
          <year>2016</year>
          , Prague, Czech Republic, pp.
          <fpage>290</fpage>
          -
          <lpage>299</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>J.</given-names>
            <surname>Kaler</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Mitsch</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. A.</given-names>
            <surname>Vázquez-Diosdado</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Bollard</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Dottorini</surname>
          </string-name>
          , and
          <string-name>
            <given-names>K. A.</given-names>
            <surname>Ellis</surname>
          </string-name>
          , “
          <article-title>Automated detection of lameness in sheep using machine learning approaches: Novel insights into behavioural differences among lame and non-lame sheep</article-title>
          ,
          <source>” R Soc Open Sci</source>
          , vol.
          <volume>7</volume>
          , no.
          <issue>1</issue>
          , p.
          <fpage>190824</fpage>
          ,
          <year>2020</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>A.</given-names>
            <surname>Prinzie</surname>
          </string-name>
          and D. Van den Poel, “
          <article-title>Random forests for multiclass classification: Random multinomial logit</article-title>
          ,
          <source>” Expert Syst. Appl.</source>
          , vol.
          <volume>34</volume>
          , no.
          <issue>3</issue>
          , pp.
          <fpage>1721</fpage>
          -
          <lpage>1732</lpage>
          ,
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>A.</given-names>
            <surname>Tripathi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Goswami</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S. K.</given-names>
            <surname>Trivedi</surname>
          </string-name>
          , and R. D. Sharma, “
          <article-title>A multi class random forest (MCRF) model for classification of small plant peptides</article-title>
          ,
          <source>” Int. J. Inf. Manag. Data Insights</source>
          , vol.
          <volume>1</volume>
          , no.
          <issue>2</issue>
          , pp.
          <fpage>100029</fpage>
          ,
          <year>2021</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>J. A. K.</given-names>
            <surname>Suykens</surname>
          </string-name>
          and
          <string-name>
            <given-names>J.</given-names>
            <surname>Vandewalle</surname>
          </string-name>
          , “
          <article-title>Least squares support vector machine classifiers</article-title>
          ,
          <source>” Neural Process Lett</source>
          , vol.
          <volume>9</volume>
          , pp.
          <fpage>293</fpage>
          -
          <lpage>300</lpage>
          ,
          <year>1999</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>V.</given-names>
            <surname>García</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R. A.</given-names>
            <surname>Mollineda</surname>
          </string-name>
          , and
          <string-name>
            <given-names>J. S.</given-names>
            <surname>Sánchez</surname>
          </string-name>
          , “
          <article-title>Theoretical analysis of a performance measure for imbalanced data</article-title>
          ,
          <source>” in 2010 20th International Conference on Pattern Recognition</source>
          , IEEE,
          <year>2010</year>
          , pp.
          <fpage>617</fpage>
          -
          <lpage>620</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>M.</given-names>
            <surname>Singla</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Ghosh</surname>
          </string-name>
          , and
          <string-name>
            <surname>K. K. Shukla</surname>
          </string-name>
          , “
          <article-title>A survey of robust optimization based machine learning with special reference to support vector machines</article-title>
          ,”
          <source>International Journal of Machine Learning and Cybernetics</source>
          , vol.
          <volume>11</volume>
          , no.
          <issue>7</issue>
          , pp.
          <fpage>1359</fpage>
          -
          <lpage>1385</lpage>
          ,
          <year>2020</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>T.</given-names>
            <surname>Hastie</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Tibshirani</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. H.</given-names>
            <surname>Friedman</surname>
          </string-name>
          , and
          <string-name>
            <given-names>J. H.</given-names>
            <surname>Friedman</surname>
          </string-name>
          ,
          <article-title>The elements of statistical learning: data mining, inference, and prediction</article-title>
          , vol.
          <volume>2</volume>
          . Springer,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>