<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Modified Technique for Constructing Nonlinear Regression Models Based on the Multivariate Normalizing Transformations</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Sergiy Prykhodko</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Natalia Prykhodko</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Admiral Makarov National University of Shipbuilding, Heroes of Ukraine Ave.</institution>
          ,
          <addr-line>9, Mykolaiv, 54007</addr-line>
          ,
          <country country="UA">Ukraine</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>size</institution>
          ,
          <addr-line>estimation, PHP</addr-line>
        </aff>
      </contrib-group>
      <fpage>156</fpage>
      <lpage>166</lpage>
      <abstract>
        <p>The technique for constructing nonlinear regression models based on the multivariate normalizing transformations and prediction intervals is modified by testing the normality of error distribution in the linear regression model for normalized data, which is applied to construct the nonlinear one. We have demonstrated that there may be multidimensional data sets (for example, software metrics) for which the constructed nonlinear regression model has poor prediction accuracy even in the cases of, first, multivariate normality of the normalized data, second, outlier cutoff using the technique based on the multivariate normalizing transformations and prediction intervals of nonlinear regression. As a rule, this is because the error distribution in the linear regression model for normalized data becomes non-Gaussian. Therefore, in this case, we propose to discard the multidimensional data point for which the value of the error modulus in the model is maximum. We have considered the application of a modified technique for constructing a nonlinear regression model with three predictors (software metrics) to estimate the size of open source PHP-based apps. This model has the better values of well-known prediction accuracy metrics compared with the model, which is only constructed based on the multivariate normalizing transformation and prediction intervals without testing the normality of error distribution in the linear regression model for normalized data and discarding the multidimensional data point for which the value of the error modulus in the linear model is maximum if the distribution of residuals in the linear one is not Gaussian. Nonlinear regression model, normalizing transformation, prediction interval, outlier, software</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Regression models are one of the general models in mathematical modeling of various dependent
random variables and other applications in many fields, including information technologies. As known,
the use of linear regression models has some limitations. For example, linear regression models have
significant limitations as practical techniques for pattern recognition, particularly for problems
involving input spaces of high dimensionality [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. And the application of linear regression models
demands the assumption fulfillment of error distribution normality [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] that is valid in particular cases
only. This leads to the need to apply nonlinear regression models.
      </p>
      <p>
        As we know [
        <xref ref-type="bibr" rid="ref2 ref3 ref4 ref5 ref6 ref7">2-7</xref>
        ], normalizing transformations are used to build nonlinear regression models. In
this case, methods for outlier detection in nonlinear regression models based on the normalizing
transformations can be applied [
        <xref ref-type="bibr" rid="ref8 ref9">8, 9</xref>
        ]. Like the method proposed in [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ], which combines a new robust
nonlinear regression method with a new method for identifying outliers, the technique to construct
nonlinear regression models considered in [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] includes a new technique for detecting outliers, but
multivariate ones in contrast to [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]. Notice that nonlinear regression models built by the above technique
[
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] based on the multivariate normalizing transformations and prediction intervals usually lead to better
results compared to the models that are constructed using univariate transformations and without taking
into account the presence of outliers. However, there may be multidimensional data sets for which the
      </p>
      <p>
        2022 Copyright for this paper by its authors.
constructed nonlinear regression model has unsatisfactory prediction accuracy even after all outliers are
removed and ending the construction of the nonlinear regression model using the technique [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. This
result, as we have found out, may be explained by the fact that the error distribution in the linear
regression model for normalized data becomes non-Gaussian at the final step of constructing the
nonlinear regression model. And as we know [
        <xref ref-type="bibr" rid="ref11 ref12 ref13 ref14 ref2 ref3 ref4 ref5 ref6 ref7">2-7, 11-14</xref>
        ], the error distribution in the linear regression
model should be Gaussian. For this reason, the technique [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] requires some modification.
      </p>
      <p>
        According to [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ], for a disturbance (error) term in a linear regression model "the assumption of
normality may be checked by examining the residuals." And the existing methods for outlier detection
in regressions by analysis of residuals (including the standardized, studentized, and studentized deleted
residuals) are based on the assumption of normality of residuals. Also, as noted in [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ], "procedures for
the detection of outliers rely almost exclusively on the detection of extreme residuals, so much so that
the two are used interchangeably by some authors and researchers." Therefore, we have proposed to
modify the technique [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] using testing the normality of error distribution in the linear regression model
for normalized data, which is applied to construct the nonlinear model and discarding the
multidimensional data point for which the value of the residual modulus in the linear regression model
is maximum in the case if this error distribution is not Gaussian.
      </p>
      <p>
        Next, we describe the proposed modification of the technique [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] and demonstrate its viability by
the example of constructing the nonlinear regression model with three predictors (software metrics) to
estimate the size of open source PHP-based apps.
      </p>
    </sec>
    <sec id="sec-2">
      <title>2. A modified technique</title>
      <p>
        We modify the technique to construct nonlinear regression models based on the multivariate
normalizing transformations and prediction intervals [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. We combined the technique [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] with outlier
detection using residuals in the linear regression model for normalized data, which is applied to
construct the nonlinear one. A modified technique follows six steps.
      </p>
      <p>Step 1. Normalize a multivariate non-Gaussian data set by a normalizing transformation. We apply
a multivariate normalizing transformation.</p>
      <p>Step 2. Determine whether one multidimensional data point of a multivariate non-Gaussian data set
is a multidimensional outlier. If there is a multidimensional outlier in a multivariate non-Gaussian data
set, then discard the one and go to step 1, else go to step 3.</p>
      <p>Step 3. Build the linear regression model for normalized data, which is applied to construct the
nonlinear one.</p>
      <p>Step 4. Test the normality of distribution of residuals in the linear regression model for normalized
data. If the distribution of residuals in the linear regression model for normalized data is not Gaussian,
then discard the multidimensional data point for which the value of the modulus of residual in the model
is maximum and go to step 1, else go to step 5.</p>
      <p>Step 5. Construct the nonlinear regression model based on the multivariate normalizing
transformation and the linear regression model for normalized data.</p>
      <p>Step 6. Build the prediction interval of nonlinear regression, and determine whether one or more
values of the response (dependent random variable) are outliers. If there are outliers in the nonlinear
regression model, then discard the ones and go to step 1, else complete constructing the nonlinear
regression model.</p>
      <p>
        Notice that outlier detection in a multivariate non-Gaussian data set is included in both the technique
[
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] and the modified one because after going to step 1 from step 6, an outlier may again appear in the
reduced data set. To detect outliers in step 2, we apply the statistical technique [
        <xref ref-type="bibr" rid="ref15 ref16">15, 16</xref>
        ] based on the
normalizing transformations and the Mahalanobis squared distance (MSD). As in [
        <xref ref-type="bibr" rid="ref15 ref16 ref9">9, 15, 16</xref>
        ], we use
the Johnson multivariate transformation to normalize a multivariate non-Gaussian data set at step 1.
      </p>
      <p>In step 3, we build the linear regression model for normalized data by the least squared method.
Next, we calculate the i-value of residual in the linear regression model for normalized data by the
formula
i  ZYi  ZˆYi  ZYi  bˆ0  bˆ1Z1i  bˆ2Z2i  bˆk Zki ,
(1)
where ZYi is the normalized i-value of the non-Gaussian dependent random variable Y; ZˆYi is a
prediction result by linear regression equation for normalized data for i-values of normalized predictors
Z1 , Z2 , , Zk , which are transformed from independent variables X1 , X2 , , X k , by a bijective
multivariate normalizing transformation T  ψP of non-Gaussian random vector
P  Y , X1 , X2 ,, XkT to Gaussian one T  ZY , Z1 , Z2 ,, ZkT , ZˆYi  bˆ0  bˆ1Z1i  bˆ2Z2i  bˆk Zki ; k is a
number of independent variables (predictors or regressors); bˆ0 , bˆ1 , bˆ2 , , bˆk are estimates of
parameters in linear regression equation for normalized data.</p>
      <p>To test the normality of distribution of residuals in the linear regression model for normalized data,
we apply the Kolmogorov-Smirnov test if the number of values of the random variable is less than 30,
and the Pearson chi-squared test in the opposite case. As we note above, if the distribution of residuals
in the linear regression model for normalized data is not Gaussian, then we discard the multidimensional
data point for which the value of the modulus of residual in the model is maximum and go to step 1. In
this case, the residual value is determined by the formula (1). The detection of extreme residual is the
same as that and the detection of an outlier. But here, we use such outlier detection because the
distribution of residuals in the linear regression model for normalized data is not Gaussian. Also, notice
that we go to step 1 if the distribution of residuals in the linear regression model for normalized data is
not Gaussian because there is no justification for the use of a linear regression model in this case.</p>
      <p>
        We construct the nonlinear regression model based on the multivariate normalizing transformation
and the linear regression model for normalized data if the null hypothesis H0 that the observed frequency
distribution of residuals is the same as the normal distribution is accepted. To do this, we apply the
technique [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. Also, we build the prediction interval of nonlinear regression according to [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ].
      </p>
      <p>
        Modification of the technique [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] consists of adding step 4 and highlighting step 3 to build the linear
regression model for normalized data, which is applied to construct the nonlinear one. Next, we consider
the example of constructing the nonlinear regression model with three predictors by a modified
technique to demonstrate its viability for estimating the size of open source PHP-based apps.
      </p>
    </sec>
    <sec id="sec-3">
      <title>3. Model construction example</title>
      <p>We consider the example of constructing a nonlinear regression model with three predictors to
estimate the size of open source PHP-based apps. We construct a model for the non-Gaussian data set
from 44 apps hosted on GitHub (https://github.com) by a modified technique. The data set was obtained
using the PhpMetrics tool (https://phpmetrics.org/). The model is constructed around the following
metrics (variables) of the app: the size (in KLOC, thousand lines of code) Y, the number of classes X1,
the average number of methods per class X2, a sum of average afferent coupling, and average efferent
coupling per class X3. Table 1 contains the data set of the above metrics from 44 rows. Moreover, Y is
the dependent variable, and X1, X2, and X3 are independent variables (predictors).</p>
      <p>
        We tested the normality of four-dimensional data from Table 1 by the Mardia test based on the
measures of the multivariate skewness 1 and kurtosis 2 [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ]. According to the Mardia test, the
fourvariate distribution of four-dimensional data from Table 1 is not Gaussian since the test statistic for
multivariate skewness N 1 6 of this data, which equals 237.49, is greater than the value of the
ChiSquare distribution quantile, which is 45.31 for 20 degrees of freedom and 0.001 significance level.
Similarly, the test statistic for multivariate kurtosis 2 , which equals 59.21, is greater than the value of
the Gaussian distribution quantile, which is 30.46 for 24 mean, 4.36 variance, and 0.001 significance
level. We apply the above test statistics because the number of rows of multi-dimensional data N is
greater than 20. Notice that otherwise, other test corrected statistics should be used according to [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ].
      </p>
      <p>
        The above results indicate to us the need for the application of the method to determine
multidimensional outliers in multivariate non-Gaussian data further. To do this we apply the statistical
technique [
        <xref ref-type="bibr" rid="ref15 ref16">15, 16</xref>
        ] based on the normalizing transformations and the MSD. In the beginning, we
normalize the four-dimensional non-Gaussian data set from Table 1 by the Johnson four-variate
transformation for the SB family, for which the components are defined using
where Z j is a standard Gaussian variable, Z j  N0,1 ;  j , j , j , and  j are parameters of the
Johnson transformation for SB family, j  0 ,  j  0 , j  1,2,3 . The component ZY is defined
analogously (2) with the only difference that instead of Z j , X j ,  j , j , j ,  j should be put
respectively ZY , Y, Y , Y , Y , Y . The estimates of parameters of the Johnson four-variate
transformation for the SB family for the data from Table 1 are calculated by the maximum likelihood
method
θˆ  argmax lP,θ ,
      </p>
      <p>θ
where the log-likelihood function is
lP,θ  NlnY Y  Nlnj j  Nk  1ln2  N lndetST 
k
j1 2 2
 i1 lnYi  Y Y  Y Yi  j1 i1 lnX ji  j j   j  X ji  12 iN1 TiT ST1Ti . (4)</p>
      <p>N k N
Here ST is the sample covariance matrix for the components of T. Notice in (4), we take into account
that for the Johnson multivariate translation system, the components of the mean vector are equal to
zero.</p>
      <p>The estimates of parameters of the Johnson four-variate transformation for SB family for the data
from Table 1 are ˆY  1.61708 , ˆ1  2.17927 , ˆ2  13.84357 , ˆ3  0.860885 , ˆY  0.538609 ,
ˆ1  0.612224 , ˆ2  1.87447 , ˆ3  0.90490 , ˆY  0.2810 , ˆ1  0.124604 , ˆ2  1.29688 ,
ˆ3  1.50142 , ˆY  201.7538 , ˆ1  3128.508 , ˆ2  11606.794 , ˆ 3  12.0339 . The sample covariance
matrix ST is following</p>
      <p>ST 
.</p>
      <p>
        We tested the normality of normalized four-dimensional data from Table 1 by the Mardia test based
on the measures of the multivariate skewness 1 and kurtosis 2 [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ]. According to the Mardia test,
the four-variate distribution of normalized four-dimensional data from Table 1 is not Gaussian since
the test statistic for multivariate skewness N 1 6 of this data, which equals 61.97, is greater than the
value of the Chi-Square distribution quantile, which is 45.31 for 20 degrees of freedom and 0.001
significance level. Similarly, the test statistic for multivariate kurtosis 2 , which equals 31.35, is greater
than the value of the Gaussian distribution quantile, which is 30.46 for 24 mean, 4.364 variance, and
0.001 significance level.
      </p>
      <p>
        According to [
        <xref ref-type="bibr" rid="ref15 ref16">15, 16</xref>
        ], row 2 from Table 1 is the multivariate outlier in four-dimensional
nonGaussian data since the MSD value for normalized data of row 2, which equals 19.71, is greater than
the value of the Chi-Square distribution quantile, which equals 14.86 for the 0.005 significance level.
After that, we discard the outlier and go to step 1. The first iteration is completed.
      </p>
      <p>Next, we normalize the four-dimensional non-Gaussian data set from 43 rows of Table 1 (without
row 2) by the Johnson four-variate transformation for the SB family using (2). In this case, the estimates
of parameters of the Johnson four-variate transformation for SB family for the data from Table 1
(without row 2) are ˆY  1.75572 , ˆ1  2.19844 , ˆ2  15,3097 , ˆ3  0.798411 , ˆY  0.559115 ,
ˆ1  0.610394 , ˆ2  2.10956 , ˆ3  0.893772 , ˆY  0.2810 , ˆ1  0.239008 , ˆ2  2.07165 ,
ˆ3  1.49364 , ˆY  212.2741 , ˆ 1  3116,897 , ˆ2  11606.864 , ˆ 3  11.8220 . The sample covariance
matrix ST is following</p>
      <p>ST 
.</p>
      <p>
        We tested the normality of normalized four-dimensional data from Table 1 (without rows 2) by the
Mardia test based on the measures of the multivariate skewness 1 and kurtosis 2 [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ]. According to
the Mardia test, the four-variate distribution of normalized four-dimensional data from Table 1 (without
rows 2) is Gaussian since, first, the test statistic for multivariate skewness N 1 6 of this data, which
equals 38.79, is less than the value of the Chi-Square distribution quantile, which is 45.31 for 20 degrees
of freedom and 0.001 significance level, second, the test statistic for multivariate kurtosis 2 , which
equals 28.73, is less than the value of the Gaussian distribution quantile, which is 30.53 for 24 mean,
4.465 variance, and 0.001 significance level.
      </p>
      <p>
        According to [
        <xref ref-type="bibr" rid="ref15 ref16">15, 16</xref>
        ], there is no multivariate outlier in the four-dimensional non-Gaussian data set
from 43 rows of Table 1 (without row 2) at the second iteration since the MSD values for normalized
data of 43 rows are less than the value of the Chi-Square distribution quantile, which equals 14.86 for
the 0.005 significance level. Therefore, we go to step 3.
      </p>
      <p>We build the linear regression model for normalized data in the form</p>
      <p>ZY  ZˆY    bˆ0  bˆ1Z1  bˆ2Z2  bˆ3Z3   ,
(5)
random variable to describe residuals,   N0,2 ,  is the standard deviation of  with the estimate
ˆ  of 0.2017.</p>
      <p>To test the normality of the distribution of residuals in the linear regression model (5), we apply the
Pearson chi-squared test. According to this test, the null hypothesis H0 that the observed frequency
distribution of residuals is the same as the normal distribution is accepted because the 2 test statistic,
which equals 1.92, does not surpass the critical value from the Chi-Squared distribution which is 7.82
for 3 degrees of freedom and 0.05 significance level. Therefore, we go to step 5.</p>
      <p>
        According to [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ], we construct the nonlinear regression model based on the Johnson four-variate
transformation for the SB family and the linear regression model (5) for normalized data from 43 rows
of Table 1 (without row 2). This nonlinear regression model has the form [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]
      </p>
      <p>Y  ˆY  ˆY 1  eZˆY ˆY  ˆY 1
with estimates of parameters in the second iteration.</p>
      <p>
        Next, according to [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ], we build the prediction interval of nonlinear regression as
      </p>
      <p> 
Y1 ZˆY  t 2,SZY 1 
</p>
      <p>N1  zX T SZ1 zX 1 2  ,
where Y is the transformation (1) for Y, Y1  Y  Y 1  eZY Y  Y  ; t 2, is a student's
tdistribution quantile with  2 significance level and  degrees of freedom;   N  k  1 ; k is a number
of independent variables (in our case, k is 3); zX is a vector with components Z1i  Z1 , Z2i  Z2 , ,
Zki  Zk for i-row; Z j 
1 N
N i1 Z ji , j  1,2,,k ; SZ2Y 


</p>
      <p>SZ1Zk 
SZ2Zk  ,
 </p>
      <p></p>
      <p>SZkZk 
 SZ1Z1
 SZ1Z2
SZ   
 SZ1Zk</p>
      <p>SZ1Z2
SZ2Z2
</p>
      <p>SZ2Zk
 43.00

SZ    15.99
 15.63</p>
      <p>N
where SZqZr  i1 Zqi  Zq Zri  Zr , q,r  1,2,,k .</p>
      <p>In the second iteration, for the data normalized by the Johnson four-variate transformation for SB
family from 43 rows of Table 1 (without row 2), the matrix (8) is following</p>
      <p>
        It turned out that only two Y values for rows 35 and 44 are outside the prediction interval, which was
calculated by (7) for a significance level of 0.05. In Table 2, the lower and upper bounds of the
prediction interval obtained in the second iteration are denoted as LB2, and UB2, respectively. In Table
2, as in [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ], the row numbers with the outliers in data are highlighted in bold at the relevant iteration,
and a dash (-) depicts the exception of the corresponding numbers of data at the relevant iteration.
      </p>
      <p>Next, we discard two outliers (rows 35 and 44). The second iteration is completed. We start the third
iteration and go to step 1. In step 1, we normalize the four-dimensional non-Gaussian data set from 41
rows of Table 1 (without rows 2, 25, and 44) by the Johnson four-variate transformation for the SB
family using (2). In this case the estimates of parameters of the Johnson four-variate transformation for
SB family for the data from Table 1 (without rows 2, 25, and 44) are ˆY  3.01458 , ˆ1  2.87815 ,
ˆ2  16.5902 , ˆ3  0.710304 , ˆY  0.698176 , ˆ1  0.652161 , ˆ2  2.322625 , ˆ3  0.878073 ,
1 N 2
 i1 ZYi  ZˆYi  ; SZ is the k  k matrix

(6)
(7)
(8)
ˆ 2  11672.25 , ˆ 3  11.5940 . The sample covariance matrix ST is following</p>
      <p>
        We tested the normality of normalized four-dimensional data from Table 1 (without rows 2, 25, and
44) by the Mardia test based on the measures of the multivariate skewness 1 and kurtosis 2 [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ].
According to the Mardia test, the four-variate distribution of normalized four-dimensional data from
Table 1 (without rows 2, 25, and 44) is Gaussian since, first, the test statistic for multivariate skewness
N 1 6 of this data, which equals 28.66, is less than the value of the Chi-Square distribution quantile,
which is 45.31 for 20 degrees of freedom and 0.001 significance level, second, the test statistic for
multivariate kurtosis 2 , which equals 26.06, is less than the value of the Gaussian distribution quantile,
which is 30.69 for 24 mean, 4.683 variance, and 0.001 significance level.
      </p>
      <p>
        According to [
        <xref ref-type="bibr" rid="ref15 ref16">15, 16</xref>
        ], there is no multivariate outlier in the four-dimensional non-Gaussian data set
from 41 rows of Table 1 (without rows 2, 25, and 44) at the third iteration since the MSD values for
normalized data of 41 rows are less than the value of the Chi-Square distribution quantile, which equals
14.86 for the 0.005 significance level. Therefore, we go to step 3.
      </p>
      <p>In step 3, we build the linear regression model (5) whose parameter estimates bˆ0 , bˆ1 , bˆ2 , bˆ3 are
equal to 0, 1.100603, 0.533514, and -0.032605, respectively. The estimate ˆ  is 0.1571.</p>
      <p>Next, we test the normality of the distribution of residuals in the linear regression model (5) by the
Pearson chi-squared test. According to this test, the null hypothesis H0 that the observed frequency
distribution of residuals is the same as the normal distribution is rejected because the 2 test statistic,
which equals 11.35, surpasses the critical value from the Chi-Squared distribution which is 7.82 for 3
degrees of freedom and 0.05 significance level. Therefore, we discard the raw 7 for which the value of
the modulus of residual in the model is maximum and equal to 0.2999. The third iteration is completed
and we go to step 1.</p>
      <p>In step 1 of the fourth iteration, we normalize the four-dimensional non-Gaussian data set from 40
rows of Table 1 (without rows 2, 7, 25, and 44) by the Johnson four-variate transformation for the SB
family using (2). In this case, the estimates of parameters of the Johnson four-variate transformation for
SB family for the data from Table 1 (without rows 2, 7, 25, and 44) are ˆY  2.331506 , ˆ1  2.502955 ,
ˆ2  16.23225 , ˆ3  0.628931 , ˆY  0.670065 , ˆ1  0.638907 , ˆ2  2.263781 , ˆ3  0.814678 ,
ˆY  0.131886 , ˆ1  0.091614 , ˆ2  2.51524 , ˆ3  1.559526 , ˆY  330.6481 , ˆ1  4464.000 ,
ˆ2  11714.039 , ˆ 3  11.1327 . The sample covariance matrix ST is following</p>
      <p>
        We tested the normality of normalized four-dimensional data from Table 1 (without rows 2, 7, 25,
and 44) by the Mardia test based on the measures of the multivariate skewness 1 and kurtosis 2 [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ].
According to the Mardia test, the four-variate distribution of normalized four-dimensional data from
Table 1 (without rows 2, 7, 25, and 44) is Gaussian since, first, the test statistic for multivariate skewness
N 1 6 of this data, which equals 28.95, is less than the value of the Chi-Square distribution quantile,
which is 45.31 for 20 degrees of freedom and 0.001 significance level, second, the test statistic for
multivariate kurtosis 2 , which equals 26.22, is less than the value of the Gaussian distribution quantile,
which is 30.77 for 24 mean, 4.8 variance, and 0.001 significance level. We apply the above test statistics
because the number of rows of multi-dimensional data N is greater than 20. Notice that otherwise, other
test corrected statistics should be used according to [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ].
      </p>
      <p>
        According to [
        <xref ref-type="bibr" rid="ref15 ref16">15, 16</xref>
        ], there is no multivariate outlier in the four-dimensional non-Gaussian data set
from 40 rows of Table 1 (without rows 2, 7, 25, and 44) at the fourth iteration since the MSD values
for normalized data of 40 rows are less than the value of the Chi-Square distribution quantile, which
equals 14.86 for the 0.005 significance level. Therefore, we go to step 3.
      </p>
      <p>In step 3 we build the linear regression model (5) whose parameter estimates bˆ0 , bˆ1 , bˆ2 , bˆ3 are
equal to 0, 1.130412, 0.547413, and -0.041326, respectively. The estimate ˆ  is 0.1525.</p>
      <p>Next, we test the normality of the distribution of residuals in the linear regression model (5) by the
Pearson chi-squared test. According to this test, the null hypothesis H0 that the observed frequency
distribution of residuals is the same as the normal distribution is accepted because the 2 test statistic,
which equals 2.69, does not surpass the critical value from the Chi-Squared distribution which is 7.82
for 3 degrees of freedom and 0.05 significance level. Therefore, we go to step 5. We construct the
nonlinear regression model (4) with estimates of parameters in the fourth iteration.</p>
      <p>After that, we build the prediction interval of nonlinear regression by (7). In the fourth iteration, for
the data normalized by the Johnson four-variate transformation for SB family from 40 rows of Table 1
(without rows 2, 7, 25, and 44), the matrix (8) is following</p>
      <p>In Table 2, the lower and upper bounds of the prediction interval obtained in the fourth iteration are
denoted as LB4 and UB4, respectively. At the fourth iteration, there are no outliers; the repeat of the
iterations is completed, and the nonlinear regression model (6) is constructed using data from 40 apps.</p>
      <p>Note, that the constructed nonlinear regression model (6) allows performing early size estimation
(in KLOC) of open source apps in PHP based on the above three metrics (independent variables X1, X2,
and X3) that may be derived from a class diagram. In model (6) size Y is the non-Gaussian dependent
random variable and  is the Gaussian random variable with zero mean and a standard deviation
estimate ˆ  of 0.1525.</p>
      <p>
        To judge the prediction accuracy of the nonlinear regression model (6) we used the well-known
prediction accuracy metrics such as a multiple coefficient of determination R2, a mean magnitude of
relative error MMRE, and prediction percentage at the level of magnitude of relative error of 0.25,
PRED(0.25) [
        <xref ref-type="bibr" rid="ref19 ref20">19, 20</xref>
        ]. The R2, MMRE, and PRED(0.25) values equal respectively 0.9812, 0.1753, and
0.750 for the nonlinear regression model (6) with the estimators of parameters which are calculated for
the 40 data rows from Table 1 (without rows 2, 7, 25 and 44). These values indicate to us good prediction
results of the nonlinear regression model (6), which is constructed by a modified technique for
parameter estimators calculated from the 40 data rows.
      </p>
      <p>
        Notice that if the nonlinear regression model (6) is built based on the data set from Table 1 by the
technique [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] then we have three iterations. At the third iteration, there are no outliers in step 5, the
repeat of the iterations is completed, and the nonlinear regression model (6) is constructed using data
from 41 rows of Table 1 (without rows 2, 25, and 44). In this case, the R2, MMRE, and PRED(0.25)
values equal respectively 0.9776, 0.1795, and 0.6829 for the nonlinear regression model (6) with the
estimates of parameters which are calculated for the 41 data rows from Table 1 (without rows 2, 25 and
44). It is not hard to see the values of all prediction accuracy metrics for the nonlinear regression model
(6) which is constructed by the technique [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] are worse than in the case of using a modified technique
we propose. This is especially true for PRED(0.25). Its value indicates unsatisfactory prediction
accuracy since the recommended minimum value of PRED(0.25) should not be less than 0.75 [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ].
      </p>
      <p>Note, a more significant advantage of the model (6) constructed by a modified technique compared
to the same model constructed using a nonmodified technique is the smaller widths of the confidence
and prediction intervals. The width of the nonlinear regression prediction interval after modification is
less than before modification by 67% for app 1 (row 1). Also, the width of the nonlinear regression
confidence interval after modification is less than before modification by 57% for app 1. App 1 is
Symfony, a PHP framework for web and console apps.</p>
      <p>Such better prediction results for the nonlinear regression model (6), which is constructed by a
modified technique for parameter estimates calculated from the 40 data rows by the maximum
likelihood method (3) with the log-likelihood function (4) might be explained by, firstly, the normality
of four-variate distribution of normalized four-dimensional data, secondly, the bigger number of data
that has been discarded as outliers, and, thirdly, the normality of distribution of residuals in the linear
regression model for normalized data.</p>
    </sec>
    <sec id="sec-4">
      <title>4. Discussion and interpretations</title>
      <p>
        The four-variate distribution of the software metrics from Table 1 is not Gaussian which the Mardia
test for multivariate normality based on measures of the multivariate skewness and kurtosis indicates.
Because we apply the statistical technique [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ] to detect multivariate outliers in the four-dimensional
non-Gaussian data from Table I based on the multivariate normalizing transformations and the MSD
for normalized data. According to [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ], there is a multivariate outlier in four-dimensional non-Gaussian
data from Table 1 (data row 2) based on the Johnson four-variate transformation for the SB family. Note,
that we have the multivariate outlier in the data from Table 1 (data row 1) without using normalization.
This may be explained by the four-variate distribution of the data from Table 1 distribution differs
significantly from Gaussian. That is why we rejected row 2 as a multivariate outlier.
      </p>
      <p>
        We apply the Johnson four-variate transformation for the SB family to build the nonlinear regression
model for estimating the size of open source PHP-based apps based on the modified technique [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] since
there are outliers in the data from Table 1, which are detected in the model construction process by the
MSD values for normalized data, nonlinear regression prediction intervals, and residuals (see Table 2).
      </p>
      <p>
        We detected four outliers in the model construction process (rows 2, 7, 35, and 44). These data
outliers are derived for the following four various apps: Faker, cphalcon, country-master, and
PHP_CodeSniffer. Faker is a PHP library that generates fake data for users [
        <xref ref-type="bibr" rid="ref21">21</xref>
        ]. Phalcon is an
opensource web framework delivered as a C language extension for the PHP language providing high
performance and lower resource consumption [
        <xref ref-type="bibr" rid="ref22">22</xref>
        ]. Country-master is an Android library project to
create a login and registration form using phone numbers (uses Google's libphonenumber) [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ].
PHP_CodeSniffer is a set of two PHP scripts that ensure your code remains clean and consistent [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ].
The data of these so different apps are united by the fact that the average number of methods to the
number of classes (predictor X2) for them is in a fairly narrow range from 2.240 to 3.125. Therefore, we
propose to divide the range of change of the predictor X2 into two ranges: the first, from 0.48 to 2.22,
and the second, from 3.14 to 32.60.
      </p>
      <p>We propose to apply the nonlinear regression model (6) for estimating the size (in KLOC) of open
source PHP-based apps such as frameworks, libs, and scripts around the following predictor ranges:
from 2 to 2075 for X1 (number of classes), from 0.48 to 2.22 or from 3.14 to 32.60 for X2, and from 1.76
to 11.48 for X3 (sum of average afferent and efferent coupling per class).</p>
      <p>Also, we additionally calculated MRE (magnitude of relative error) values for prediction regression
results by the model (6) for well-known PHP frameworks Symfony, Yii2, Laravel, and CakePHP
(respectively data rows 1, 3, 6, and 10), which equal 0.0322, 0.2595, 0.1581, and 0.1748. Note, MRE
values for prediction regression results by the model (6) before using the modified technique for the
same PHP frameworks are 0.0775, 0.2748, 0.2594, and 0.1753, respectively. The above calculations
indicate better prediction results of the size of open source PHP-based apps by the nonlinear regression
model (6), which was constructed using the modified technique.</p>
    </sec>
    <sec id="sec-5">
      <title>5. Conclusions</title>
      <p>
        The technique for constructing nonlinear regression models based on the multivariate normalizing
transformations and prediction intervals is modified by testing the normality of error distribution in the
linear regression model for normalized data, which is applied to construct the nonlinear one. Using the
example of building the nonlinear regression model with three predictors based on the Johnson
fourvariate transformation for the SB family to estimate the size (in KLOC) of open source PHP-based apps
(such as frameworks, libs, and scripts), we have demonstrated that there may be multidimensional data
sets (software metrics) for which the constructed nonlinear regression model has an unsatisfactory
prediction accuracy of software size even after all outliers are removed as in the technique [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. From
the example, we have concluded that a modified technique is promising to apply, including its
application for the construction of nonlinear regression models to estimate a software size. The
nonlinear regression model, which is constructed by a modified technique, has better values of
wellknown prediction accuracy metrics such as R2, MMRE, and PRED(0.25) compared to the model, which
is only constructed based on the multivariate normalizing transformation and prediction intervals
(without testing the normality of error distribution in the linear regression model for normalized data
and discarding the multidimensional data point for which the value of the error modulus in the linear
model is maximum if the distribution of residuals in the linear one is not Gaussian). Also, calculated
MRE values for well-known PHP frameworks (Symfony, Yii2, Laravel, and CakePHP) indicate better
prediction results of the size of open source PHP-based apps by the nonlinear regression model, which
was constructed using the modified technique. Further research may be directed to search other
multidimensional data sets to confirm that the modified technique works we have proposed, including
the construction of nonlinear regression models for estimating the size of other types of apps and
applying other multivariate normalizing transformations.
6. References
      </p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>C. M.</given-names>
            <surname>Bishop</surname>
          </string-name>
          ,
          <source>Pattern Recognition and Machine Learning</source>
          , Springer-Verlag, New York,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>D. M.</given-names>
            <surname>Bates</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. G.</given-names>
            <surname>Watts</surname>
          </string-name>
          ,
          <source>Nonlinear Regression Analysis and Its Applications</source>
          , 2nd ed., John Wiley &amp; Sons, New York,
          <year>1988</year>
          . doi:
          <volume>10</volume>
          .1002/9780470316757.
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>G. A. F.</given-names>
            <surname>Seber</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C. J.</given-names>
            <surname>Wild</surname>
          </string-name>
          , Nonlinear Regression, John Wiley &amp; Sons, New York,
          <year>1989</year>
          . doi:
          <volume>10</volume>
          .1002/0471725315.
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>T. P.</given-names>
            <surname>Ryan</surname>
          </string-name>
          , Modern Regression Methods, John Wiley &amp; Sons, New York,
          <year>1997</year>
          . doi:
          <volume>10</volume>
          .1002/9780470382806.
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>N. R.</given-names>
            <surname>Drapper</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Smith</surname>
          </string-name>
          , Applied Regression Analysis, 3rd ed., John Wiley &amp; Sons, New York
          <year>1998</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>R. A.</given-names>
            <surname>Johnson</surname>
          </string-name>
          , D. W. Wichern, Applied Multivariate Statistical Analysis,
          <source>Pearson Prentice Hall</source>
          ,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>S.</given-names>
            <surname>Chatterjee</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. S.</given-names>
            <surname>Simonoff</surname>
          </string-name>
          ,
          <source>Handbook of Regression Analysis</source>
          , John Wiley &amp; Sons, New York,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>S.</given-names>
            <surname>Prykhodko</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Prykhodko</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Makarova</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Pukhalevych</surname>
          </string-name>
          ,
          <article-title>Outlier detection in non-linear regression analysis based on the normalizing transformations</article-title>
          ,
          <source>in: Proceedings of the 15th International Conference on Advanced Trends in Radioelectronics</source>
          , Telecommunications and Computer Engineering (TCSET), IEEE,
          <year>2020</year>
          , pp.
          <fpage>407</fpage>
          -
          <lpage>410</lpage>
          . doi:
          <volume>10</volume>
          .1109/TCSET49122.
          <year>2020</year>
          .
          <volume>235464</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>S.</given-names>
            <surname>Prykhodko</surname>
          </string-name>
          , N. Prykhodko,
          <article-title>Mathematical modeling of non-Gaussian dependent random variables by nonlinear regression models based on the multivariate normalizing transformations</article-title>
          , in: S. Shkarlet,
          <string-name>
            <given-names>A.</given-names>
            <surname>Morozov</surname>
          </string-name>
          ,
          <string-name>
            <surname>A</surname>
          </string-name>
          . Palagin (eds.)
          <source>Mathematical Modeling and Simulation of Systems (MODS'</source>
          <year>2020</year>
          ),
          <source>MODS</source>
          <year>2020</year>
          , volume
          <volume>1265</volume>
          <source>of Advances in Intelligent Systems and Computing</source>
          , Springer, Cham.,
          <year>2021</year>
          , pp.
          <fpage>166</fpage>
          -
          <lpage>174</lpage>
          . doi:
          <volume>10</volume>
          .1007/978-3-
          <fpage>030</fpage>
          -58124-4_
          <fpage>16</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>H. J.</given-names>
            <surname>Motulsky</surname>
          </string-name>
          ,
          <string-name>
            <surname>R. E. Brown,</surname>
          </string-name>
          <article-title>Detecting outliers when fitting data with nonlinear regression - a new method based on robust nonlinear regression and the false discovery rate</article-title>
          .
          <source>BMC Bioinformatics 7</source>
          ,
          <issue>123</issue>
          (
          <year>2006</year>
          )
          <fpage>1</fpage>
          -
          <lpage>20</lpage>
          . doi:
          <volume>10</volume>
          .1186/
          <fpage>1471</fpage>
          -2105-7-123.
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>D. C.</given-names>
            <surname>Montgomery</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E. A.</given-names>
            <surname>Peck</surname>
          </string-name>
          , Introduction to Linear Regression Analysis, 2nd ed., John Wiley &amp; Sons,
          <year>1992</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>E. J.</given-names>
            <surname>Pedhazur</surname>
          </string-name>
          , Multiple Regression in Behavioral Research: Explanation and Prediction, 3rd ed.,
          <string-name>
            <surname>Fort</surname>
            <given-names>Worth</given-names>
          </string-name>
          , Harcourt Brace College Publishers,
          <year>1997</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>G. A. F.</given-names>
            <surname>Seber</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. J.</given-names>
            <surname>Lee</surname>
          </string-name>
          ,
          <string-name>
            <surname>Linear Regression Analysis</surname>
          </string-name>
          , 2nd ed., John Wiley &amp; Sons,
          <year>2003</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>S.</given-names>
            <surname>Weisberg</surname>
          </string-name>
          , Applied Linear Regression, 3rd ed., John Wiley &amp; Sons,
          <year>2005</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>S.</given-names>
            <surname>Prykhodko</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Prykhodko</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Makarova</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Pugachenko</surname>
          </string-name>
          ,
          <article-title>Detecting outliers in multivariate nonGaussian data on the basis of normalizing transformations</article-title>
          ,
          <source>in: Proceedings of the First Ukraine Conference on Electrical and Computer</source>
          Engineering (UKRCON), IEEE,
          <year>2017</year>
          , pp.
          <fpage>846</fpage>
          -
          <lpage>849</lpage>
          . doi:
          <volume>10</volume>
          .1109/UKRCON.
          <year>2017</year>
          .
          <volume>8100366</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>S.</given-names>
            <surname>Prykhodko</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Prykhodko</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Makarova</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Pukhalevych</surname>
          </string-name>
          ,
          <article-title>Application of the squared Mahalanobis distance for detecting outliers in multivariate non-Gaussian data</article-title>
          ,
          <source>in: Proceedings of the 14th International Conference on Advanced Trends in Radioelecrtronics</source>
          , Telecommunications and Computer Engineering (TCSET), IEEE,
          <year>2018</year>
          , pp.
          <fpage>962</fpage>
          -
          <lpage>965</lpage>
          , doi: 10.1109/TCSET.
          <year>2018</year>
          .
          <volume>8336353</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>K. V.</given-names>
            <surname>Mardia</surname>
          </string-name>
          ,
          <article-title>Measures of multivariate skewness and kurtosis with applications</article-title>
          ,
          <source>Biometrika</source>
          <volume>3</volume>
          (
          <issue>57</issue>
          ), (
          <year>1970</year>
          )
          <fpage>519</fpage>
          -
          <lpage>530</lpage>
          . doi:
          <volume>10</volume>
          .1093/biomet/57.3.519.
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>K. V.</given-names>
            <surname>Mardia</surname>
          </string-name>
          ,
          <article-title>Applications of some measures of multivariate skewness and kurtosis in testing normality and robustness studies</article-title>
          ,
          <source>Sankhya: The Indian Journal of Statistics</source>
          , Series B (
          <year>1960</year>
          -2002),
          <volume>36</volume>
          (
          <issue>2</issue>
          ) (
          <year>1974</year>
          )
          <fpage>115</fpage>
          -
          <lpage>128</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>L. C.</given-names>
            <surname>Briand</surname>
          </string-name>
          ,
          <string-name>
            <surname>I. Wieczorek</surname>
          </string-name>
          ,
          <article-title>Resource modeling in software engineering</article-title>
          , in: J.
          <string-name>
            <surname>Marciniak</surname>
          </string-name>
          (Ed.),
          <source>Encyclopedia of Software Engineering</source>
          , 2nd ed. John Wiley &amp; Son, New York,
          <year>2002</year>
          , pp.
          <fpage>1160</fpage>
          -
          <lpage>1196</lpage>
          . doi:
          <volume>10</volume>
          .1002/0471028959.sof282.
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>D.</given-names>
            <surname>Port</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Korte</surname>
          </string-name>
          ,
          <article-title>Comparative studies of the model evaluation criterions MMRE and PRED in software cost estimation research</article-title>
          ,
          <source>in: Proceedings of the 2nd ACM-IEEE International Symposium on Empirical Software Engineering and Measurement</source>
          , ACM, New York,
          <year>2008</year>
          , pp.
          <fpage>51</fpage>
          -
          <lpage>60</lpage>
          . doi:
          <volume>10</volume>
          .1145/1414004.1414015.
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21] fzaninotto/Faker,
          <year>2020</year>
          . URL: https://github.com/fzaninotto/Faker.
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22] phalcon/cphalcon,
          <year>2021</year>
          . URL: https://github.com/phalcon/cphalcon.
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23] uglymittens/country-master,
          <year>2015</year>
          . URL: https://github.com/uglymittens/country-master.
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24] squizlabs/PHP_CodeSniffer,
          <year>2021</year>
          . URL: https://github.com/squizlabs/PHP_CodeSniffer.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>