This work provides an alternative way to preprocessing procedure for consolidated data. Two methods are proposed. The first one is used for feature selection based on ensemble of machine learning algorithms. And the second one organizes missing data imputation based on combination of functional dependencies and associative rules. Ensemble methods for processing multimodal data based on a hierarchical classifier, a set of weak classifiers and a number of methods for selecting important characteristics with a much higher value of accuracy on unbalanced data sets compared to existing machine learning methods are developed. The methods are validated on medical dataset. The percentage of recovery data is on 1.2% comparing with associative rules. The proposed missing data imputation method creates additional data values operating a based domain and functional dependencies and includes these values to available training data. The correctness of the filled-in values is proved on the predictor built on the original dataset. The proposed PPD method conducts 12% better than RF and EM models for 30% missing data.
are not used inefficiently. Thus, there is a problem of developing an effective strategy of data analysis that can be applied to dispersed databases of various domains.
Data analysis methods are effectively used on clear and previously processed data. That is why the paper aimed to analyses and develop methods for data preprocessing. Particularly, we have taken into account feature selection (feature engineering) and missing data imputation.
The main contribution of the paper is as follows: 1. A new hybrid ensemble feature selection model for a machine learning-based post-COVID19 prediction system is proposed as an automatic feature cut-off rank identifier.
2. A method based on probabilistic dependence is developed and tested. The percentage of recovery data is on 1.2% comparing with associative rules.
3. Development of a method for finding dependencies in large data sets with gaps and uncertainties based on an ensemble of clustering methods and auto-associative dependencies. Boruta, Decision Tree and Random Forest are used to select an object. The importance of variables is different for different methods (logistic regression, Support vector machine, Naive Bayes, XGBoost, Random Forest, neural network, decision tree). This means that the relationship between the parameters is supported only for part of the data set. That is why we propose to find the dependency for separate clusters and use this dependence for classification.
Selecting the appropriated features can be a more significant task than lessening computation time
or enhancing classification or predictive accuracy. For example, in medicine [
Selecting important features (for example, determining genes appropriate for a specific type of cancer) can help decipher the mechanisms underlying the welfare problem for research. A complete enumeration of features can implement the selection method, that is, having checked all possible sets, selecting those signs for which the error is minimal. This method is simple to implement, but it is entirely ineffective on big data. Therefore, in this case, other algorithms are most often used.
The three primary classes of feature extract algorithms – filters, wrappers, and built-in algorithms
are used [
Filters are based on some metrics that are independent of the classification method. For example,
the correlation of features with the target vector and information content criteria. They are applied
before classification. The most significant benefit of filtering is that it can be used as preprocessing to
reduce space dimensionality and overcome overfitting. Filtering methods are generally fast. Filters are
used to choose features in clustering or to build an initial approximation [
Embedded algorithms organize feature extraction during the classifier training approach, and it is
they explicitly optimize the set of features used to achieve better accuracy [
Wrappers rely on feature significance information from several classifications or regression
models and can thus find deeper patterns in dataset than filters. Wrappers can be built on any
classifier that defines the degree of significance of the features [
Imputation is a procedure for assessing unknown or missing values based on available data, which allows you to form a complete set of data with some plausible estimates.
Methods of imputation are divided into non-model and model-based approaches. There are
approaches based on single and multiple imputation methods in terms of the quantity of values
received from imputation methods [
The method based on mean substitution solves the problem of incomplete data by replacing each
missing variable with an average value. There are the following types of substitutions: median value,
mean value for subgroup [
Hot-deck imputation approach replaces each gap with a random value taken from an existing
dataset [
Cold-deck imputation (CD) approach implements the replacement of each gap by some constant
value from an external source [
The regression model applies to replace data gaps with expected values emanated from a regression equation constructed from a complete dataset. Disadvantages of regression completion include the need to accurately define regression models, exaggerate correlation and covariance, the likelihood of moving to predicted values outside the logical sequence, and the need for extensive quantities of data to obtain consistent estimates.
The association rules (AR) mining method uses the constructed associative rules for the data
imputation [
Diabetes increases the likelihood of severe COVID19. New clinical data and investigations show that this might work in the opposite direction: scientists are recording new cases where COVID-19 has sharply provoked type 1 diabetes in humans. The World Health Organization views diabetes as one of the existing diseases, on a par with respectable age, making someone more vulnerable to severe COVID 19 infection. Cellular immunity is essential for protecting against viral diseases, and its effectiveness decreases with age. In particular, a decrease in T-cell receptors explains the significant increase in mortality of COVID19 with age.
Therefore, another important factor is the search for possible relationships between biomarkers of aging and COVID resistance.
Existing research and the data sets collected for this use standard machine learning methods, while
demonstrating not very high accuracy of prediction. Thus, in the paper [
The analysis of literature sources showed the lack of a comprehensive approach to solving the problem of prolonging the active period of life and preventing exacerbation of chronic diseases. At the same time, as we see in the case of COVID, chronic diseases (diabetes and obesity) can not only reduce the ability to work, but also increase the likelihood of severe course of other diseases.
Biomarkers of aging are used to predict possible changes in the body that lead to disability due to functional age-related changes. Biomarkers of aging are markers that can predict the functional capacity of an organism at a certain age better than chronological age. Since the immune system is a change the language here for English version
leading factor in aging, the main impact of which is realized through increased inflammation and reduced effectiveness of cellular immunity, it becomes clear the need to involve relevant markers to develop interventions to increase the duration of healthy longevity.
Thus, control of chronic age-related diseases (diabetes and obesity) and biological markers can be used to predict functional changes in the body, and analysis of other personal indicators will determine how to reduce the negative effects of such changes by extending the period of active longevity.
Sociological research also shows that people in certain regions remain active for a long time and there are far fewer people who are obese. Therefore, it is also advisable to analyze the parameters of the environment and habitat and its impact on the parameters of the organism.
The baseline of the hybrid ensemble feature selection model looks like the following: Several selectors using, Aggregation of the results.
Several wrapper algorithms will be used in the preprocessing stage for the first stage.
The correlation matrix shows the numerical value of the correlation coefficient for all possible combinations of variables. It is mainly used to find out the relationship between more than two variables. The decision tree returns the feature weight as the criterion for evaluating features. It allows building a ranked list of selected features using different measures. Classification and Regression Trees (CART) was used for feature selection with Gini-index as a measure in our case.
Random Forest is an ensemble of numerous training-sensitive algorithms (decision trees). Mentioned approach has a slight compensation. The bias of the training method is the deviation of the average response of the trained algorithm from the reaction of the ideal algorithm. Each of these classifiers is built on a random subset of entities and a random subset of characters.
Boruta is a heuristic algorithm for choosing important features based on Random Forest approach
[
The Jaccard index [
The schema of the hybrid ensemble feature selection model is given in Fig. 1.
Next, the method of missing data imputation is developed. This method is based on classical functional dependencies in relational databases and association rules from non-relational databases. It consists of two parts: Probabilistic Production Dependencies mining; the Probabilistic Production Dependencies usage for missing data imputation.
Investigating extensive data requires specifying attribute clusters that form functional
dependencies (FD) [
Figure 1: The hybrid ensemble feature selection model.
Probabilistic Production Dependency is a dependence similar to associative rule in the primary
ratio selection that is proper for many entities. The significance threshold should be specified expertly
or based on estimation of the probability of erroneous selection of this dependence. The main
distinction between associative rules and PPD is that PPD will be generated from existing functional
dependencies (FD) in the dataset [
FI : K ai, ai A, D aj, aj A,:P k K d D p where k and d are the tuples of groups of attributes K and D , respectively.
The gaps and missing data presented among the values of the attribute Y of the relation r are classified using PPD. The following algorithm for PPD mining is proposed.
Algorithm 1. PPD mining algorithm
3. To calculate the Support and Confidence, Imputation of the tuple selected in step 2);
To fill in missing data, the PPD should be built.
Next, the novel algorithm for missing data imputation is developed.
Completeness=0
fill in empty values using PPDset
Next, a hierarchical classifier as a two-stage data prediction algorithm is developed. The first stage
is clustering; the next step is to build a classification model for each separate cluster. K-means [
To validate the proposed methods, dataset with medical data is used.
Dataset consists of 35 features and 122 instances collected from Lviv regional rehabilitation center for post-COVID patients with short- and long-term (more than 20 days) treatment and rehabilitation.
The personal data were removed from the dataset and replaced with unique random identifiers. The next feature, sex, is processed using one-hot encoding technics and in the final dataset is given in two components – female and male. Features like age, weight, height, BMI, CAT, pulse, the function of external respiration are taken as physiological parameters measured before inpatient treatment. The rest of the features were immune-based biomarkers as described below.
Zero cells (0-lymphocytes) do not carry T- and B-cells markers. Zero cells make 10–20 % of the total lymphocytes in human peripheral blood. Some researchers consider them immature or overripe T- or B-lymphocytes because they have a small number of antigens common to B- and T-cells. Zero cells include K-cells and NK-cells.
CD3+ is a surface marker specific to all T-lymphocyte subpopulation cells. By function, it belongs to the family of proteins that form a complex of membrane signaling associated with the T-cell receptor. Mature T-lymphocytes are "responsible" for cellular immune reactions and conduct immunological monitoring of antigenic homeostasis in the body.
CD4+ is a characteristic of helper T-cells; also represented on monocytes, macrophages, dendritic
cells. It binds to class II MHC molecules expressed on antigen-presenting cells, facilitating the
recognition of peptide antigens. Helper T-lymphocytes (CD4+) are helpers (inducers) of the immune
response, cells that regulate the strength of the body's immune response to a foreign antigen, as well
as control the stability of the body's internal environment (antigenic homeostasis) and cause increased
antibody synthesis [
CD8+ is a characteristic of suppressor and cytotoxic T-cells, NK-cells, mostly thymocytes. It is a T-cell activation receptor that facilitates the recognition of cell-bound class I MHC antigens.
CD16+ natural killers are part of innate immunity; they are involved in early response against viral infections and intracellular bacteria. Compared with cells of specific immunity (T- and Blymphocytes), they have the advantage that they do not require long-term activation. Besides, NKcells complement the action of T-cytotoxic cells can also regulate the immune response by producing various cytokines, including interferon-γ. They are the primary cells of antitumor protection. Their role is vital in manifesting cellular immunity in viral, protozoan, fungal and bacterial diseases caused by intracellular parasites. Their action is enhanced by interferon. The functions performed by natural killers can be divided into two main types: the production of cytokines that regulate the work of other cells of the immune system and the direct destruction of damaged cells.
Mature B lymphocytes express CD22+ markers. B-lymphocytes are responsible for the humoral adaptive immune response, primarily at removing extracellular infectious agents. After binding to a specific antigen, B-lymphocytes, in cooperation with T-lymphocytes and T-helpers proliferate, differentiate into plasma cells that secrete antibodies/immunoglobulins and memory cells. Defects of humoral immunity associated with the B-cells are sporadic, so a common hypoimmunoglobulinemia is mainly caused by other reasons.
CD4/CD8 immunoregulatory index reflects the ratio of CD4+ cells (T-helpers) to CD8+ cells
(Tcytotoxic cells). It is a relative indicator that has an indicative value. Its small increase or decrease has
no independent diagnostic value. Changes in the index significance the clinician to focus on the
reasons for the deviation of this index. The immunoregulatory index is assessed relative to the phase
of the immune response. In the period of exacerbation and remission of clinical manifestations, the
immunoregulatory index reaches high values due to the high percentage of T-helpers (CD4+ T-cells).
During the recovery period, the indicator's value decreases due to the increase in the level of CD8+
Tcells (killers). Violation of this pattern indicates the inadequacy of the immune response and the
possibility of chronic infection due to incomplete removal of the pathogen [
We implemented our approach in Rstudio. The essential packages we used were caret, rpart, Metrics, Boruta, Random forest, rules and ggplot2 for visualization. To generate PPD, the minimal support threshold equal to 0.0001 in the a priori algorithm is chosen. In addition, all rules with confidence below 0.001 are filtered out.
First, the predictive accuracy for whole dataset and selected features was analyzed (Table 1).
The much higher accuracy is obtained with SVM (Support vector machine, polynomial kernel) and artificial neural network (one hidden layer with 12 neurons in it, sigmoid activation function) for selected features [22]. The used measures are the following [23]: mean squared error MSE, mean absolute error MAE, R2.
Next, missing data imputation method is used. The developed method was compared with the existing ones: associative rules (AR), random forest (RF), support vector machine (SVM), multilayered perceptron (MLP), expectation-maximization (EM) and k-nearest neighbor (KNN) (Fig. 2). The recovery error is presented using normalized root-mean-square error (NRMSE).
The proposed missing data imputation method creates additional data values operating a based domain and functional dependencies and includes these values to available training data. The correctness of the filled-in values is proved on the predictor built on the original dataset. The proposed PPD method conducts 12% better than RF and EM models for 30% missing data.
Current trends in the development of information technology and databases are analyzed. As a result, unsolved problems in the field of dependency search in large databases of several subject areas, in particular, in medicine, were revealed. The analysis of existing methods and means of detection of dependences in data is carried out. This made it possible to identify a new subclass of dependencies Probabilistic Production dependencies. A method for deriving PPD in relational databases has been developed.
A new hybrid ensemble feature selection model for a machine learning-based post-COVID prediction system is proposed as an automatic feature cut-off rank identifier. Ensemble methods for processing of multimodal data based on a hierarchical classifier, a set of weak classifiers and a number of methods for selecting important characteristics with a much higher value of accuracy on unbalanced data sets compared to existing machine learning methods are developed.
The associative rules are found together with weak predictors usage to improve the classification quality.
The proposed missing data imputation method creates additional data values operating a based domain and functional dependencies and includes these values to available training data.
The correctness of the filled-in values is proved on the predictor built on the original dataset. The proposed PPD method conducts 12% better than RF and EM models for 30% missing data. The EM method looks the best for more additional missing data (the range of about 40%-50% missing data), and the PPD has equivalent results with the SVM (support vector machine).
Comparison of approaches for investigating and modeling the statistical processes by qualitative criteria verified the proposed method has the subsequent benefits: • retaining the characteristics of resistance to errors in the data; • allowing the parallel implementation in distributed databases; • automating and performing the analysis of various data types.
[22] D.A. Otchere, et al., Application of supervised machine learning paradigms in the prediction of petroleum reservoir properties: Comparative analysis of ANN and SVM models, Journal of Petroleum Science and Engineering 200 (2021): 108182. [23] D. Chicco, M. J. Warrens, G. Jurman, The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation. PeerJ Computer Science, 7, e623 (2021).