The paper presents the architecture of the platform for Big data preprocessing and processing. The method for prevention of the risk disease based on Probabilistic Production Dependencies is developed. The investigation is started from the platform for Big data in medical domain analysis. The platform consists of 6 layers: Data layer, Communication layers, Preprocessing layer, Data processing layer, User layer, Integrational layer. The platform architecture for Big data processing in medical sector is developed. The accuracy of the proposed method is estimated.
In recent years, the health policies are transforming form acoute intervention driven towards prevention and self-responsibility of the individuals by implementing measures for increasing health literacy. Particularly important for the implementation of the prevention programs are the literacy of young medical staff, the completeness and timeliness of receiving information about patients, the ability to observe the patient, not only in the hospital, but also at home. That is why integration of information technologies, intelligent devices and systems into all spheres of life takes place. It is important that the technologies used by the patient be created taking into account the requirements of the universal design, that is, without the need for adaptation or design changes if used by a person with special needs. The engineering branch is engaged in the design of "smart" homes, businesses and cities. Such complex intelligent systems collect process and transmit large amounts of data during their operation, including in the medical sector. In this process, microcontrollers, communication equipment, sensors, as well as ambulances, the next branch of a health facility, family doctors, etc., process the information coming from patients. Devices may have failures, as well as errors of the first and second kind, which leads to incomplete or inaccurate information about the patient, and, consequently, adversely affects the adoption of medical decisions.
The aim of the paper is analysis of the existing stage of data mining technics and Big data approach in EHealth and development of the novel technic for prevention of the risk disease based on Probabilistic Production Dependencies. 2
Data mining is widely used in medicine. Norman et al [
Currently, Hybrid Systems of Computational Intelligence are widely used to solve
a wide class of Medical Data Mining problems, combining the approximating and
extrapolating properties of artificial neural networks (ANNs), both traditional shallow
ones, and more advanced deep neural networks (DNNs), interpretability of the results
of fuzzy reasoning systems (FRS), the ability to find the best architectures for solving
specific problems, provided by using the apparatus of evolving systems [
In modified applications, the most problematic are the problems of diagnostics and early fault detection of controlling biomedical signals, in doing so comes to the fore not only the accuracy of the results obtained but also the time required to obtain them come to the fore, which leads to the need to abandon traditional mutiepoch learning.
Currently, to solve the mentioned problems, the most effective from the accuracy of the obtained results are deep neural networks, which are however completely ineffective in the conditions of short learning sets and unsuitable for operating in online mode, when data are fed to processing in real time.
Probabilistic neural networks (PNN) introduced by D. Specht [
Bhatt [
Therefore, data mining techniques are used to solve many of the problems of processing and analyzing medical information. However, there are no comprehensive studies aimed at identifying the patient's condition without specifying the type of anamnesis. Specific tasks have been addressed in this area, but studies have only partially addressed the phenomena of Big Data and the Internet of Things, in-depth analysis, and visualization of accumulated data to support personalized treatment decisions.
Big Data has attracted much attention in academia and industry fields in the last
few years. The trend of utilizing medical big data has increased tremendously as well.
The medical big data is generated from medical records, medicine, research, etc. but it
is not well connected due to its long-term development in the last decades in most
hospitals without high-level strategic guide and plan [
In the context of “Big Data +”, people began to study the application of data
visualization to medical data. Data visualization can make full use of the human sensory
vision system to guide users through data analysis and present information hidden
behind the data in an intuitive and easy-to-use manner. There is the problem of
visualizing such a large amount of data in a way that is understandable to humans: doctors,
patients. In addition, proper visualization allows you to extract general knowledge and
track trends and dependencies. An example of a platform for the visualization of large
medical data sets is presented in [
Healthcare (Cardiovascular diseases) has now become a major concern in the
economic as well social regiment of almost everyone. The medical system should be able
to handle long-term and continuous monitoring, inconspicuous monitoring and should
be highly sensitive to the patient's ECG signal. IoT helps the ECG signals to get
transmitted through the sensor via a gateway using communication protocols like
Bluetooth, Zigbee, 3G/4G, Wi-Fi, LAN, etc. Then this data can be sent to numerous
places like the doctor's end, caregiver or the cloud-storage for analysis or processing,
etc. The doctor at the remote location can view the patient's report on various smart
devices, all thanks to the computing technology. This helps in dealing with the
emergencies. Big Data plays a pivotal role in this system as it provides with data analysis,
decision-making, extracting useful information via algorithms, intelligent storage etc
[14. 15]. The combination of these three technologies will make the world a more
secure place to live. Also medical imaging is often used and gives big data medical
records [
So, for the healthcare domain all of Big data representation models can be used.
The first layer consists of all possible sources of information. Such approach allows usage of this platform as part of smart house too. Combination of indoor and outdoor parameters allows finding dependencies between patient’s conditions and the weather and using these dependencies for the prediction of the new state of the patient.
The second layer is used for the communication between the rests of layers. It
is very important to provide the secure transmission and saving of the personal data.
In this paper, we are not considered the security aspects. More details about security
features of medical data is given in [
The next layer is preprocessing layer. The medical data is collected from
various sources given at data layer. That is why the filtering, cleaning, wrapping and
missed data imputation methods are used. The next important thing is that medical
data is collected from different countries. Each country has own medical protocols
description written in different languages. In addition, different medical devices are
used in hospitals from different countries. That is why the main tasks of preprocessing
layers are:
Anonimazation and Data protection – for medical data privacy,
Missing data recovery [
Data processing layer consists of the following modules: Online diagnostic based on fuzzy neural networks [27] – for fast training process, Personalization of the treatment based on the decision tree [28] – for the most appropriative medications selection, The risk prevention model – see below in more details, Statistical model for the risk prediction [29] – for dependencies mining between paramenets of the pation and the rest of smart home parameters. The ontology development [30] – for increasing of the literacy of the young doctors.
The user layer represents the different groups of user interested in the platform usage.
The integration layer represents the list of standards and open platforms related to the proposed platform. 3.2
The usage of a variety of processing technologies of emerging Big Data will allow to study causal mechanisms of modelling and predicting treatment stages, taking into account individual peculiarities of the patient, to analyze medicines’ data and find their key characteristics. This information we use for development of innovative approaches to improving the methodology of risk stratification, to model the therapeutic schemes, to improve medical care quality through personalization of treatment regimens for patients.
The analysis of patients’ information will help to identify common social characteristics and thus contribute to the development of recommendations for disease prevention
This also will stimulate the healthy people to monitor their health parameters.
Analyzing large amounts of data requires identifying attribute groups that form functional dependencies. However, in the real world, data sets are much more common, with important dependencies defined only on a subset of key attribute group values; we will call such dependencies partial functional dependencies. The partial functional dependencies are modified associative rules but for part of the data. The main idea of this method is to process data given in different structure or semistructured data. The algorithm for the Probabilistic Production Dependencies building is built on lazy calculation approach (modification of FP-tree). It allows reducing the time complexity and using the parallel and distributed mode for calculation.
Probabilistic Production Dependency (PPD) is the kind of functional dependency (F-dependency) arized in relational databases. This is also similar dependency to associative rules built on Aproory algorithm or something similar. The main difference between associative rules and PPD is that PPD will generate from existing functional dependencies in dataset. For this purpose, the Armstrong rules are used.
PPD: = { }, ∈ , = { }, ∈ , : ( ∈ → ∈ ) = , where k and d are the tuples of groups of attributes K and D respectively.
The main indicator of the reliability of such a dependency is the ratio of the number of objects that such PPD has to the number of objects in the selection: (PPD) = | ∈ ∧ ∈ ( )|
| ∈ ( )|
The calculation of the reliability of the implementation of such dependence is based on the possibility of decomposition of such dependence into components of the PPD: (1) (2) (3) ( ∈ → ∈ ) = ∑∈ ( ∈ → = ) = ∑ ∈
∑| = ∧ = |
∑| = |
As in the case of F-dependencies (functional dependencies), the set of classification rules that take place in a given relation can be represented by some subset of them, from which by means of output rules all classification rules of a given relation can be obtained. Since classification rules are an extension of F-dependencies, it is worth considering axiom transformations for functional dependencies for classification rules.
Therefore, the algorithm for PPD mining consists of two steps: 1. Frequent pattern mining – based on Apriory algorithm. 2. The existing pattern splitting and decomposition – based on reliability ration. In the patient data analysis, the sequence of events is often of interest. When detecting regularities in such sequences, it is possible to predict with some degree the occurrence of events in the future, which allows us to make more correct decisions. A sequence is called an ordered set of objects. Using the hierarchy allows you to determine the connection that goes into higher levels of the hierarchy, since the support for the set can increase if the entry of the group, and not its object, is counted. In the hierarchical structure of objects, you can change the nature of the search by changing the analyzed level. Moving up the hierarchy, we summarize the data and reduce their number, and vice versa. So, Probabilistic Production Dependencies will be a combination of associative rules and sequential rules.
The proposed algorithm makes it possible to assert that the task of detecting Probabilistic Production Dependencies in distributed databases belongs to the class of Ptasks. Low asymptotic complexity of the established association rules mining algorithm and a wide set of data types supported for analysis allow to apply the established algorithm in practically all subject areas working with association dependencies in data. Algorithm for finding association dependencies is well-solved with MapReduce. 4
The proposed algorithm is tested on open data set for associative riles mining [31]. The data set used for this application is the Adult data set in the Machine Learning Repository UCI.
The AOG model is an oriented acyclic graph, where each vertex of a graph corresponds to a variable with assigned parameters. In Bayesian networks, parameters are given as local conditional distribution of probabilities of the values of the variables P (Xi | F (Xi)).
And in Gaussian networks - as the coefficients of linear equations (for edges) and dispersion of deviations (for vertices).
The construction of the AOG-models meets the problem of reproducing the model from the statistical data. These include methods for restoring the "Collifinder" and "Proliferator-C" AOG model, generalizing the Chow & Liu method. The use of Collifinder and Proliferator-C allows recognition of transitive, synergistic, and combined associations, and thus provides a reliable and effective method for reproducing structures of single-threaded dependency models without first-level tests.
The problem of the above-described methods is the need to specify elementary conditional relationships for constructing a graph of the AOG-model.
Finding such dependencies goes beyond these algorithms.
Own algorithm. The paper proposes an algorithm for extracting PPD. For the relationship with the scheme = { , ( )}, = ̅1̅̅,̅̅̅, it allows you to find statistiA cally significant rules that reflect the dependence of the attribute m on the utes 1, 2, . . . , −1 , that is, the dependence of the species 1, 2, . . . , −1 → .
As a measure of statistical significance, the Kullbach-Leibler information measure is used.
The algorithm allows you to look for only dependencies defined on the whole set of input data; in addition, it has a high computational complexity if there are many classification rules.
The result of own algorithm is given on fig. 2. If we need to search for associative rules among individual subjects, you can find that in many cases, associations with high support for individual events are practically absent. Therefore, in spite of the fact that, in general, event association support1 -> Event2 can be very high, association support between individual types of events is likely to be low. Thus, such associations, although they may be of interest, will be excluded from consideration as they will not meet a certain minimum support threshold Smin . longs:
To solve this problem when seeking associative rules, not individual subjects, but their hierarchy is considered. If there are no such interesting associations on the lower hierarchical levels, then they may occur at higher levels. In other words, support for an individual object will always be less than the support of the group to which it be ( ) > ( ), ( ) = ∑ =1 ,
Where I the group is in the hierarchy; j - this item is included in the given group. The reasons for this are obvious: the total support for the group is equal to the amount of support for the items included in it: where n - the number of items in the group.
Associative rules found for objects or events located at different hierarchical levels are called multilevel rules.
Going down to the lower levels of abstraction, the descendants of only those categories and subcategories that are frequent sets are analyzed, that is, there are at least a predetermined number of times, where k - the number of the level.
Searching for frequent sequences runs from level 1 to the maximum possible. The results of successive passes will be presented in the Table 2.
F1
Thus, the maximum sequences are <1; 2; 3; 4>, <1; 3; 5> and <4; 5> because they are not contained in sequences of greater length. They will be sought after by successive templates. 5
The study presents the architecture of the platform for Big data preprocessing and processing. The state of art in this domain is presented. The method for prevention of the risk disease based on Probabilistic Production Dependencies is developed. The accuracy of the proposed method is estimated. 26. Bodyanskiy, Y., Tyshchenko, O. (2019). A Hybrid Cascade Neuro–Fuzzy Network with Pools of Extended Neo–Fuzzy Neurons and its Deep Learning. International Journal of Applied Mathematics and Computer Science, 29(3), 477-488. 27. Shakhovska, N., Fedushko, S., Shvorob, I., & Syerov, Y. (2019). Development of mobile system for medical recommendations. Procedia Computer Science, 155, 43-50. https://doi.org/10.1016/j.procs.2019.08.010 28. Melnykova, N. (2017). Semantic search personalized data as special method of processing medical information. In Advances in Intelligent Systems and Computing (pp. 315-325).
Springer, Cham. 29. Fedushko, S., Ustyianovych, T., & Gregus, M. (2020). Real-time high-load infrastructure transaction status output prediction using operational intelligence and big data technologies. Electronics, 9(4), 668. https://doi.org/10.3390/electronics9040668 30. Lytvyn, V., Vysotska, V., Dosyn, D., Lozynska, O., Oborska, O. (2018). Methods of building intelligent decision support systems based on adaptive ontology. In 2018 IEEE Second International Conference on Data Stream Mining & Processing (DSMP), pp. 145-150.