Graph approaches are becoming increasingly popular in various areas of life and provide a powerful tool for displaying complex relationships between different modalities of data. This study considers a new graph document-oriented structure for processing multimodal medical data in order to improve disease prediction and analysis of patient treatment outcomes. The study emphasizes the importance of using graph models in health care and demonstrates their advantages for processing large and heterogeneous data sets, which allows obtaining more accurate and prompt medical conclusions. In particular, medical images, electronic health records (EHR), records from wearable devices, environmental records, psycho-emotional state and others are taken into account. The proposed data processing pipeline includes several stages: preprocessing, creation of a base graph, minimization of the number of edges, recalculation of edge weights. In the future, it is advisable to apply graph neural network (GNN) approaches, in particular Graph Convolutional Networks (GCN) and Graph Attention Networks (GAT), for better integration of internal data, which will allow for a more comprehensive view of patient data.
Graphs are a powerful tool for representing complex relationships in data and are increasingly being
used in a variety of industries, including healthcare. In particular, their use to represent multimodal
information that includes a variety of data sources, such as text documents, images, videos, and
signals, opens up new opportunities for improved data management and decision-making.
Implementation of graph-centric approaches at the enterprise level requires [
Despite the complexity of such integration, investments in digital technologies in the healthcare
sector pay off due to the ability to digitize medical solutions, thereby improving the quality of
services provided and increasing treatment efficiency [
Graphs allow to represent the relationships between complex and large data sets, and graph-based
approaches are particularly popular in the processing of multimodal medical information and have
shown significant success in various biomedical applications. Applied to disease prediction tasks,
researchers build a graph manually based on a specific modality (e.g., demographic information) and
then integrate other modalities to obtain a representation of the data through Graph Representation
Learning (GRL) [
The main limitation of the Multi-modal Graph Learning model [
In the healthcare industry, there are many types of data that are captured during patient examinations. These include: electronic health records (EHRs) medical images data from wearable devices genomic data sensor data, ecological and environmental data psychological and behavioral portraits.
A more detailed list of data types can be found in Figure 1. These data of different modalities contain unstructured data in their specific formats.
During processing, this data is transformed, synchronized and combined to become knowledge.
Data transformation and synchronization is realized by means of methods for structuring EHRs,
extracting features from medical images, and analyzing data from wearable devices, and knowledge
fusion and generalization will be performed using the Graph Representation Learning method based
on the Multi-modal Graph Learning model [
In this study, we will consider a methodology and architectural solution for a medical data
processing system that includes the selection of key features, data transformation, synchronization,
and fusion in order to efficiently process the available multimodal data. In addition, the research faces
a number of challenges such as data quality, privacy, security, processing, analysis, clinical
integration, ethics, and interpretation of results. Since multimodal data fusion has the potential to
transform healthcare, this study serves as a starting point for future research and progress in the field
of smart medicine. It lays the groundwork for improving patient outcomes and facilitates the
implementation of personalized solutions in medical innovation [
The main contributions of this work include: 1. The use of Graph Representation Learning based on the Multi-modal Graph Learning model in the context of multimodal data fusion for smart healthcare. 2. Creating a pipeline for processing multimodal data with the subsequent application of adaptive learning to reveal the hidden structure of the graph and can be optimized with a patient disease prediction model. 3. Evaluation of the results. description of challenges and growth points. Description of future ideas for the implementation and application of the developed model, for the purpose of future research.
This paper is organized as follows: Section 2 provides an overview of existing techniques for applying the graph approach to multimodal information processing. Section 3 proposes methodologies for data processing using Graph Representation Learning based on the Multi-modal Graph Learning model. Section 4 presents a comparison of the time and computational complexity of the proposed model for medical data processing. Sections 5 and 6 discuss and compare the results of the study.
Graph-based approaches are becoming increasingly popular in modern multimodal data processing research. These approaches are often used in various scientific fields where it is important to take into account the modalities and relationships between different features of data. In this section, we will review the methods that use graph structures to analyze multimodal data, as well as their effectiveness in different research contexts [9].
In the study “A multimodal graph neural network framework for cancer molecular subtype classification” [10] presents the following key contributions: A novel generalized integrative model based on graph neural networks (GNNs) for classifying molecular subtypes of cancer. The use of a supra-graph approach to incorporate both intra-omics and inter-omics biological knowledge in the form of graphs. Representation of multimodal data as a heterogeneous multilayer graph. Comparative analysis of models based on GCN and GAT with different combinations of omics data and graph structures. The study analyzes the effectiveness of GCN and GAT for different data structures and modules, including neural networks for increasing dimensionality, a decoder, and a surface parallel network.
The proposed research graph is built on the basis of three different types of omics data on the left and two graphs with previous biological knowledge on the right [11]. The visualization can be seen in Figure 2, where the mRNA data (orange table) and CNV data (yellow table) are gene-based and have the same size, while the miRNA data (green table) have the same number of rows but a different number of features for each sample.
In the study “Multimodal Representation Learning using Adaptive Graph Construction” [12], two approaches are considered for building a multimodal data representation: a fully connected graph (FCG) and a minimal spanning tree (MST): Fully connected graph (FCG): Includes all possible connections between modalities, where each modality is represented as a node, and there is an edge between each pair of nodes. This approach allows you to fully capture complex correlations between modalities, which is especially useful in contrastive learning. Minimum spanning tree (MST): Based on the Kraskal algorithm, this tree retains only the most significant connections between modalities, eliminating redundant and noise connections. This provides a more interpretable structure focused on the key correlations between modalities.
Thus, FCG provides a complete overview of the relationships between modalities, while MST allows you to simplify the graph by focusing on the most important correlations, check Figure 3 for details.
Based on the researches reviewed above, we can conclude that graph-based systems are particularly relevant for processing multimodal data due to their ability to integrate and model complex relationships between different types of information [13]. Research shows that graph methods have significant potential for implementation in the medical field, in particular for the analysis of various types of data. This will be especially relevant for the health care sector and allow you to effectively combine data from different sources, providing more accurate and comprehensive results. Such approaches open up new opportunities for application in personalized medicine and improving the quality of medical services [14].
In working with multimodal data, storing as much data as possible in the fastest form for use is an important task.
This paper proposes an object-oriented graph database [15] that provides the ability to store data parts (multimodal data) as graph nodes and establishes dependencies between them in the form of edges.
A database based on a multimodal object-oriented graph incorporates the complexity of a data graph node, that is when a node is an element with many different characteristics. Such an implementation uses an object to provide flexibility in querying graph databases, and key/value storage provides fast data retrieval [16].
Graph vertices are objects. Each object has a unique identifier. Objects can be simple (atomic) or complex. Simple objects do not have outgoing edges but can take the value of one of the unimodal attributes.
The object G of such a database is represented as follows:
G=( N , E ) , (1) where N is the set of graph nodes, N = {n1, … nm}, E – is the set of graph edges, E = {e1, … en}.
Given that a directed graph is used, a graph node can be represented as an object that contains a set of key/value parameters 〈k,v〉, as well as the value of the NodeType, e.g.,
N ={⟨ k , v ⟩ , NodeType } , vi∈ {SE }
A graph edge is represented as an object that contains pointers to the ParentNode and the ChildNode, the value of the EdgeType, EdgeType ∈ Pr and the Weight:
ei={ ParentNodei , ChildNodei , EdgeTypei , Weighti } ,
Thus, a database based on an object-oriented multimodal graph in this case is a very convenient means of storing multimodal data. It should be noted that for the task of processing multimodal data, it is necessary to consider all the extracted features from the data of different modalities. Further research will be aimed at expanding and optimizing the database to take these factors into account.
Further generalization of the display of relationships between multimodal data objects using graph databases consists of assigning to edges and arcs some quantitative values, qualitative features, or characteristic properties, which are called weights.
Definition: An object-oriented graph, each edge e of which is assigned a real number w(e) is called weighted. This number is called the weight of the edge e [17].
The weight of an edge can be the ordinal numbering of edges and arcs, which indicates the order in which they are considered (priority or hierarchy); path length, bandwidth; the number of points scored; the nature of relations between objects, etc.
There are many ways to represent a weighted graph. Some of them are discussed below [18]. Let the graph be given where |V|=n, |E|=m
Method 1. Setting the weight matrix W, which is analogous to the adjacency matrix. For such a matrix, the element wij, if the edge (vi,vj) ⊂ E, will be denoted as
If the edge (vi,vj) ⊄ E, then, depending on the problem to be solved, the element of the matrix will be denoted as
G=( V , E ) , wij=w ( vi , v j ) , wij=0∨wij=∞ ,
Method 2. Sometimes a graph is defined as a list of edges. For a weighted graph, three cells can be allocated under each element of the list E - two for an edge and one for its weight, that is, 3m cells are needed in total.
Method 3. The graph can be presented as an adjacency list. For a weighted graph, each list Adj[u] contains, in addition to pointers to all nodes v of the set G(u) numbers w(u,v).
Consider the graph object given before: where N – graph node, E – the set of edges.
The object of such graph, considering the weights, will have the following form:
G={⟨ N , E ⟩ } ,
G={⟨ N , E , W ⟩ } , where W – set of edge weights.
The set of edge weights is represented in the following form:
W ={⟨ w1 , … , wn ⟩ }, So, an object-oriented weighted graph will be represented as
G={⟨ {⟨ k , v ⟩ , NodeType } , {⟨ e1 , … , en ⟩ } , {⟨ w1 , … , wn ⟩ }⟩ },
The example of saving and processing data about medicines by building an object-oriented graph is represented below.
The graph nodes are of two types: NodeType = {Drug, Disease}. Edges will also be of two types: EdgeType = {Indication, Contraindication}.
Figure 5 demonstrates the generated object-oriented graph for multimodal medical data with two types of nodes. (4) (5) (6) (7) (8) (9) (10)
A further generalization of the relationship mappings between multimodal data objects using object-oriented graph databases is in defined edges and arcs of some number of values, qualitative features, or characteristic properties that have meaning.
The same operations are performed on an object-oriented multimodal graph as on ordinary ones path search, construction of a frame tree, traversal of the graph, etc. [19]. However, some operations must be redefined due to the existence of different types of edges and complex vertices.
The predefined operations on graphs are presented as follows.
Union of graphs G1(N1,E1) and G2(N2,E2) with a set of intersecting nodes: adding non-intersecting nodes:
N 1k= N 1 {N ¿2 , N 2k= N 2 {N ¿1 , N k= N 1k∪ N 2k , Ek= E1k∪ E2k
ek ( ChildNode j )={ e2( ChildNode j ) , e1( EdgeTypei )=e2( EdgeType j ) e1( ParentNode j ) , e1( EdgeTypei )≠ e2( EdgeType j ) , N k= N 1 ∩ N 2 , Ek= Ek∪ {e } .
1. Search for a terminal pair - drain S and drain T: (11) (12) (13) (14)
S : G1 ( e ( ParentNodei ))=G2 ( e ( ParentNode j )) , G1 ( e ( EdgeTypei ))=G2 ( e ( EdgeTy(p1e5)j )) , T : G1 ( e ( ChildNodei ))=G2 ( e (ChildNode j )) , G1 ( e ( EdgeTypei ))=G2 ( e ( EdgeType j )) (16)
Drain union G1 and G2;
Union G1 and G2.
1. Search for the terminal pair - drain S and drain T, as for the parallel connection; 2. Union of drain from G1 with drain of G2; 3. Union G1 and G2.
The results of our research are a justification of the expediency of choosing a graph database for processing information in the medical field. In particular, it is worth comparing with other known types of databases, in particular NoSQL [20] and relational databases, which largely depends on the specific requirements of the project, the qualifications of the developers, hardware resources, as well as the requirements for the speed of the designed system of the system. However, graph databases have shown particular advantages for processing large volumes of multimodal data, which are often found in the medical field. A brief summary of the results obtained during research could be organize these points: A graph structure is appropriate for use in tasks where complex relationships between different types of data, such as genomics, medical images, and electronic health records (EHRs), need to be efficiently discovered and connected. It allows you to store and analyze data in the form of vertices and edges, and even optimize and minimize edges during processing to improve the structure of the data model. This is especially important in the context of real-time medical data processing, for example, for making rapid clinical decisions. That one advantage of graph databases is manifested when data processing requires high speed of requests and flexibility in finding connections between large volumes of disparate information. Compared to relational databases, graph databases can significantly speed up analysis when working with multimodal medical data, which allows to increase the effectiveness of diagnostics and personalized treatment. However, it is worth noting that graph databases can take up much more space and require more time to optimize. In addition, query complexity can affect system performance, especially if the system processes a significant amount of redundant information. In such cases, it is possible to combine document-oriented and graph databases to ensure a balance between the efficiency of data storage and the speed of access to them.
Given these factors, the use of graph structures in the medical field has great potential, especially for projects where the speed of searching and analyzing large volumes of data is critical.
As a result of the conducted research, a new approach to the processing of multimodal medical data based on graphs was proposed, by using adaptive reweighting of edges in document-oriented graphs. The main achievement was the application of algorithms to optimize the graph structure and minimize the number of edges, through the selection of the most important connections between different types of patient data. This made it possible to significantly increase the accuracy and efficiency of the analysis, as well as reduce the complexity of the graph structure, which is critical in the context of working with large volumes of medical data.
Popular methods were also successfully adapted in the study, the selection of the most significant signs in medical indicators, which were based on considering the quality of data selection, which ensured the flexibility and adaptability of the approach.
The proposed approach has great potential for implementation in intelligent health care systems, in the tasks of personalized treatment and diagnostics. Given the possibility of using graphs to process multimodal data, this method can be integrated into more complex machine learning models, which will contribute to the further development of innovative solutions in the medical field.
Further research may focus on extending this approach, including the application of other graph learning techniques, such as Graph Convolutional Networks (GCN) or Graph Attention Networks (GAT), to improve the results and increase the scalability of the solution.
Summarizing the research, the work proposed a new method of calculating the weights of ribs in document-oriented graphs, taking into account multimodal medical data and assessments of the importance of relationships between data. The proposed method consists of several stages that improve the process of selecting and analyzing data in a graph structure.
1. At the first stage, the initial multimodal data is imported and a request is created to build the corresponding graph. Using this approach provides adaptive graph construction based on input data, which increases the efficiency of its processing. 2. At the second stage, we apply a method to optimize the structure of the graph by removing irrelevant vertices, cutting off irrelevant connections between different types of data. This process significantly reduces the number of graph elements, leaving only the most important nodes and complementary nodes for further analysis. The peculiarity of this method is the introduction of quality assessments for the selection of data, which play a key role in calculating the weights of the edges between the vertices of the graph. After that, the weight of the edges is recalculated based on the entered quality assessments, which allows dynamically adjusting the connections between the vertices, which increases the informativeness and relevance of the analyzed data. 3. At the third stage, the weights of the edges are recalculated between the input data and the selected results. Taking into account the quality scores ensures that the weights are adapted according to the quality scores, making the graph more sensitive to the importance of individual data. This approach allows not only to reduce the size of the graph, but also to increase its efficiency in the context of medical data, where accuracy and relevance are critical.
Thus, the proposed method of calculating the weights of edges in document-oriented graphs is an effective tool for processing multimodal medical data sets. It allows you to optimize the process of information selection and processing, integrating expert assessments, which increases the quality of analysis and decision-making, in particular in the medical field.
During the preparation of this work, the authors used ChatGPT, Grammarly in order to: Grammar and spelling check, Paraphrase and reword. After using this tool/service, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content. [9] W. Huang, Multimodal Representation Learning using Adaptive Graph Construction, in: S.
Mullender (Ed.), Distributed Systems, 2nd. ed., ACM Press, New York, NY, 1990, pp. 19–33. doi:10.1145/90417. 90738. [10] B Li, B., Nabavi, S. A multimodal graph neural network framework for cancer molecular subtype classification. BMC Bioinformatics 25, 27 (2024). https://doi.org/10.1186/s12859-023-05622-4 [11] O. Basystiuk, and N. Melnykova. "Multimodal Approaches for Natural Language Processing in
Medical Data." In IDDM, pp. 246-252. 2022. [12] D. E. Knuth, The Art of Computer Programming, Vol. 1: Fundamental Algorithms (3rd. ed.),
Addison Wesley Longman Publishing Co., Inc., 1997. [13] Shvorob, I. (2017). New Approach for Saving Semistructured Medical Data. In: Shakhovska, N. (eds) Advances in Intelligent Systems and Computing. Advances in Intelligent Systems and Computing, vol 512. Springer, Cham. https://doi.org/10.1007/978-3-319-45991-2_3 [14] Vrahatis AG, Lazaros K, Kotsiantis S. Graph Attention Networks: A Comprehensive Review of
Methods and Applications. Future Internet. 2024; 16(9):318. https://doi.org/10.3390/fi16090318 [15] H. Zhou, S. Liao, F. Guo. TriGCN: Graph Convolution Network Based on Tripartite Graph for Personalized Medicine Recommendation System. Systems. 2024; 12(10):398. https://doi.org/10.3390/systems12100398. [16] Basystiuk, O., Shakhovska, N., Bilynska, V., Syvokon, O., Shamuratov, O., & Kuchkovskiy, V. (2021). The Developing of the System for Automatic Audio to Text Conversion. In IT&AS (pp. 18). [17] C-M. Jîtcă, G. Jîtcă, I. Silvia. Off-Label Pediatric Medication Prescribing and Dispensing: Awareness and Attitudes among Community Pharmacists: A Questionnaire-Based Study.
Pharmacy. 2024; 12(5):149. https://doi.org/10.3390/pharmacy12050149. [18] Nikolskyi Yu.V. Dyskretna matematyka / Nikolskyi Yu.V., Pasichnyk V.V., Shcherbyna Yu.M. –
Lviv: «Mahnoliia2006», 2008. – 608 p. [19] Kapitonova Yu.V. Osnovy dyskretnoi matematyky. Kyiv : Naukova dumka, 2002. 580 p. [20] Graph Theory (on Demand Printing Of 02787). CRC Press, 2018. URL: https://doi.org/10.1201/9780429493768. [21] C-M. Jîtcă, G. Jîtcă, I. Silvia. Off-Label Pediatric Medication Prescribing and Dispensing: Awareness and Attitudes among Community Pharmacists: A Questionnaire-Based Study.
Pharmacy. 2024; 12(5):149. https://doi.org/10.3390/pharmacy12050149. [22] Nikolskyi Yu.V. Dyskretna matematyka / Nikolskyi Yu.V., Pasichnyk V.V., Shcherbyna Yu.M. –
Lviv: «Mahnoliia2006», 2008. – 608 p. [23] Kapitonova Yu.V. Osnovy dyskretnoi matematyky. Kyiv : Naukova dumka, 2002. 580 p. [24] Graph Theory (on Demand Printing Of 02787). CRC Press, 2018. URL: https://doi.org/10.1201/9780429493768.