Background: Compared to unstructured representation of clinical evidence in bibliographic databases such as PubMed, structured results data available in clinical trial registries may be more timely, complete, and accessible. But these data remain underutilized. Objective: The clinical trial information is extracted from the semi-structured records on ClinicalTrials.gov to construct a PICObased knowledge graph for representing clinical evidence. The knowledge graph is expected to give a whole picture on the research protocol and reported results of clinical trials. It can be quickly searched, visualized, and exported in batches and on-demand. Methods: We collected 6279 registered clinical trials on COVID19 in ClinicalTrials.gov, among which 71 trials had reported results. Information extraction and term standardization were carried out in a semi-automated manner. The knowledge graph was constructed using neo4j. Results: Two knowledge graphs are constructed. The first COVID19 Trial Knowledge Graph (CTKG) contains 66856 nodes with 10 types and 1217673 relations with 9 types in total. The second COVID-19 Trial Results Knowledge Graph (CTRKG) contains 1067 nodes with 12 types and 1405 relations with 13 types in total. The graphs allow for queries, batch exports, and provide data for comprehensive clinical evidence based on PICO. • Knowledge Graph • Information Retrieval
Conclusions: Our work validated the idea of “computable evidence synthesis” via presenting PICO data elements results data in trial registries in a standardized, structured format with consistent ontologies, such as UMLS. Queries and batch export of information can be acquired in Graph Database built by neo4j through Cypher. It can help researchers obtain the latest data in batches and form a basis for the synthesis of real-world research evidence. Our methodology is also generalizable to other conditions and can incorporate registered clinical trial data from more platforms to achieve field unification of multi-source heterogeneous data.
1
Ravaud, P et al. proposed “the ‘one-off’ approach of
systematic reviews is no longer sustainable; we need to move
toward producing ‘living’ evidence syntheses (i.e., comprehensive,
based on rigorous methods, and up-to-date) [
The representative developments include building semantic
representation for clinical trials and medical guidelines [
Efforts aimed at increasing the pace of evidence synthesis
have been primarily focused on the use of published articles, but
these are a relatively delayed, incomplete, and at times biased
source of study results data. Existing studies results have shown
Copyright 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
that the same trial results are quite different from those reported in
ClinicalTrials.gov and journal articles. Negative results in
ClinicalTrials.gov tend to be frequently reported than in journal
articles. This may be related to the fact that journal articles always
report positive results. Further efforts should focus on decreasing
existing published discrepancies to enhance the transparency and
accuracy of data reporting in clinical trials [
To solve this problem, we believe that the information in registered clinical trials can be made into a structured knowledge graph. Given the increasing importance of text analysis in biology and medicine, we believe a local graph database of clinical trials will provide helpful computing infrastructure for researchers. To overcome this, automation technology is being explored by many researchers. Our research implements this process in a semiautomated progressing. We have carried out knowledge extraction, structured data visualization and query work. The specific contents include: 1) Automatically obtaining clinical trial data from registration platform. 2) Visualizing the structured data of the clinical trial registration platform through a knowledge graph. 3) Standardize indicators to realize the searchability of medical knowledge. 2 2.1
Our research flowchart is shown in Figure 1. ClinicalTrials.gov
is a database storing privately and publicly funded clinical studies
conducted around the world. ClinicalTrials.gov provides storage
and download of structured data, such as csv format, but the key
information is not It is not completely accurate, such as not
distinguishing intervention from comparison, and the combination
of drugs involved in Intervention cannot be accurately displayed;
in addition, the outcome indicator lacks a detailed description. The
U.S. National Library of Medicine (NLM) [
We extracted 14 fields from 6279 clinical trial XML files as following: Nctid, Title, Location, Study design, URL, other IDs, Conditions, Intervention/ Comparison, Outcomes, Funding, Enrollment, Gender, Age and Study type. Seventy one studies with results were added with two fields on this basis,i.e., Start Date and Completion Date.
According to the clinical trial registration information, valid fields were extracted and divided into 5 categories: 1) The metadata of Clinical trials: including 10 fields study information. 2) Conditions: since this study uses COVID-19 as an example, all clinical studies included the same condition, which is COVID-19. And we also extracted all other diseases and/or conditions in these fields. The complications of the disease are not distinguished. 3) Population Feature: three demographic characteristics of the population, i.e., age, gender, and enrollment were extracted. 4) Intervention/comparison: According to the registration information of the existing clinical trial registrants, the intervention/comparison measure types are divided into 7 categories. This classification has the subjectivity of the research designer. Since the CT.gov web information only displays the intervention field, most current studies do not distinguish between intervention and comparison. We have obtained the measure type grouping situation from the XML documents, experimental and comparator group. As long as there exists evident cue word “experimental” in the type, the measure type is classified as intervention, and the null value or other value is classified as comparison. However, the type of many observational clinical trials is empty, and its grouping cannot be represented correctly. 5) Outcomes: We extracted all the outcomes descriptions from the XML data. As many clinical trials have not yet produced results or are not completed, many outcomes only stay at the trial design stage without corresponding reported result details. Fields contained in all categories, information in nodes are shown as Figure 2. make better use of the visualized results and search for the research integrated into the knowledge graph, we give a standardized terms on fields such as conditions, intervention/comparison, and outcome.
The Medical Text Indexer (MTI) produced by the NLM was
adopted to extract standardized medical subject heading [
MetaMap is a highly configurable program developed by NLM to map biomedical text to the Unified Medical Language System (UMLS) Metathesaurus or, equivalently, to discover referred Metathesaurus concepts in the text. We standardized the intervention/comparison of 6279 clinical trials by using MetaMap tool. Among them, 5706 clinical trials have intervention/arm/group. No specific intervention was written in the 573 clinical trials. It may be because some clinical trials have not yet started, or because some clinical trials are observational studies without arm/intervention information. Finally, we analyzed a total of 5311 clinical trials. Since the clinical trials of has results announced specific plans for 2.3
“The lack of a standardized outcome classification system
results in inconsistencies due to ambiguity and variation in how
outcomes are described across different studies.” [
Since clinical trials that have not yet found results are only at the design stage, we only mapped the outcomes of 71 studies through the MetaMap tool and standardized them with manual work. In order to improve the visualization, all the unified standardized names first to be select to show is the abbreviation, and the detailed description in the field is contained in the nodes. 2.4
Graph database query language is indispensable for medical
information queries. The Clinical Quality Language (CQL) [
We built the graph database in Neo4j 4.25. The Neo4j graph
database and its query language, Cypher, provide efficient access
to the complex Reactome data model, facilitating easy traversal and
knowledge discovery. Cypher is a graph database query language
that is easy to understand and does not require any deep
programming knowledge [
At present, many researchers make a great effort in the
establishment of graph database in Neo4j, such as the developed
applications to link Cytoscape and Neo4j by Summer et al. and
providing a high-performance pathway data resource to the
community through adopting graph database technology by
Fabregat et al. [
We collected clinical trial studies related to COVID-19 at https://clinicaltrials.gov/ on August 3, 2021. A total of 6279 studies were collected, of which 71 studies have results. As we all know, the publication of bibliographic databases is delayed compared with clinical trials. In order to find the time difference between COVID-19-related clinical trials and publications, we searched 71 studies of registered clinical trial numbers on the Dimensions platform. 59 publications (containing 4 not open access studies, 8 not report the results) related to 35 studies in 71 registered clinical trial were retrieved. The completion time of clinical trials is on 2 https://www.nlm.nih.gov/research/umls/new_users/online_learning/Meta_005.html average（79.81 ± 125.73） days earlier than the bibliographic publications. 3.2
Totally, there are 6279 registered clinical trials related to COVID-19 included in our research. Among them, 6208 (98.87%) studies have not yet registered results. So we generated two knowledge graphs. One contains all 6279 registration studies related to COVID-19, which is used to summarize the latest clinical trial studies to help researchers understand the trial progress and design plan. The other one includes 71 clinical trials that have registered results. Based on the framework of the first graph, we added fields for each intervention/comparison’s group name, baseline data, and post-test data to the outcome node in the second graph. The grouping of each intervention is clearer than the enrollment node. This data helps researchers to generate evidence synthesis based on clinical trial data from studies screened by the same conditions.
The first COVID-19 Trial Knowledge Graph, which contains 66856 nodes with 10 types and 1217673 relations with 9 types in total. The second is COVID-19 Registered Results Knowledge Graph contains 1067 nodes with 12 types and 1405 relations with 13 types in total. The statistics of nodes and relations can be seen in Tables 1 and 2, respectively. The graph data files and codes are public available at: https://github.com/baiym13/COVID-19-TrialKnowledge-Graph/find/main. outcome, filtered based on the P-value extracted from the XML file. The established relations are “has significant” or “no difference”. This is used to help researchers understand the effects of current measures on specific outcomes more intuitively.
The knowledge graph below shows the basic information (age groups, genders, study types, start dates, completion dates, conditions, interventions, outcomes) of the 71 clinical trials with registered results. For visualization purposes, this graph only shows the top 300 nodes. The nodes at the center of the graph have more relations with other nodes. Several obvious clusters appear on the graph, which provides an overview of the 71 clinical trials results. The main age group of enrollments is 18 and older, with 11 clinical trials including children in the study population. Most of the studied populations are on both male and female, while 4 studies focus on the population of male and one study only focus on the population of female. The number of participants lies between 1 and 20460. The majority (63 out of 71) of studies are interventional, and the rest are observational. More than half of the clinical trials started in the period from March to May 2020 and were finished in the period from May 2020 to May 2021. Among all the studies with results, National Institutes of Health (NIH) has funded 5 studies, related industries have funded 23 studies, and the rest of the studies are funded by others, including individuals, universities and organizations). Most clinical trials focus solely on COVID-19, while 5 studies respectively investigated the relationship between COVID-19 and androgenetic alopecia, pregnancy, neoplasm, influenza and adolescent depression. As for interventions, among the 300 nodes, two drug interventions in three different trials were found to have significant difference on the experimental groups, which are Avigan and HCQ (Figure 3).
Conditions: There are 1395 nodes of conditions in 6279 studies, of which only 1278 have DUIs. We use the original description to supplement the conditions that are not mapped by the MTI tool. We used 0 to supplement the missing values of standard interventions and DUI in our knowledge graph. Some clinical trials regard the occurrence of COVID-19 as the clinical outcome of certain highrisk groups. When the high-risk situation is refers to a disease, COVID-19 cannot be used as one of the complications. Therefore, in the list of conditions, the complications of the disease are not distinguished. There are 127 descriptions of conditions in 71 studies (the same concept is not de-duplicated), all of which have DUIs. After deduplicating 23 identical concepts in the same clinical trial, a total of 104 nodes are left about conditions in the knowledge graph. The standardization method is the same as that of 6279 clinical trials. A total of 26 standard condition-concepts were obtained after deduplication.
Intervention/Comparison: We use MetaMap to standardize intervention/comparison terms. The standardized results were mapped to CUI. The results that cannot be mapped to the CUI used the original description. The problem of combining measures has been solved by manual verification, such the same drugs Favipiravir, Avigan and Aavilavir were merged to one. The standardized results preferentially use the abbreviation from MetaMap mapping through manual screening.
Outcomes: The text of the original outcome description cannot be mapped to a single field in MetaMap, and there may be multiple entity types in the same field. Therefore, in this study, we only extracted the standardized results of the outcome of 71 clinical trials with registered results, taking into account the complexity of the mapping results. Field in outcome: the empty value in the field, Time frame, is filled by 0. Other fields involve numerical values, so NA is used to fill in missing values.
Other nodes: funding, enrollment, gender, age, study type, start date, completion date are extracted directly from the download XML files.
In addition to the relationship directly displayed in the structured data of clinical trials, we have established a new relationship between the node intervention/comparison and the
Case Query 3.4.1 Nodes information of Clinical trial knowledge graph An example of a single clinical trial query is shown in Figure4. By using Cypher, the study record of 'NCT04452435' is extracted. The general information of the study and the population features are connected directly to the clinical trial node. In this clinical trial, the intervention is C21 (an experimental drug) and the comparison is placebo, therefore, the relationship between the clinical trial and intervention is captured by a directed connection from the trial node to the intervention node, named as has_intervention. Similarly, the comparison node is connected to the clinical trial node with a directed relationship, named as has_comparison. In this clinical trial, the intervention measure was found to take effect on the experimental groups, resulted in statistical significance on several outcomes. Therefore, on the graph, the three red relations, named as has_significant, indicate that the intervention, C21, led to a significant difference on the three outcomes compared to the comparator group. Similarly, by examining the p-value, the intervention was also found to have no significant difference on the other 8 outcomes. In these cases, the green lines, named as no_difference, are used to formulate this relationship. The last outcome, oxygen (supply), has no connections with the intervention, which means no statistical information can be extracted to deduct their relations and no connection is drawn (Figure 4). 3.4.2 Case query for outcome data Taking “c-reactive protein” as an example, we use the Cypher language to query all the outcome nodes with the standardized name "CRP". The result shows a total of 8 CRP nodes. Through the expansion of the node-related relationship, it can be seen that there are to 6 clinical trials, 3 interventions and 1 comparison. And it contains 4 relations “has significant” (Figure 5).
Cypher query command:
MATCH (n{standard_outcome:'CRP'}) RETURN n;
All the node information obtained by the query can be exported as a csv file or a Json file in neo4j. This study takes all pieces of information in Json format in the CRP query result as an example (Figure 6). The intervention/comparison measures are numbered “O+number” in the XML files. The number of repeated occurrences of the fields “group info”, “participate group num” and “outcome group num” corresponds to the “time frame” in order. We parse the Json format to export operable data through selfcompiled code.
3.4.3 Relationship Query One feature of our knowledge graph that is different from the past is the establishment of a relationship between intervention/comparison and outcome. The purpose of this is to help researchers find all the measures that can or cannot improve the outcome of the disease in a particular trial. According to the automatically obtained P-value, the relationship “has_significant” or “no_difference” are defined according to the 95% test level. Taking the "has_significant" query as an example, the result is shown in Figure.7. Through standardizing intervention/comparison, the number of various nodes can be counted and the measures that affect COVID-19 most can be found. The research purpose is well practiced (Figure 7).
Cypher query command:
MATCH p=()-[r:has_significant]->() RETURN p LIMIT 300
Meta-analyses are increasingly used to address the evidence
synthesis problem. But they filter out a lot of information during
the data processing. [
Instead of using unstructured claims in scientific publication, our work validated the idea of “computable evidence synthesis” via presenting prespecified PICO data elements results data in trial registries in standardized, structured formats with controlled vocabularies. We used COVID-19 as a case in our research. By parsing the XML file, more detailed information can be obtained than the csv or txt format downloaded from Clinicaltrials.gov. In addition to the necessary elements contained in PICO, we also extracted data from the clinical trials results. To help form the basis of computable medical evidence. Query and batch export information in Graph Database built by neo4j through Cypher language. It can help researchers obtain the latest data in batches and form a basis for the synthesis of real-world research evidence. Compared with publications in bibliographic database, these data include negative and positive outcomes. More comprehensive and objective. Our methodology is also generalizable to other conditions, such as cancer clinical trials. We will incorporate registered clinical trial data from more platforms to achieve field unification of multi-source heterogeneous data in future research.
This work was funded by Chinese Scientific and Technical Innovation Project 2030 (2018AAA0102100), the National Natural Science Foundation of China (71603280, 72074006), Peking University Health Science Center and the Young Elite Scientists Sponsorship Program by China Association for Science and Technology (2017QNRC001).