<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Annotation of existing databases using Semantic Web technologies: making data more FAIR</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Johan van Soest</string-name>
          <email>johan.vansoest@maastro.nl</email>
          <xref ref-type="aff" rid="aff1">1</xref>
          <xref ref-type="aff" rid="aff2">2</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Ananya Choudhury</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Nikhil Gaikwad</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Matthijs Sloep</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Michel Dumontier</string-name>
          <xref ref-type="aff" rid="aff2">2</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Andre Dekker</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          ,
          <addr-line>Maastricht</addr-line>
          ,
          <country country="NL">the Netherlands</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Developmental Biology, Maastricht University Medical Centre</institution>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>Institute of Data Science, Maastricht university</institution>
          ,
          <addr-line>Maastricht</addr-line>
          ,
          <country country="NL">The Netherlands</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Making data FAIR is an elaborate task. Hospitals and/or departments have to invest into technologies usually unknown and often do not have the resources to make data FAIR. Our work aims to provide a framework and tooling where users can easily make their data (more) FAIR. This framework uses RDF and OWL-based inferencing to annotate existing databases or comma-separated files. For every database, a custom ontology is build based on the database schema, which can be annotated to describe matching standardized terminologies. In this work, we describe the tooling developed, and the current implementation in an institutional datawarehouse pertaining over 3000 rectal cancer patients. We report on the performance (time) of the extraction and annotation process by the developed tooling. Furthermore, we do show that annotation of existing databases using OWL2-based reasoning is possible. Furthermore, we show that the ontology extracted from existing databases can provide a description framework to describe and annotate existing data sources. This would target mostly the “Interoperable” aspect of FAIR.</p>
      </abstract>
      <kwd-group>
        <kwd>FAIR</kwd>
        <kwd>annotations</kwd>
        <kwd>terminologies</kwd>
        <kwd>linked data</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Introduction</title>
      <p>
        Semantic interoperability has been a topic in medical informatics since the
introduction of the digital patient chart [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. However, in recent years, the interoperability
aspect is only one of the issues, covered by the FAIR (findable, accessible,
interoperable, reusable) principles [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. These principles extend the interoperable aspect towards
methods to find and access information, and hence promoting reuse of clinical data
for primary or secondary purposes.
      </p>
      <p>Although these principles are perceived as the way forward, implementation is an
elaborate task. Hospitals and/or their departments have to invest into concepts usually
unknown to them, with limited direct insights into the benefit of making data FAIR.</p>
      <p>Furthermore, it needs many competences from different specializations ranging from
the IT department (where is what information stored?), and the medical specialty
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
itself (what does a specific value actually mean, and what’s the provenance of this
information?).</p>
      <p>Next to these organizational hurdles, the current tools to make data FAIR are not
guiding novice users. Most of the tools are heavily tied into Semantic Web
technologies and require knowledge about these technologies. This approach works for
institutes and companies investing into these technologies, however, doesn’t scale towards
making data (more) FAIR for hospitals with limited dedicated IT staff. As an effect,
the adoption of these new technologies becomes even more difficult as the return on
investment is never shown due to a lack of investigation.</p>
      <p>Hence, our aim was to provide a framework and tooling where users can easily
make their data (more) FAIR, while reducing the amount of information needed to get
started. Specifically, to annotate existing datasets using RDFS and OWL statements.
2</p>
    </sec>
    <sec id="sec-2">
      <title>Methods</title>
      <p>This section is split into two parts. First, we will introduce the methods used to make
data more FAIR. The second part describes our experimental setup.
2.1</p>
      <sec id="sec-2-1">
        <title>FAIR data descriptions extraction</title>
        <p>Our implementation expects a database, or folder with comma-separated values files
as input of our framework. In this process, the data description language (DDL) is
being read from the given database structure. This DDL contains the table definitions
(table name, column names) as well as key information (primary keys, foreign keys).
This information is used to build an ontology description, where all tables and
columns are unique classes. More specifically, all table class definitions are subclasses of
the overarching class dbo:TableRow, and all columns are subclasses of the
overarching class dbo:ColumnCell. All dbo:ColumnCell classes are annotated to which
dbo:TableRow they are belonging. Primary key columns are specified as a subclass of
dbo:PrimaryKey, where this class itself is a subclass of dbo:ColumnCell. The same
holds for ForeignKey, with additional class annotations regarding the table and
column it refers to. A visual representation of this schema is given in Figure 1.</p>
        <p>After describing the database structure in the given ontology, instance triples can
be materialized for every row and cell in the given database. Hence, every database
row will be an instance of the subclass of dbo:TableRow, and every cell for this row
will be an instance a subclass of dbo:ColumnCell. Instances of dbo:TableRow and
dbo:ColumnCell are associated using the predicate dbo:has_column. The URIs of
dbo:TableRow instances will be based on the table name and primary key column(s).
This URI is also used as the base for the dbo:ColumnCell URIs. The actual cell values
are literals connected to the instance of dbo:ColumnCell. For foreign key
relationships, no instances for dbo:ForeignKey will be created, only a direct relation from
dbo:TableRow to the referred instance of dbo:ColumnCell will be created.</p>
        <p>This process of materialization can be executed regularly (e.g. daily) to extract
updated information from clinical systems, or regularly updated databases.</p>
        <p>Although the ontology specifies the database schema and relationships, it does not
add any binding to standardized terminologies for the values in the database. In this
process, the subclasses of dbo:ColumnCell are annotated with rdfs:equivalentClass
axioms. For example, the classes myOntology:myTable.age and
myOntology:myTable.gender (both a subclass of dbo:ColumnCell), can have equivalent class
definitions ncit:C25150 and ncit:C28421, respectively. Using reasoning, it would
make the instances of myOntology:myTable.age also findable as the more
standardized term ncit:C17357.</p>
        <p>Depending on the type of variable, this would solve the terminology binding issue
for numerical values (depending on the strictness of the equivalent class term
definition). However, for categorical variables, the literal value of the instance does not
correspond to a category defined in a target terminology. We can overcome this issue
by defining restriction classes. For example, when we have gender coded as “m” and
“f”, we could implement the following definitions described in Manchester syntax:</p>
        <sec id="sec-2-1-1">
          <title>Equivalent class for Male (ncit:C20197):</title>
          <p>ncit:C28421 and (dbo:has_value “m”^^xds:string)</p>
        </sec>
        <sec id="sec-2-1-2">
          <title>Equivalent class for Female (ncit:C16576):</title>
          <p>ncit:C28421 and (dbo:has_value “f”^^xds:string)</p>
          <p>By adding these RDF statements to the ontology, the reasoner will assign the NCIT
term for male or female to the instance of myOntology:myTable.gender. This is
possible as it was previously defined that this was an equivalent class with ncit:C28421.</p>
          <p>This database ontology, combined with annotations to bind standardized
terminology terms, can be published as a description of the dataset at hand. Being RDF data,
we can search in this ontology using SPARQL to find which data elements are
available, and its database schema. Furthermore, when materialization of database contents
is performed, we can directly query the instances as well. The workflow of all steps in
this process are described in Figure 2.</p>
          <p>Fig. 2. Workflow of the complete process. Green elements are automated processes, orange
elements are manual tasks, and blue elements are the products of these processes.</p>
        </sec>
      </sec>
      <sec id="sec-2-2">
        <title>Testing environment</title>
        <p>We tested the above explained methods by developing a tool which can extract the
DDL from the given database to build the given ontology. Furthermore, based on this
ontology, the tool can materialize the triples for the database rows and cells.</p>
        <p>
          This tool was executed on a data-mart of our institutional datawarehouse
containing diagnostic, treatment and questionnaire information of over 4000 rectal cancer
patients treated with radiation therapy [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ]. This data mart is nightly refreshed, and
hence our materialization process was synchronized with this nightly refresh.
        </p>
        <p>Afterwards we defined our terminology binding using the provided methods in the
previous section. We stored these triples in a separate graph (together with the
ontology) to make these triples persistent over the nightly refresh process.</p>
        <p>All of this was executed on a virtual machine running Ubuntu 18.04 with 2 vCPUs
and 8GB of RAM. We used Ontotext GraphDB version 8.4.1 as our Graph database
management system, with “OWL2-RL (optimized)” reasoning enabled in the
endpoint. The Java Virtual Machine (GraphDB execution environment) was configured
with a maximum memory consumption of 4GB RAM. We measure the times for the
following individual steps in the process: (1) extracting the ontology from the given
database, (2) materializing the triples, (3) uploading the ontology (4) uploading the
triples into the RDF store (5) adding the annotation reasoning rules. This process was
measured for 20 consecutive executions.
3</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>Results</title>
      <p>The tool to extract the ontology, and to materialize the triples is publicly available at
https://gitlab.com/UM-CDS/fair/tools/triplifier. This tool is available as a stand-alone
java application or can be executed as a service in a docker container. Further
instructions are given in the repository itself.</p>
      <p>The created ontology, and the annotation triples are stored in
https://gitlab.com/UM-CDS/fair/data/maastro-rectum. Here we can see that 9
equivalent classes are defined, and 13 rules for mapping terminological values for 4
categorical variables. Next to these separate files, there is also an integrated ontology file,
where both extracted ontology and annotation rules are combined. We also included
an example query to retrieve whether a given variable is available, and in which table
this information is available. Furthermore, an extended query is available which also
retrieves the instances for a given patient. Due to patient privacy reasons, this query
can only be executed on the institutional RDF endpoint.</p>
      <p>The time measurements for the conversion process are given in Table 1. This
shows reasonable performance for a daily refresh where we see that the actual upload
and parsing process within GraphDB takes most of the time.
We have shown that it is possible to make data more FAIR, by automatically
extracting existing data schemas, and annotating these schemas with inferencing rules.
Furthermore, we have shown this process can be performed with materialization of data,
however it can be performed without the materialization step as well. In the latter
case, the ontology and reasoning rules can be used to deduce where to find specific
information in an existing database. Afterwards, specific database query mechanisms
can be used to retrieve the actual information. Hence, the materialization step in our
current workflow is an optional step, which can be used if necessary.</p>
      <p>This also applies to the reasoning rules used to annotate the given database. We
used an inferencing-enabled database in the current example; however this is not
mandatory. Although this is a machine-readable approach of describing the data, we
can describe the given information in a human-interpretable manner (e.g. male
gender, defined by ncit:C20197, is in table “patient” column “gender”, coded as the cell
value “m”).</p>
      <p>
        Instead of following the R2RML direct mapping recommendation [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ], we built our
own tool which does not make use of this recommendation. To deviate from this
recommendation, we had several arguments. First, R2RML Direct Mapping means we
have another intermediate description format (based on RDF), where we omitted this
by developing an ontology directly linked to the database. Hence, in our solution, the
end-users do not need to understand an additional description language (R2RML) to
understand where specific information is available. Second, different R2RML
implementations build specific SQL queries, which sometimes become more complex
(using join conditions) based on the database schema given. In our approach, we only
perform one SQL query to materialize the data (“SELECT * FROM &lt;tablename&gt;”),
which is almost sure to work on every RDBMS system (even a folder of CSV files
using the CSV JDBC connector1). Third, the W3C direct mapping specification
defines every cell value as a literal, belonging to an instance of the row. This is different
from our current implementation, where every cell is an instance of the column class.
1 http://csvjdbc.sourceforge.net/
The latter approach makes it possible to reason over equivalent classes, which is not
possible when table cells are literals. This could be solved by making custom R2RML
files, however these would not follow the direct mapping specification.
      </p>
      <p>
        Our work is currently limited in only handling RDBMS and CSV/TSV filesystems,
however the concept of annotating RDF classes to standardized terminologies is more
broadly applicable. For example, these annotations can be used in combination with
the Data2Services approach [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]. With respect to Data2Services, our approach differs
in the process of terminology binding. Where Data2Services requires one to build
SPARQL queries to convert terminologies and the data schema in one (or multiple)
queries, our approach only performs the terminology binding. This can be seen as
both a limitation and strength. The limitation is that the schema is going to vary,
making the task for the data consumer more intensive. On the other hand, by not changing
the data schema, it is clearer to the data consumer what the intrinsic issues with the
source data are [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. Furthermore, the knowledge to convert data (using SPARQL
queries) is not readily available in many hospitals or data providers. Hence, we feel that a
choice between our approach and Data2Services depends on the use case, and who
should handle data conversion.
      </p>
      <p>
        This brings us to the future work, where we want to build a web-interface on top of
this framework, to guide hospitals to annotate their data source(s). This would hide
the complexity of the reasoning rules to annotate specific database fields.
Furthermore, we want to use this platform (including the web-interface proposed) in the
Personal Health Train infrastructure [
        <xref ref-type="bibr" rid="ref7 ref8">7, 8</xref>
        ], to more rapidly include more FAIR data
stations to expand the network [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ].
      </p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Shortliffe</surname>
            ,
            <given-names>E.H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cimino</surname>
          </string-name>
          , J.J. eds: Biomedical Informatics: Computer Applications in Health Care and Biomedicine. Springer-Verlag, London (
          <year>2014</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Wilkinson</surname>
            ,
            <given-names>M.D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dumontier</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Aalbersberg</surname>
          </string-name>
          , Ij.J.,
          <string-name>
            <surname>Appleton</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Axton</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Baak</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Blomberg</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Boiten</surname>
            , J.-W., da Silva Santos,
            <given-names>L.B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bourne</surname>
            ,
            <given-names>P.E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bouwman</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Brookes</surname>
            ,
            <given-names>A.J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Clark</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Crosas</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dillo</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dumon</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Edmunds</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Evelo</surname>
            ,
            <given-names>C.T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Finkers</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gonzalez-Beltran</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gray</surname>
            ,
            <given-names>A.J.G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Groth</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Goble</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Grethe</surname>
            ,
            <given-names>J.S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Heringa</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          , 't Hoen,
          <string-name>
            <given-names>P.A.C.</given-names>
            ,
            <surname>Hooft</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            ,
            <surname>Kuhn</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            ,
            <surname>Kok</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            ,
            <surname>Kok</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            ,
            <surname>Lusher</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.J.</given-names>
            ,
            <surname>Martone</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.E.</given-names>
            ,
            <surname>Mons</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            ,
            <surname>Packer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.L.</given-names>
            ,
            <surname>Persson</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            ,
            <surname>RoccaSerra</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            ,
            <surname>Roos</surname>
          </string-name>
          , M.,
          <string-name>
            <surname>van Schaik</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            ,
            <surname>Sansone</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.-A.</given-names>
            ,
            <surname>Schultes</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            ,
            <surname>Sengstag</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            ,
            <surname>Slater</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            ,
            <surname>Strawn</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G.</given-names>
            ,
            <surname>Swertz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.A.</given-names>
            ,
            <surname>Thompson</surname>
          </string-name>
          , M.,
          <string-name>
            <surname>van der Lei</surname>
          </string-name>
          , J., van
          <string-name>
            <surname>Mulligen</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Velterop</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Waagmeester</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wittenburg</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wolstencroft</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zhao</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Mons</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          :
          <article-title>The FAIR Guiding Principles for scientific data management and stewardship</article-title>
          .
          <source>Sci. Data</source>
          .
          <volume>3</volume>
          ,
          <issue>160018</issue>
          (
          <year>2016</year>
          ). https://doi.org/10.1038/sdata.
          <year>2016</year>
          .
          <volume>18</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Meldolesi</surname>
            , E., van Soest,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Alitto</surname>
            ,
            <given-names>A.R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Autorino</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dinapoli</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dekker</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gambacorta</surname>
            ,
            <given-names>M.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gatta</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tagliaferri</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Damiani</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Valentini</surname>
          </string-name>
          , V.:
          <article-title>VATE: VAlidation of high TEchnology based on large database analysis by learning machine</article-title>
          .
          <source>Colorectal Cancer</source>
          .
          <volume>3</volume>
          ,
          <fpage>435</fpage>
          -
          <lpage>450</lpage>
          (
          <year>2014</year>
          ). https://doi.org/10.2217/crc.14.34.
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <given-names>A</given-names>
            <surname>Direct</surname>
          </string-name>
          <article-title>Mapping of Relational Data to RDF</article-title>
          , https://www.w3.org/TR/rdb-directmapping/.
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Emonet</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Malic</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zaveri</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Grigoriu</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dumontier</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>Data2Services: enabling automated conversion of data to services</article-title>
          .
          <volume>10</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Lovis</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>Digital health: A science at crossroads</article-title>
          .
          <source>Int. J. Med</source>
          . Inf.
          <volume>110</volume>
          ,
          <fpage>108</fpage>
          -
          <lpage>110</lpage>
          (
          <year>2018</year>
          ). https://doi.org/10.1016/j.ijmedinf.
          <year>2017</year>
          .
          <volume>12</volume>
          .006.
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7. van Soest,
          <string-name>
            <surname>JohanP.A.</surname>
          </string-name>
          , Dekker,
          <string-name>
            <given-names>AndreL.A.J.</given-names>
            ,
            <surname>Roelofs</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            ,
            <surname>Nalbantov</surname>
          </string-name>
          , G.:
          <article-title>Application of Machine Learning for Multicenter Learning</article-title>
          . In: El Naqa,
          <string-name>
            <given-names>I.</given-names>
            ,
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            , and
            <surname>Murphy</surname>
          </string-name>
          , M.J. (eds.)
          <source>Machine Learning in Radiation Oncology</source>
          . pp.
          <fpage>71</fpage>
          -
          <lpage>97</lpage>
          . Springer International Publishing (
          <year>2015</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Deist</surname>
            ,
            <given-names>T.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Jochems</surname>
            , A., van Soest,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Nalbantov</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Oberije</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Walsh</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Eble</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bulens</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Coucke</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dries</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dekker</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lambin</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Infrastructure and distributed learning methodology for privacy-preserving multi-centric rapid learning health care: euroCAT</article-title>
          .
          <source>Clin. Transl. Radiat. Oncol. 4</source>
          ,
          <fpage>24</fpage>
          -
          <lpage>31</lpage>
          (
          <year>2017</year>
          ). https://doi.org/10.1016/j.ctro.
          <year>2016</year>
          .
          <volume>12</volume>
          .004.
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Lambin</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Roelofs</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Reymen</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Velazquez</surname>
            ,
            <given-names>E.R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Buijsen</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zegers</surname>
            ,
            <given-names>C.M.L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Carvalho</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Leijenaar</surname>
          </string-name>
          , R.T.H.,
          <string-name>
            <surname>Nalbantov</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Oberije</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Scott Marshall</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hoebers</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Troost</surname>
          </string-name>
          , E.G.C.,
          <string-name>
            <surname>van</surname>
            <given-names>Stiphout</given-names>
          </string-name>
          ,
          <string-name>
            <given-names>R.G.P.M.</given-names>
            ,
            <surname>van Elmpt</surname>
          </string-name>
          , W., van der Weijden, T.,
          <string-name>
            <surname>Boersma</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Valentini</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dekker</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>'Rapid Learning health care in oncology' - An approach towards decision support systems enabling customised radiotherapy'</article-title>
          .
          <source>Radiother. Oncol</source>
          .
          <volume>109</volume>
          ,
          <fpage>159</fpage>
          -
          <lpage>164</lpage>
          (
          <year>2013</year>
          ). https://doi.org/10.1016/j.radonc.
          <year>2013</year>
          .
          <volume>07</volume>
          .007.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>