<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Representing SNOMED CT Concept Evolutions using Process Profiles.</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Werner CEUSTERS</string-name>
          <email>ceusters@buffalo.edu</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Jonathan P. BONA</string-name>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Division of Biomedical Ontology, Department of Biomedical Informatics, Jacobs School of Medicine and Biomedical Sciences, University at Buffalo</institution>
        </aff>
      </contrib-group>
      <abstract>
        <p>SNOMED CT is a very large biomedical terminology supported by a concept-based ontology. In recent years it has been distributed under the new release format 'RF2'. RF2 provides a more consistent and coherent mechanism for keeping track of changes over versions, even to the extent that - in theory at least any release will contain enough information to allow reconstruction of all previous versions. In this paper, using the January 2016 release of SNOMED CT, we explore various ways to transform change-assertions in RF2 into a more uniform representation with the goal of assessing how faithful these changes are with respect to biomedical reality. Key elements in our approach are (1) recent proposals for the Information Artifact Ontology that provide a realism-based perspective on what it means for a representation to be about something, and (2) the expectation that the theory of what we call 'process profiles' can be applied not merely to quantitative information artifacts but also to other sorts of symbolic representations of processes.</p>
      </abstract>
      <kwd-group>
        <kwd />
        <kwd>SNOMED CT</kwd>
        <kwd>ontological realism</kwd>
        <kwd>changes in ontologies</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        There are many differing views on what it means to do research conducted under the
term ‘ontology’, on what ontologies as representational artifacts exactly are, on what
the precise role of ontologies in information systems is, on what they should or should
not be used for, and on what qualities or capabilities they should have [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. Our view is
that an ontology should be a faithful representation of the part of reality that it covers
[
        <xref ref-type="bibr" rid="ref2 ref3">2, 3</xref>
        ]: looking through the c(/g)lasses of an ontology and how they are organized
therein, should give us exactly the same view as if we were looking directly at the
structure of the corresponding part of reality. This would hold both for T-box and
Abox assertions, as well as throughout time. Imagine on one side of a room an aquarium
with 12 fish, three of each of 4 types and some plants of various types, rocks, etc. and
on the other side a holographic simulation of that aquarium and its relevant
environment powered by a faithful ontology (including an A-box). If the simulation is
synchronized from the start with the exact configuration of the aquarium at that time,
though without access to the aquarium and its contents itself, then any change in the
aquarium would happen in exactly the same way as in the simulation. If we would run
the simulation faster, then we would see exactly what is going to happen in the
aquarium at a future time. To keep the simulation faithful, the maintenance contract for
the aquarium and its contents, should go hand in hand with a maintenance contract for
the ontology. If, for instance, fish of another type were to be added, then the ontology
would need to be updated accordingly. These updates should be such that, by
inspecting the ontology directly, we could find out exactly what happened in reality.
This means also that the formalism, language, data structures, i.e. the entire
representational machinery in and through which the ontology is expressed, must allow
us to detect which changes in the ontology correspond to changes in reality, and which
are purely ontology- or simulation internal. If the ontologist who maintained the
ontology for this simulation is replaced by one who is color-blind and therefore
changed the ontology so as to write out in words the names of typical fish colors on the
avatars in the simulation, then we should not be forced to believe that the goldfish in
our aquarium suddenly have the word ‘orange’ written all over their bodies.
      </p>
      <p>
        It is this line of thinking that formed the basis of Evolutionary Terminology
Auditing, a framework designed to measure quality improvements in ontologies over
time using reality as benchmark by taking into account changes in reality itself,
changes in our scientific understanding thereof, and pure editorial changes such as
corrections of mistakes or changes in representation that are not inspired by changes in
reality [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. In [
        <xref ref-type="bibr" rid="ref5 ref6">5, 6</xref>
        ] this framework was applied to 18 versions of the Systematic
Nomenclature of Medicine / Clinical Terms (SNOMED CT) [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ] with the conclusion
that changes to concepts over those versions do not necessarily correspond to
improvements in quality, and that many changes are due to idiosyncrasies in the
underlying ontology rather than to changes in the domain or in our scientific
understanding. In [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ], the method was found to have predictive power over future
quality improvements in the Gene Ontology. It was also applied to the Basic Formal
Ontology (BFO) [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] which led to a number of improvements to the framework itself [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ].
      </p>
      <p>
        In these past efforts we looked at consecutive versions of an ontology from the
perspective of reality, the goal being to assess quality improvements of the ontology in
terms of corresponding changes in reality. Here we look instead at mechanisms that an
ontology can offer to let us see changes in reality in a reliable way by examining the
changes in the ontology. We use as foundations the Basic Formal Ontology (BFO) [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]
and recent proposals for the Information Artifact Ontology (IAO) [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ] that provide a
realism-based perspective for what it means for a representation to be about something.
SNOMED CT is an ideal candidate for such analytical exploration as its distribution in
the last few years includes a new release format known as ‘RF2’ which is characterized
by a more elaborate, and – as we will demonstrate unfortunately not yet totally –
coherent and consistent representation of changes in its content to the extent that each
newly released version includes all previous versions rolled up inside itself. Our
exploration forms the basis for a long-term research objective to determine whether the
totality of assertions about changes in SNOMED CT rather than about external reality
constitutes in and of itself a valuable resource to identify patterns that allow detecting
mistakes in assertions about external reality that have thus far not been discovered.
      </p>
    </sec>
    <sec id="sec-2">
      <title>2. SNOMED CT as a concept-based ontology</title>
      <p>
        SNOMED CT – the name used to be an acronym for Systematic Nomenclature of
Medicine / Clinical Terms but is now considered a mere brand name of a new product
that grew out of this nomenclature – is developed by the International Health
Terminology Standards Development Organization (IHTSDO) and is claimed,
probably rightly, to be the largest healthcare terminology currently available [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. The
International Edition released on January 31, 2016 is supported by an ontology
consisting of 319,446 active concepts which are connected by in total 962,497 active
relationships and described by 1,097,028 active descriptions which link 999,639 terms
to these concepts. The relationships reported here are those generated by IHTSDO’s
EL++ description logic classifier on the basis of 655,312 active so-called stated
relationships which have been directly edited by authors or editors prior to running the
classifier on the logic definitions [11, p108].
      </p>
      <p>
        In addition to active components – ‘component’ being the umbrella term used by
IHTSDO for concept or relationship or description – SNOMED CT contains also
inactive components which were active in one or more prior versions but at some point
have been inactivated for one or other reason. Indeed, SNOMED CT is regularly
updated [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ], not only to correct mistakes, but also to reflect changes in biomedical
science. Concepts are classified under several hierarchies. Most top classes correspond
to the types of entities instances of which are encountered by clinicians during their
work (body parts, organisms, diseases, substances, procedures, etc.) while other top
classes correspond to types instantiated by descriptive elements of the SNOMED CT
knowledge representation itself, for example classes denoted by terms such as ‘inactive
concept’, ‘navigational concept’, and ‘core metadata concept’ [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ]. Although the
number of classes of this sort was originally – and is still – rather small, it is increasing
as a result of the move from Release Format 1 (RF1) to Release Format 2 (RF2). The
latter was introduced in 2012 to implement a more robust and consistent representation
of versions including an added hierarchy to represent metadata about the structure of
SNOMED CT itself [11 p127, 14].
      </p>
      <p>At the heart of SNOMED CT is the notion of ‘concept’ which in the SNOMED CT
documentation is defined as ‘a clinical idea to which a unique concept identifier has
been assigned’[11, p38]. What is represented by a specific concept cannot be
determined on the basis of the identifier, but ‘the meaning of a concept can be
determined from relationships to other concepts and from associated descriptions that
include human readable terms’ [11, p87]. Descriptions provide for each concept a
Fully Specified Name (FSN): ‘Each concept has at least one Fully Specified Name
(FSN) intended to provide an unambiguous way to name a concept. The purpose of the
FSN is to uniquely describe a concept and clarify its meaning’ [11, p40]. Furthermore:
‘Each FSN term ends with a “semantic tag” in parentheses. The semantic tag indicates
the semantic category to which the concept belongs (e.g. clinical finding, disorder,
procedure, organism, person, etc.). The “semantic tag” helps to disambiguate different
concepts which may be referred to by the same commonly used word or phrase’ [11,
p41]. For example, it is the semantic tag ‘morphologic abnormality’ in the FSN
‘Hematoma (morphologic abnormality)’ that disambiguates the concept to which this
FSN is assigned from a second concept with FSN ‘Hematoma (disorder)’. The former
is intended to be used for what ‘a pathologist sees at the tissue level’, while the latter
‘represents the clinical diagnosis that a clinician makes when they decide that a person
has a “hematoma”’ [11, p41].</p>
      <p>
        SNOMED CT’s authors have noted – and have to a certain extent started to act
upon, though not completely satisfactorily – the confusions around what ‘concept’
might denote [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ]. Despite their definition of ‘concept’ as a clinical idea, the term is
also stated to be a homonym for ‘concept identifier’ as well as for ‘the real-world
referent(s) of the concept identifier, that is, the class of entities in reality that the
concept identifier represents’ [11, p127]. One consequence is that there are doubts
about the sort of ontological commitments that are made by SNOMED CT authors and
editors [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ]. Another consequence is that SNOMED CT contains many ambiguities and
competing interpretations of, for instance, pathological conditions and disorders [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ].
      </p>
      <p>Another consequence of this ambiguity, the one we address specifically in this
paper, is that it also requires every occurrence of the word ‘concept’ in the SNOMED
literature – and indeed, in the literature about concept-based ontologies in general – to
be disambiguated in terms of whether it is used to denote something which is outside or
inside the ontology. Tumors, procedures and other entities clinicians come in contact
with while at work are outside SNOMED CT. Examples of something inside the
SNOMED CT representation are the SNOMED CT concept identifier ‘313029009’ and
the corresponding FSN ‘Brachytherapy – action (qualifier value)’, both of which are
supposed to denote the method involved in what it takes for a procedure to be of a sort
denoted both by ‘384692006’ and by the term ‘Brachytherapy procedure’.</p>
      <p>This ambiguity arises not only in the documentation but also in SNOMED CT
itself ! We can safely assume that the relationship (T1), between a procedure and a
qualifier value, extracted from the SNOMED CT relationships file and rendered in
human readable form by using FSNs is, as SNOMED CT puts it, about ‘a class of
entities in reality’, thus about something outside SNOMED CT. More concretely: the
term ‘Intracavitary brachytherapy (procedure)’ is inside SNOMED CT, but that what
this term denotes and of which a specific brachytherapy procedure carried out on a
specific patient is an instance (see section 3), is on the outside.</p>
      <p>‘Intracavitary brachytherapy (procedure)’
– ‘Method (attribute)’</p>
      <p>– ‘Brachytherapy – action (qualifier value)’
‘Actions by modality (qualifier value)’
– ‘Is a (attribute)’</p>
      <p>– ‘Action (qualifier value)’
‘Brachytherapy – action (qualifier value)’
– ‘Is a (attribute)’
– ‘Actions by modality (qualifier value)’
(T1)
(T2)
(T3)</p>
      <p>We can however no less safely assume that the triples (T2) and (T3) are to be
interpreted as statements about how SNOMED CT classifies certain actions, perhaps in
order to allow for easier browsing when SNOMED CT is used in some application as
an interface terminology. These are thus statements about something inside SNOMED
CT, rather than that ‘actions by modality’, on the outside, are a special kind of actions
in and by itself of which brachytherapy actions are an example. These distinctions are
important if we want to quantify reliably how much of external reality is represented in
SNOMED CT and how SNOMED CT is qualitatively improving as a representation
using reality as benchmark. For example, although (T2) and (T3) together use three
concepts – (1) ‘actions by modality’, (2) ‘action’, and (3) ‘brachytherapy’ – only two of
them, (2) and (3), correspond to an entity in reality.</p>
    </sec>
    <sec id="sec-3">
      <title>3. SNOMED CT as an Information Content Entity</title>
      <p>
        One way to address these issues is to perceive a version of SNOMED CT as an
instance of an Information Content Entity (ICE), i.e. the sort of entity which is
represented as the root of the IAO which is under development as a BFO-compatible
ontology for information artifacts [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]. Table 1 summarizes the definitions (Dn) and
elucidations (En) as they crystalized out of several proposals in the past few years [
        <xref ref-type="bibr" rid="ref10 ref18 ref19 ref20 ref21">10,
18-21</xref>
        ]. They are themselves based in part on the terms ENTITY, GENERICALLY
DEPENDENT CONTINUANT, MATERIAL ENTITY, QUALITY, FUNCTION and ROLE as well as
the notions of specific and generic dependence as defined in BFO [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. These definitions
allow us to perceive a version of SNOMED CT as an ICE of which concretizations
exist as INFORMATION ARTIFACTS in the form of, for example, a paper print out, or the
portion of a hard drive which contains the RF2 distribution files each one of which can
be rendered as a table on a computer screen by using appropriate software.
      </p>
      <p>
        In this light, the PORTION OF REALITY (PoR) described by SNOMED CT, includes,
from the ontological realist perspective as we perceive it [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]:
1. universals, such as, for instance, the universal denoted by the SNOMED CT
concept identifier ‘126838000’ which is further annotated by means of the
description (with ID ‘126016’) stating that the term ‘neoplasm of colon’ is an
allowed term since the January 2002 version;
2. relations, for instance the formal subsumption relation which in SNOMED CT
is represented by the concept identifier ‘116680003’ and by the corresponding
term ‘Is a (attribute)’;
3. instances, for example the one denoted by the concept ID ‘223502009’ and
corresponding FSN ‘Europe (geographic location)’; and,
4. configurations, for instance the one directly referred to in the ICE concretized
by the triple (T1), and the one indirectly represented by combining the triples
(T3) and (T2) used as examples in section 2.
      </p>
      <p>
        Note that the representation formalism used by SNOMED CT is not able to let us
distinguish universals from instances [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ]. Configurations are formally represented
through records in, for instance, the relationships file. This includes configurations
formed by ICEs themselves such as those denoted by records in Historical Association
Reference Sets (section 4.2). Relations are implicitly represented as such by being
subsumed by the concept with FSN ‘Attribute (attribute)’ and explicitly through their
specific position in records of, for example, the relationships file in RF2.
      </p>
      <p>To avoid the confusions arising from the word ‘concept’ as used in the SNOMED
CT documentation, we will use in this paper the term ‘SNOMED CT concept’ – or
‘concept’ for short – exclusively in the ICE sense, i.e. to denote a representational
element inside the SNOMED CT representation. If this representational element
succeeds in being about something (see D9), we will denote that something by terms
such as ‘the corresponding PoR’ or ‘the corresponding universal’. This holds also for
the other SNOMED CT components such as descriptions and relations. These terms
will exclusively be used to denote representational elements inside SNOMED CT.</p>
    </sec>
    <sec id="sec-4">
      <title>4. Changes in SNOMED CT</title>
      <sec id="sec-4-1">
        <title>4.1. Additions and deactivations</title>
        <p>
          The content of SNOMED CT evolves with each release. The types of changes made
include the addition and inactivation of concepts, descriptions, and relationships as well
as updates in definitions, and to a certain extent also the provision of motivations for
these changes. Once released, SNOMED CT components are persistent and their
identifiers are not reused [11, p45]. When a component becomes inactive this is
indicated by the value of the active field, a field which is present in all components.
Components continue to be distributed even when they are no longer active. This
allows a current release to be used to interpret data entered using an earlier release.
Whereas in RF1 the history mechanism was only used to annotate changes in concepts
and descriptions, RF2 annotates changes in a consistent fashion for all components,
though only for changes that occurred since the January 2002 release. Within RF2, all
changes in components are represented in the corresponding files by adding a new row,
with the same component ID, a new effective time and any necessary change in the
component values. As an example, Table 2 shows that the concept ‘301381004’ with
FSN ‘Discomforting present pain (finding)’ was set to active in release 20020131 and
to inactive in 20080131. Table 3 shows that during the life time of that concept, it
underwent considerable changes in its reported relationships to other concepts after full
DL classification. It must however be noted that the SNOMED CT documentation
remains silent on whether these reported changes are syntactical changes, effectual
changes or a combination thereof. From Table 3 alone it can indeed not be assessed
whether the relationship ‘Isa – Pain (finding)’ is truly inactivated, or whether it is still
active in the historical transitive closures, something that can be computed on the basis
of the history information available in RF2. Between 2009 and 2011 there were
typically more effectual changes (74%) than ineffectual ones (26%); within the
removals there was a high number of ineffectual changes (37%) whereas in the
additions there were on average more effectual changes (84%) than ineffectual ones
(16%) [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ]. Note, however, that ‘effectual change’ in [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ] is to be understood as a pure
change inside SNOMED CT from one version to another, and not as an assertion that
an effectual change corresponds to a change in reality or SNOMED CT’s authors’
knowledge thereof.
        </p>
        <p>Table 4 demonstrates how changes in the descriptions of concepts are similarly
logged. Only one description record with the same descriptionID field is current at any
point in time. The current record is the one with the most recent Effective Time before
or equal to the point in time under consideration. If the active field is false (‘0’), then
the description is inactive at that point in time. If it is true (‘1’), then the description is
associated with the concept identified by the conceptId field (not shown in Table 4).</p>
        <p>
          Table 4 points out another weakness in the concept-orientation adhered to by
SNOMED CT, and its consequent reliance on ‘meanings’ and all problems that arise
therefrom [
          <xref ref-type="bibr" rid="ref23">23</xref>
          ]. The SNOMED CT documentation states that ‘only limited changes may
be made to the “term” field, as defined by editorial rules’ [11, p145]. This is consistent
with the view that ‘the meaning of a concept can be determined […] from associated
descriptions that include human readable terms’ [11, p87]. This editorial rule is also
used as an argument for not retiring the concept to which it is attached in cases where
the FSN undergoes minor changes. Indeed, ‘Minor changes in the FSN are those
changes that do not alter its meaning. A change to the semantic type shown in
parentheses at the end of the FSN may sometimes be considered a minor change if it
occurs within a single top-level hierarchy (e.g. a change from a finding tag to a
disorder tag, or a change from a procedure tag to a regime/therapy tag), but a move to
a completely different top-level hierarchy is regarded as a significant change to the
Concept's meaning and is prohibited’ [11, p393]. Yet, a change from ‘finding’ to
‘context-dependent category’ (later renamed ‘situation’) is precisely a move from one
top-level category to another. Despite this change, the concept was not deactivated!
This can only be explained – unless it was a mistake introduced in 2003 and detected
prior to the release of the July 2009 version in which this concept became deactivated –
if we assume that the SNOMED CT editors at that time clearly realized that whatever
they change inside SNOMED CT does not have an impact on how matters are on the
outside.
        </p>
        <p>Thus what stays fixed – modulo the appearance of truly new entities such as new
drugs, mutated viruses, and, perhaps new disorder types caused by newly developed
techniques or chemicals – are the entities on the outside, the portions of reality denoted
by some SNOMED CT component on the inside. This holds, of course, also for the
massive number of changes that occur at the level of the SNOMED CT relationships:
although they clearly change ‘the meaning’ of the concept in many cases, they are still,
from a realist perspective, intended to denote the very same PoRs.
descriptionID Effective Active Description Term</p>
        <p>Time Type
410015012 20020131 1 Synonym Asthenia [D]
410015012 20020731 0 Synonym Asthenia [D]
666971011 20020131 1 FSN Asthenia [D] (finding)
666971011 20030131 0 FSN Asthenia [D] (finding)
1237162017 20020731 1 Synonym Asthenia [D]
1472277017 20030131 1 FSN [D]Asthenia (context-dependent category)
1472277017 20060731 0 FSN [D]Asthenia (context-dependent category)
1489933012 20030131 1 Synonym [D]Asthenia
2610401019 20060731 1 FSN [D]Asthenia (situation)
Legend: Active: 1=active, 0=inactive. Columns irrelevant for our purposes here are not shown. For
readability, Description Type identifiers have been replaced by their corresponding term – omitting their
semantic tag ‘(core metadata concept)’.</p>
      </sec>
      <sec id="sec-4-2">
        <title>4.2. Replacements</title>
        <p>
          RF2 replaces the ‘history mechanism’ implemented in RF1 [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ] by means of Historical
Association Reference Sets (HARS) and Component Inactivation Reference Sets
(CIRS). HARSs (Table 5) are used to indicate, for example, which deactivated
concepts are in one way or another related to other active concepts, and CIRSs (Table
6) to indicate the reasons for inactivating a component – such as errors, duplication of
another component and ambiguity of meaning [11, p506]. Records that express such
association are called reference set members. The primary purpose of these reference
sets is to specify which (if any) of these associations should be followed in a fashion
similar to following ‘Is a (attribute)’ relations when determining whether to retrieve a
record entry previously coded with a concept that has since then been inactivated.
Whereas ‘same as’ and ‘replaced by’ associations can be followed unproblematically,
the solution for ambiguous concepts related by ‘possibly equivalent to’ associations is
less clear-cut [11, p654].
        </p>
        <p>Use
From an ambiguous concept to one or more active concepts that represents one of
the possible meanings of the inactive concept.</p>
        <p>From a component to a namespace to which the component has been moved
From an erroneous or obsolete inactive component to a single active replacement
component.</p>
        <p>From a duplicate component to the active component that this component
duplicates.</p>
        <p>From an inactive classification concept such as "not otherwise specified" to the
active concept that was formerly its most proximal supertype.</p>
        <p>From an inactive classification concept derived from ICD-9 Chapter XVI
'Symptoms signs and ill-defined conditions' with the most similar active concept.</p>
        <p>From an inactive description which is inappropriate to the concept it is directly
linked to but instead should refer to the concept referenced.</p>
        <p>Interestingly, the very same concepts can not only appear as source concept in one
HARS member and as target concept in another HARS member, but also appear in
members of distinct HARSs. This allows the computation of association networks of
concepts by randomly selecting a concept from a HARS member and recursively
collecting all reference set members in which this concept appears with the goal of
processing each concept in the same way until no more concepts can be found.</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5. Discussion: towards process profiles for changes in SNOMED CT components</title>
      <p>
        As instances of ICE, thus continuants, components have a history – an occurrent
process – in which they participate for the entire time of their existence. This is
comparable to the history of an organism, i.e. the process in which an organism
participates for the entire temporal period during which it exists. For organisms, there
is a process of shorter duration with can be qualified as life, the process in which the
organism participates for the entire time it is alive and which is an occurrent-part [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] of
the organism’s history. In a similar sense, a component can be perceived as being alive
or dead, when declared to be active or inactive respectively. Furthermore, depending on
the type of component, it can be alive or dead in different ways. While a concept is
active, it can be ‘fully’ alive or, when it is marked for a pending move, ‘dying’ (Table
6). Prior to 2010, it could also be alive in a ‘limited’ way. In [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ], process profiles were
identified as something that is not numerically but qualitatively ‘the same’ in distinct
processes such as the ‘same’ temperature change of two rocks in our aquarium when
the water temperature changes. These processes each have as part an instance of a
quality process profile of exactly the same (determinate) type, i.e. ‘that part of a
process which serves as the target of selective abstraction focused on a sequence of
instances of determinate temperature qualities’ [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ]. It is speculated in [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ] that the
theory of process profiles can be applied not merely to quantitative information
artifacts but also to other sorts of symbolic representations of processes. It is this that
we try to achieve with respect to changes that occur in SNOMED CT components,
including memberships in HARSs and CIRSs. Although SNOMED CT’s RF2 format is
more coherent than its predecessor at the syntactic level, it requires more restructuring
of the data to arrive at a uniform view of what changed in relation to a specific concept,
and from there to infer what might have happened on the side of the corresponding PoR
(in case there is one).
      </p>
      <p>Table 7 uses 5 concepts (C1 … C5) as examples of how to construct process
profile representations (PPRs) in a (nearly) uniform way for the various sorts of
changes the concepts – from this perspective – underwent. Each PPR consists of 29
characters, 1 for each version, each one representing the status of some quality-like
feature that can be ascribed to the concept. The column ‘Attr.’ represents those features
at a level on a par with ‘temperature’, ‘color’, etc. For the rows with neutral
background, the combination of what appears in the ‘Attr.’ and ‘Value’ columns
represents those features at the most determinate level that we were able to measure,
comparable to ‘37.2 centigrade temperature’. Here ‘FSN+T-367’, f.i, means that the
term ‘General symptom NOS (finding)’, the 367th term (randomly numbered) out of
999,639 terms was ‘measured’ as the determinate value for ‘FSN’ (since we used a
FSN-thermometer, not a Synonym-thermometer). The table shows that this quality-like
feature was found to inhere in the concepts C1 through C4, whereby, as can be
determined from the respective PPRs, the histories of C1, C2 and C3 all share some
occurrent-part which instantiates the same most-determinate process profile universal,
and they do this at the same time (starting from the 14th version). ‘A’ in this case
stands for ‘active’, while ‘_’ means that there is at the respective time no instance of
the quality-like feature inhering in the concept. C4, in contrast, exhibits a different PPR
for this feature, one that is the result of a start in the 9th version. For the rows in grey
background, the value in ‘Value’ does not correspond to a measurement at a specific
point in time, but with a most-determinate PPR type itself. ‘DSP-05’, for instance, is
one out of 34 most-determinate PPR types for the quality-like feature ‘Dstatus’
(definitional status). It is C1, C2, and C3 that exhibit an instance of this type.</p>
    </sec>
    <sec id="sec-6">
      <title>6. Related work</title>
      <p>
        Computer scientists and logicians have developed a number of theoretical
approaches to deal with logical changes in description logic based ontologies. For
instance, in [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ] a model-theoretic semantics for ontology versioning based on
firstorder-logic is proposed that can be applied to ontologies expressed in RDF and OWL.
[
        <xref ref-type="bibr" rid="ref26">26</xref>
        ] reports on the development of a Multi-version Ontology REasoner (MORE) based
on using temporal logics to perform reasoning across multiple versions of ontologies.
MORE was tested on small ontologies in two different domains. In [
        <xref ref-type="bibr" rid="ref27">27</xref>
        ], a change
detection approach for OWL based on a logical change definition language and
temporal logic is proposed. [
        <xref ref-type="bibr" rid="ref28">28</xref>
        ] presents a tool for tracking and visualizing differences
between two versions of an ontology. [
        <xref ref-type="bibr" rid="ref29">29</xref>
        ] describes an interactive tool for visualizing
and exploring ontology changes that offers both overview and concept-based analyses.
      </p>
      <p>
        In [
        <xref ref-type="bibr" rid="ref30">30</xref>
        ] the rate of changes in SNOMED CT was characterized and quantified from
2002 to 2005, finding that most changes were occurring among relationships, and in
particular subsumption relationships, and concluding that implementers must ‘carefully
examine mechanisms for handling this degree of change’. By examining changes in
SNOMED CT over three years as recorded in the Component History and Concept
Model with a focus on the subset of concepts in the NLM CORE Problem List, four
types of changes (present in over 40% of the target concepts over the studied timespan)
were identified that are likely to impact health recordkeeping [
        <xref ref-type="bibr" rid="ref31">31</xref>
        ]. In [
        <xref ref-type="bibr" rid="ref32">32</xref>
        ], an approach
is presented to identify idiosyncrasies such as relation reversals (a particularly dramatic
type of structural change) in the evolution of SNOMED CT, finding 48 such reversals
since 2009. [
        <xref ref-type="bibr" rid="ref33">33</xref>
        ] demonstrates how changes between two SNOMED CT versions
affected a majority of concepts used in a legacy mapped interface terminology,
including unexpected effects of structural changes in SNOMED CT, and argues for a
consideration of impact on such implementations as part of terminology development.
Motivated by [
        <xref ref-type="bibr" rid="ref33">33</xref>
        ], [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ] presents indicators that can be computed to assess whether an
upgrade from one version to the next would be worth the effort.
      </p>
    </sec>
    <sec id="sec-7">
      <title>7. Conclusion</title>
      <p>
        Many efforts have been made to measure the amount and type of changes occurring
between SNOMED CT versions. To our best knowledge, a method based on the
representation of process profiles has thus far not been attempted. The results we
obtained in our exploration are promising although more work on our side towards
further harmonization is required. In any case, when in 2011 we asked ourselves the
question whether with RF2 SNOMED CT’s future is bright [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ], we were not able to
answer it. Now we believe we can: when complemented with an approach as proposed
here, it is! We strongly recommend any ontology to be distributed using such an
improved RF2 format – or semantic equivalent along the lines described here – since
without such mechanisms data annotated in terms of previous versions lose value
dramatically.
      </p>
    </sec>
    <sec id="sec-8">
      <title>Acknowledgments</title>
      <p>This work was supported in part by Clinical and Translational Science Award NIH 1
UL1 TR001412-01 from the National Institutes of Health, by grant R21LM009824
from the National Library of Medicine (NLM), and by grant 1R01DE021917-01A1
from the National Institute of Dental and Craniofacial Research (NIDCR). The content
of this paper is solely the responsibility of the authors and does not necessarily
represent the official views of the NIDCR, the NLM or the National Institutes of Health.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>F.</given-names>
            <surname>Fonseca</surname>
          </string-name>
          , “
          <source>The Double Role of Ontologies in Information Science Research,” Journal of the American Society for Information Science and Technology</source>
          , vol.
          <volume>58</volume>
          , no.
          <issue>6</issue>
          , pp.
          <fpage>786</fpage>
          -
          <lpage>793</lpage>
          ,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          , and
          <string-name>
            <given-names>W.</given-names>
            <surname>Ceusters</surname>
          </string-name>
          , “
          <article-title>Ontological realism: A methodology for coordinated evolution of scientific ontologies</article-title>
          ,
          <source>” Applied Ontology</source>
          , vol.
          <volume>5</volume>
          , no.
          <issue>3-4</issue>
          , pp.
          <fpage>139</fpage>
          -
          <lpage>188</lpage>
          ,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>R.</given-names>
            <surname>Arp</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          , and
          <string-name>
            <given-names>A. D.</given-names>
            <surname>Spear</surname>
          </string-name>
          ,
          <article-title>"Building ontologies with basic formal ontology,"</article-title>
          The MIT Press,,
          <year>2015</year>
          , p.
          <article-title>1 online resource</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>W.</given-names>
            <surname>Ceusters</surname>
          </string-name>
          , and
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          ,
          <article-title>"A Realism-Based Approach to the Evolution of Biomedical Ontologies,"</article-title>
          <source>Biomedical and Health Informatics: Proceedings of the 2006 AMIA Annual Symposium</source>
          , pp.
          <fpage>121</fpage>
          -
          <lpage>125</lpage>
          ,
          <string-name>
            <surname>Washington</surname>
            <given-names>DC</given-names>
          </string-name>
          : American Medical Informatics Association,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>W.</given-names>
            <surname>Ceusters</surname>
          </string-name>
          , “
          <article-title>Applying Evolutionary Terminology Auditing to SNOMED CT,” AMIA Annu Symp Proc</article-title>
          , vol.
          <year>2010</year>
          , pp.
          <fpage>96</fpage>
          -
          <lpage>100</lpage>
          ,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>W.</given-names>
            <surname>Ceusters</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K. A.</given-names>
            <surname>Spackman</surname>
          </string-name>
          , and
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          ,
          <article-title>"Would SNOMED CT benefit from Realism-Based Ontology Evolution?." In Teich JM</article-title>
          ,
          <string-name>
            <surname>Suermondt</surname>
            <given-names>J</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hripcsak</surname>
            <given-names>C</given-names>
          </string-name>
          . (eds.),
          <source>American Medical Informatics Association 2007 Annual Symposium Proceedings, Biomedical and Health Informatics: From</source>
          Foundations to Applications to Policy, Chicago IL,
          <year>2007</year>
          ;:
          <fpage>105</fpage>
          -
          <lpage>109</lpage>
          ..
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <surname>Donnelly</surname>
            <given-names>K</given-names>
          </string-name>
          ,
          <article-title>"</article-title>
          SNOMED CT:
          <article-title>The Advanced Terminology and Coding System for eHealth," Studies in Health Technology and Informatics - Medical and Care Compunetics 3</article-title>
          . Vol 121,
          <string-name>
            <surname>Bos</surname>
            <given-names>L</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Roa</surname>
            <given-names>L</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Yogesan</surname>
            <given-names>K</given-names>
          </string-name>
          et al., eds., pp.
          <fpage>279</fpage>
          -
          <lpage>290</lpage>
          , Amsterdam: IOS Press,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>W.</given-names>
            <surname>Ceusters</surname>
          </string-name>
          , “
          <article-title>Applying Evolutionary Terminology Auditing to the Gene Ontology,”</article-title>
          <source>Journal of Biomedical Informatics; Special Issue of the Journal of Biomedical Informatics on Auditing of Terminologies</source>
          , vol.
          <volume>42</volume>
          , no.
          <issue>3</issue>
          , pp.
          <fpage>518</fpage>
          -
          <lpage>529</lpage>
          ,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>S.</given-names>
            <surname>Seppälä</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          , and
          <string-name>
            <given-names>W.</given-names>
            <surname>Ceusters</surname>
          </string-name>
          ,
          <article-title>"Applying the Realism-Based Ontology-Versioning Method for Tracking Changes in the Basic Formal Ontology</article-title>
          .,
          <source>" Formal Ontology in Information Systems, Frontiers in Artificial Intelligence</source>
          and
          <string-name>
            <given-names>Applications P.</given-names>
            <surname>Garbacz</surname>
          </string-name>
          and O. Kutz, eds., pp.
          <fpage>227</fpage>
          -
          <lpage>240</lpage>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          , and
          <string-name>
            <given-names>W.</given-names>
            <surname>Ceusters</surname>
          </string-name>
          , “Aboutness:
          <article-title>Towards Foundations for the Information Artifact Ontology</article-title>
          ,” in International Conference on Biomedical Ontology, Lisbon, Portugal,
          <year>2015</year>
          , pp.
          <fpage>47</fpage>
          -
          <lpage>51</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <surname>IHTSDO</surname>
          </string-name>
          ,
          <string-name>
            <surname>"International Health Terminology Standards Development Organization - SNOMED CT® Technical Implementation Guide - January 2015 International Release (US English)</surname>
          </string-name>
          ,
          <year>" 2015</year>
          , p.
          <fpage>757</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>W.</given-names>
            <surname>Ceusters</surname>
          </string-name>
          , “
          <article-title>SNOMED CT revisions and coded data repositories: when to upgrade?,” AMIA Annu Symp Proc</article-title>
          , vol.
          <year>2011</year>
          , pp.
          <fpage>197</fpage>
          -
          <lpage>206</lpage>
          ,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>S.</given-names>
            <surname>Schulz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Suntisrivaraporn</surname>
          </string-name>
          , and
          <string-name>
            <given-names>F.</given-names>
            <surname>Baader</surname>
          </string-name>
          , “
          <article-title>SNOMED CT's problem list: ontologists' and logicians' therapy suggestions,” Stud Health Technol Inform</article-title>
          , vol.
          <volume>129</volume>
          , no.
          <source>Pt 1</source>
          , pp.
          <fpage>802</fpage>
          -
          <lpage>6</lpage>
          ,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>W.</given-names>
            <surname>Ceusters</surname>
          </string-name>
          , “
          <article-title>SNOMED CT's RF2: Is the future bright?,” Stud Health Technol Inform</article-title>
          , vol.
          <volume>169</volume>
          , pp.
          <fpage>829</fpage>
          -
          <lpage>33</lpage>
          ,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          ,
          <article-title>"Beyond concepts: ontology as reality representation,"</article-title>
          <source>Proceedings of the third international conference on formal ontology in information systems (FOIS</source>
          <year>2004</year>
          ), pp.
          <fpage>73</fpage>
          -
          <lpage>84</lpage>
          , Amsterdam: IOS Press,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>S.</given-names>
            <surname>Schulz</surname>
          </string-name>
          , and
          <string-name>
            <given-names>R.</given-names>
            <surname>Cornet</surname>
          </string-name>
          ,
          <article-title>"SNOMED CT's Ontological Commitment,"</article-title>
          ICBO: International Conference on Biomedical Ontology,
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          , ed., pp.
          <fpage>55</fpage>
          -
          <lpage>58</lpage>
          ,
          <string-name>
            <surname>Buffalo</surname>
            <given-names>NY</given-names>
          </string-name>
          : National Center for Ontological Research,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>S.</given-names>
            <surname>Schulz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Rector</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. M.</given-names>
            <surname>Rodrigues</surname>
          </string-name>
          et al., “
          <article-title>Competing interpretations of disorder codes in SNOMED CT and ICD,” AMIA Annu Symp Proc</article-title>
          , vol.
          <year>2012</year>
          , pp.
          <fpage>819</fpage>
          -
          <lpage>27</lpage>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>W.</given-names>
            <surname>Ceusters</surname>
          </string-name>
          , “
          <article-title>An information artifact ontology perspective on data collections and associated representational artifacts,” Stud Health Technol Inform</article-title>
          , vol.
          <volume>180</volume>
          , pp.
          <fpage>68</fpage>
          -
          <lpage>72</lpage>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Malyuta</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Rudnicki</surname>
          </string-name>
          et al.,
          <article-title>"IAO-Intel: An Ontology of Information Artifacts in the Intelligence Domain,"</article-title>
          <source>CEUR Workshop Proceedings</source>
          . pp.
          <fpage>33</fpage>
          -
          <lpage>40</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>W.</given-names>
            <surname>Ceusters</surname>
          </string-name>
          , and
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          , “
          <article-title>Foundations for a realist ontology of mental disease</article-title>
          ,
          <source>” Journal of Biomedical Semantics</source>
          , vol.
          <volume>1</volume>
          , no.
          <issue>10</issue>
          , pp.
          <fpage>1</fpage>
          -
          <issue>23</issue>
          ,
          <issue>9</issue>
          <year>December 2010</year>
          ,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Kusnierczyk</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Schober</surname>
          </string-name>
          et al.,
          <article-title>"Towards a Reference Terminology for Ontology Research and Development in the Biomedical Domain,"</article-title>
          <source>KR-MED</source>
          <year>2006</year>
          , Biomedical Ontology in Action.,
          <string-name>
            <surname>Baltimore</surname>
            <given-names>MD</given-names>
          </string-name>
          , USA
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <given-names>R. S.</given-names>
            <surname>Gonçalves</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Parsia</surname>
          </string-name>
          , and U. Sattler, “
          <article-title>Facilitating the analysis of ontology differences,” in Joint workshop on knowledge evolution and ontology dynamics (EvoDyn) 2011</article-title>
          , pp.
          <fpage>20</fpage>
          -
          <lpage>35</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          ,
          <article-title>"Against Fantology," Experience and Analysis</article-title>
          ,
          <string-name>
            <given-names>M. E.</given-names>
            <surname>Reicher</surname>
          </string-name>
          and J. C. Marek, eds., pp.
          <fpage>153</fpage>
          -
          <lpage>170</lpage>
          , Wien,
          <year>2005</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24]
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          , “
          <source>Classifying Processes: An Essay in Applied Ontology,” Ratio (Oxf)</source>
          , vol.
          <volume>25</volume>
          , no.
          <issue>4</issue>
          , pp.
          <fpage>463</fpage>
          -
          <lpage>488</lpage>
          , Dec 1,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [25]
          <string-name>
            <given-names>J.</given-names>
            <surname>Heflin</surname>
          </string-name>
          , and
          <string-name>
            <given-names>Z.</given-names>
            <surname>Pan</surname>
          </string-name>
          ,
          <article-title>"A model theoretic semantics for ontology versioning,"</article-title>
          <source>Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)</source>
          ,
          <year>2004</year>
          , pp.
          <fpage>62</fpage>
          -
          <lpage>76</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          [26]
          <string-name>
            <given-names>Z.</given-names>
            <surname>Huang</surname>
          </string-name>
          , and
          <string-name>
            <given-names>H.</given-names>
            <surname>Stuckenschmidt</surname>
          </string-name>
          ,
          <article-title>"Reasoning with multi-version ontologies: A temporal logic approach,"</article-title>
          <source>Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)</source>
          ,
          <year>2005</year>
          , pp.
          <fpage>398</fpage>
          -
          <lpage>412</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          [27]
          <string-name>
            <given-names>P.</given-names>
            <surname>Plessers</surname>
          </string-name>
          ,
          <string-name>
            <surname>O. De Troyer</surname>
            , and
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Casteleyn</surname>
          </string-name>
          , “
          <article-title>Understanding ontology evolution: A change detection approach</article-title>
          ,” Web Semantics: Science,
          <source>Services and Agents on the World Wide Web</source>
          , vol.
          <volume>5</volume>
          , no.
          <issue>1</issue>
          , pp.
          <fpage>39</fpage>
          -
          <issue>49</issue>
          , 3//,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          [28]
          <string-name>
            <given-names>N. F.</given-names>
            <surname>Noy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Kunnatur</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Klein</surname>
          </string-name>
          et al.,
          <source>"Tracking Changes during Ontology Evolution," Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)</source>
          ,
          <year>2004</year>
          , pp.
          <fpage>259</fpage>
          -
          <lpage>273</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          [29]
          <string-name>
            <given-names>M.</given-names>
            <surname>Hartung</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Kirsten</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Gross</surname>
          </string-name>
          et al., “OnEX:
          <article-title>Exploring changes in life science ontologies</article-title>
          ,
          <source>” BMC Bioinformatics</source>
          , vol.
          <volume>10</volume>
          , pp.
          <fpage>250</fpage>
          ,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          [30]
          <string-name>
            <given-names>K. A.</given-names>
            <surname>Spackman</surname>
          </string-name>
          , “
          <article-title>Rates of Change in a Large Clinical Terminology: Three Years Experience with SNOMED Clinical Terms,”</article-title>
          <source>AMIA Annual Symposium Proceedings</source>
          , vol.
          <year>2005</year>
          , pp.
          <fpage>714</fpage>
          -
          <lpage>718</lpage>
          ,
          <year>2005</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          [31]
          <string-name>
            <given-names>D.</given-names>
            <surname>Lee</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Cornet</surname>
          </string-name>
          , and
          <string-name>
            <given-names>F.</given-names>
            <surname>Lau</surname>
          </string-name>
          , “
          <article-title>Implications of SNOMED CT versioning</article-title>
          ,”
          <source>International Journal of Medical Informatics</source>
          , vol.
          <volume>80</volume>
          , no.
          <issue>6</issue>
          , pp.
          <fpage>442</fpage>
          -
          <issue>453</issue>
          , 6//,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref32">
        <mixed-citation>
          [32]
          <string-name>
            <given-names>S.</given-names>
            <surname>Tao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Cui</surname>
          </string-name>
          , W. Zhu et al.,
          <article-title>“Mining Relation Reversals in the Evolution of SNOMED CT Using MapReduce,”</article-title>
          <source>AMIA Summits on Translational Science Proceedings</source>
          , vol.
          <year>2015</year>
          , pp.
          <fpage>46</fpage>
          -
          <lpage>50</lpage>
          ,
          <issue>03</issue>
          /23,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref33">
        <mixed-citation>
          [33]
          <string-name>
            <given-names>G.</given-names>
            <surname>Wade</surname>
          </string-name>
          , and S. T. Rosenbloom, “
          <article-title>The impact of SNOMED CT revisions on a mapped interface terminology: Terminology development and implementation issues</article-title>
          ,
          <source>” Journal of Biomedical Informatics</source>
          , vol.
          <volume>42</volume>
          , no.
          <issue>3</issue>
          , pp.
          <fpage>490</fpage>
          -
          <issue>493</issue>
          , 6//,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>