ICBO 2014 Proceedings
Pain Assessment Terminology in the NCBO
BioPortal: Evaluation and Recommendations
Werner CEUSTERS
Department of Biomedical Informatics and Institute for Healthcare Informatics
University at Buffalo
Buffalo, NY - USA
ceusters@buffalo.edu
Abstract—The International Association for the Study of Pain TABLE I. PAIN TERMS ANALYZED
(IASP) publishes since 1986 a relatively frequently updated list of Allodynia: pain due to a stimulus that does not normally provoke pain.
pain terms currently known as the ‘IASP Taxonomy’. It was Note: The stimulus leads to an unexpectedly painful response.
examined how nine terms defined in this taxonomy and used by Analgesia: absence of pain in response to stimulation which would
pain specialists to describe findings of somatosensory testing and normally be painful.
pain assessment are classified in the representational artifacts Dysesthesia: an unpleasant abnormal sensation, whether spontaneous or
accepted in the NCBO BioPortal. It was found that the majority evoked. Note: Special cases of dysesthesia include hyperalgesia and
of the BioPortal resources cover the terms poorly and that the allodynia.
quality of the hierarchies and the mappings are below acceptable Hyperalgesia: increased pain from a stimulus that normally provokes pain.
quality standards. It is concluded that without the BioPortal Hyperesthesia: increased sensitivity to stimulation, excluding the special
senses. Note: Hyperesthesia includes both allodynia and hyperalgesia,
studies of this nature are hard to perform, but also that for the
but the more specific terms should be used wherever they are
BioPortal to become an instrument which is useful for other applicable.
purposes than determining that its content is of poor quality, the Hyperpathia: a painful syndrome characterized by an abnormally painful
internal quality assurance principles used for its development reaction to a stimulus.
and maintenance need to be improved and documented. Hypoalgesia: diminished pain in response to a normally painful stimulus.
Hypoesthesia: decreased sensitivity to stimulation, excluding the special
Keywords—pain terminology, NCBO BioPortal, quality senses.
assurance Paresthesia: an abnormal sensation, whether spontaneous or evoked. Note:
it has been agreed to recommend that paresthesia be used to describe
an abnormal sensation that is not unpleasant while dysesthesia be used
I. INTRODUCTION preferentially for an abnormal sensation that is considered to be
unpleasant. There is a sense in which, since paresthesia refers to
Findings based on the various kinds of responses that patients abnormal sensations in general, it might include dysesthesia,
may report when subjected to stimuli to test their
somatosensory status, are typically described using terms such
as ‘allodynia’, ‘hyperesthesia’, and so forth. Standard It is therefore not possible to use these definitions in the
definitions for these terms were first proposed in 1979 [1] and Ontology for Pain-Related Mental Health and Quality of Life
are since then regularly updated by the International (OPMQoL) which is being developed as part of the NIDCR-
Association for the Study of Pain (IASP), in print for the last funded project R01DE021917 with the goal to integrate five
time in 1994 [2], with more regular electronic updates on the datasets gathered in four different countries from patients
IASP webpage [3], the last one May 2012 (subset in Table 1). suffering from one or other form of orofacial pain [4, 5].
These definitions, together with the IASP definition for It was hypothesized that alternatives could be found in the
‘pain’ as ‘an unpleasant sensory and emotional experience BioPortal of the National Center for Biomedical Ontology
associated with actual or potential tissue damage, or described
in terms of such damage’, suggest the hierarchy displayed in
Fig. 1. IASP pain assessment terminology hierarchy
Fig. 1 in which terms displayed in SMALL CAPS are the
immediate superordinate terms found in the definitions and the
arrows stand for the classical subsumption relation. Although
the individual definitions follow the Aristotelian style ‘an A is
a B which C’, the defined terms do not lead all together to a
complete directed graph with an overarching top, not even if all
29 IASP terms would be included. Furthermore, the terms
‘allodynia’ and ‘hyperalgesia’ have superordinate terms which
under their standard meanings should represent disjoined
classes: although sensation and sensitivity are certainly related,
nothing which is a kind of one can also be a kind of the other.
1
ICBO 2014 Proceedings
TABLE II. SUMMARY ASSESSMENT OF TERMINOLOGICAL AND ONTOLOGICAL QUALITY OF THE SEARCH TERM RELATED BIOPORTAL CLASSES RETRIEVED
Representational Artifact →
SNOMEDCT
SOPHARM
WHO-ART
COSTART
RH-MESH
ICD10CM
MEDDRA
NIFSTD
GALEN
ICPC2P
CTCAE
PHARE
NDFRT
OMIM
CRISP
MESH
SYMP
Totals
DOID
SNMI
CSSO
NCIT
ICNP
BDO
RCD
PDQ
MP
HP
Assessment Parameter Norm
AP1 IASP search terms covered 9 1 6 2 1 2 1 0 5 1 2 2 8 3 4 4 4 1 1 0 1 5 3 7 9 4 6 2
AP2 Number of direct class matches >8 1 6 2 1 2 1 1 5 1 2 8 9 7 4 5 5 1 1 1 1 5 11 7 14 4 6 2 113
Direct classes with wrong IASP
AP3 0 1 2 1 3 2 9
synonymy
AP4 Direct classes with definitions =AP2 2 1 2 1 4 7 4 5 3 1 6 36
Number of direct classes with
AP5 0 6 6 5 17
inappropriate homonymy
Number of additional direct
AP6 0 1 3 1 2 3 13
classes through spelling variants
AP7 Number of class matches >AP2 7 24 12 2 8 7 9 39 5 16 16 15 72 31 35 40 3 5 3 6 56 60 38 164 39 24 26 762
AP8 Foreign classes in hierarchy 0 8 7 5 1 1 5 7 6 40
Number of hierarchy classes
AP9 0 7 4 2 4 1 6 15 11 1 3 4 5 2 49 24 60
with disjointness violations
Evaluation
Maximum number of norm violations 6 8 8 6 8 6 1 8 6 8 8 8 8 8 8 8 6 6 1 6 8 8 8 8 8 8 8
Number of norm violations (except P8) 4 4 3 2 3 3 1 4 3 3 6 3 5 2 3 5 3 5 1 4 6 5 5 5 3 2 5
(NCBO) [6] which contains to date 370 representational To assess the extent to which the search terms are
artifacts with over 5.6 million classes. The objectives of the adequately covered in the individual BioPortal resources, and
work reported on here were to assess (1) whether these in the BioPortal as a whole, well-known quality assessment
resources offer a more adequate view on pain assessment criteria and recommendations – see results and discussions for
terminology, and (2) to what extent the BioPortal is a useful details – for terminologies [10, 11] and ontologies [12] were
instrument in determining whether (1) is indeed the case. used. To assess the adequacy of the backbone hierarchy within
individual resources 7 disjoint collections of in total 10 high
II. METHODOLOGY level groupings, inspired by the various preferred terms that
were retrieved, were constructed: [Adverse event], [Body part],
The nine terms – henceforth called ‘search terms’ – from Table [Discipline], [Disease, Disorder or Finding; NON-pain
1 were submitted to the on-line version of the BioPortal disorder; Pain / sensation finding], [Pharm. Effect / Endpoint],
Annotator [7] thereby using the following annotator options: [Function / Process; Technique / Therapy] and [Meta / Top].
(1) ‘longest match only’ unselected, (2) manual mappings Each class (with disambiguation where required as for instance
included, and (3) inclusion of all ancestors. With these options for ‘analgesia’) was classified into one of these groupings on
thus set, the annotator returned for each search term ST in this the basis of its preferred term. Examples of classes labelled
step one or more records, each such record containing (1) the Meta are classes with preferred terms such as Inactive Concept
unique identifier of a class CL in relation to which ST was and Unclassified, whereas the Top labelling include classes
found (2) the name of the representational artifact RA to which such as Snomed CT Concept and Topical descriptor [8].
CL belongs, (3) whether CL was retrieved on the basis of what
the annotator qualifies as a ‘direct match’ between ST on the The adequacy of the mappings between directly matched
one hand and a preferred term, synonym or identifier of CL on classes was assessed semi-automatically. Mapping records in
the other hand, or on the basis of being – mostly within RA, but which the semantics of at least one of the classes could not be
occasionally also within a representational artifact other than determined, were excluded. Records where only one of the
RA – an ancestor of a class which matches directly, and (4) the classes was marked as being Meta, were automatically tagged
preferred term PT of CL [8]. as obsolete. Records for which the preferred names of both
classes were identical, except in the case of ‘analgesia’ given
In a second step, all detailed terminological information its homonymous semantics, were automatically assigned as
available for each CL matching directly was retrieved, being correct. All other cases were assessed manually.
including a visualization of the subsumption graph and all the
mappings – if any at all – of CL to classes in other
representational artifacts within the BioPortal. The raw data III. RESULTS
and analysis file is available as [8]. Mappings between classes Querying for the 9 search terms in the BioPortal Annotator
from different representational artifacts are further qualified by exactly as displayed in Table 1 returned 762 annotation records
the BioPortal as being the result of enjoying shared Concept of which 113 were about in total 104 candidate annotation
Unique Identifiers (CUIs) from the Unified Medical Language classes labelled by the Annotator as ‘direct’ and which
System (UMLS), and/or being automatically generated using originated from 27 different sources [8] out of the 371 total
the Lexical OWL Ontology Matcher (LOOM), which generates artifacts at the time this work was performed. 17 annotation
mappings based on lexical similarity of the preferred name and records revealed that in the ICPC2, RH-MeSH and SNOMED
synonyms between pairs of ontologies [9]. CT some of the search terms matched directly to more than one
2
ICBO 2014 Proceedings
class (Table 3, AP5 in Table 2) – thus reflecting homonymy, TABLE III. MAPPING OF SEARCH TERMS TO PREFERRED TERMS IN THE
REPRESENTATIONAL ARTIFACTS
while 9 records showed that some of the classes were mapped
to by distinct search terms (AP3 in Table 2) – thus reflecting Representational
Grand Total
SNOMEDCT
Occurrence
SOPHARM
synonymy for the terms involved within the context of that
WHO-ART
RH-MESH
ICD10CM
COSTART
MEDDRA
Artifact →
NIFSTD
ICPC2P
NDFRT
PHARE
GALEN
OMIM
CTCAE
MESH
SYMP
CRISP
SNMI
source. Ignoring capitalization, the 104 direct annotation Search Term
DOID
CSSO
ICNP
NCIT
PDQ
BDO
RCD
MP
HP
Preferred Term
classes exhibited in total 25 distinct preferred terms. In Table 3 Allodynia 1 1 1 1 1 1 1 1 1 9 9
it is displayed how these preferred terms are related to the Allodynia
Hyperalgesia
1 1 1
1
1 1 1 1 1 8
1
8
1
original search terms in each resource. Analgesia 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 17 16
Analgesia 1 1 1 1 1 1 2 1 1 10 9
225 additional candidate annotation records [8] were No sensitivity to pain 1 1 2 2
retrieved by querying for three of the spelling variants pain agnosia
Analgesia [PE]
1
1
1
1
1
1
suggested by some of the retrieved preferred terms obtained by Hypalgesia 1 1 1
querying for the original search terms (Table 3): 77 for Pain Therapy
Dysesthesia 1 1 1 1
1
1
1
1 2
2
8
2
7
hyperaesthesia, 76 for hypesthesia, and 72 for hypoaesthesia. Dysesthesia 1 1 1 1 1 2 7 6
These records reveal that these terms match directly with 14 Paresthesia
Hyperalgesia 1 1 1
1
1 1 1 1 1 1
1
1 3 1 3 1 1 1 20
1
16
classes that were not matched with the original search terms, O/E - hyperesthesia present (& [hyperalgesia]) 2 2 1
thereby bringing ICD10 on board as extra representational Hyperalgesia [Disease/Finding]
Hyperalgesia 1 1 1 1 1 1 1
1
1 1 3 1 1 1 1
1
16
1
14
artifact. These records are not included in any further analysis. HYPERAESTHESIA 1 1 1
649 annotation records were labelled by the Annotator as Hyperesthesia 1 1 1 4 1 1 1 3 1 1 1 16 11
Hyperesthesia [Disease/Finding] 1 1 1
containing hierarchical ancestors of the classes matched HYPERAESTHESIA 4 4 1
directly, totaling 206 distinct ancestor classes with together 169 Hyperesthesia 1 1 1 1 1 3 1 1 1 11 9
distinct preferred terms [8]. One class, labelled Hyperpathia
Hyperalgesia
1 1 1 2
1 1
5
2
4
2
‘UMLS:OrphanClass’ appeared in 40 records involving the 8 Hyperpathia 1 1 1 3 3
representational artifacts labeled ICPC2, MESH, NDFRT, Hypoalgesia
HYPALGESIA
1
1
1 1 1 1 1 1 1 8
1
8
1
OMIM, PDQ, RCD, SNMI, and SNOMEDCT. 1036 mapping HYPOAESTHESIA 1 1 1
records were retrieved for all 104 classes matched directly to Hypoalgesia
Hypoesthesia 1 1
1
1 1
1
1 1
1 1 1 1
1 2 1
6
10
6
9
the search terms, of which 71 duplicates, yielding 965 records Hypesthesia [Disease/Finding] 1 1 1
further analyzed [8]. 399 of those records required manual Reduced sensation of skin 1 1 1
Sensory impairment 1 1 1
assessment. Hypesthesia 1 1 1 3 3
Hypoesthesia 1 1 1 1 4 4
Paresthesia 1 1 1 1 1 4 1 1 1 1 1 3 1 1 1 20 15
A. Quality of BioPortal Resources Retrieved Paresthesia [Disease/Finding] 1 1 1
paraesthesia 4 4 1
Table 2 provides – with the exception of assessment parameter Paresthesia 1 1 1 1 1 1 1 1 1 3 1 1 1 15 13
AP8 – a summary assessment of the terminological and Grand Total 1 6 2 1 2 1 1 5 1 2 8 9 7 4 5 5 1 1 1 1 5 11 7 14 4 6 2 113
ontological quality of the classes (and by extension of the Occurrence 1 6 2 1 2 1 1 5 1 2 2 9 7 4 5 5 1 1 1 1 5 4 7 9 4 6 2 95
resources from which they originate) that were retrieved for the B. Adequacy of the NCBO BioPortal
9 search terms. Further details about certain aspects are
available in Table 3 and Table 4. 9 APs are considered, and for Out of the 27 representational artifacts which have at least one
each AP a norm is determined. Table 2 thus illustrates that: class with a direct match to a search term, 22 have classes
which by the BioPortal are mapped to at least one other class
x only SNOMED CT covers the 9 search terms in the from another artifact. 618 of these mappings are within these
lexical form provided by the IASP (AP1), while 22 sources whereas 347 mappings are towards classes from 18
MeDDRA has complete coverage if lexical variants are target representational artifacts outside these sources. Of these
taken into account (AP6), (it was not checked whether 18, MeDDRA and RH-MeSH are the only two that have
resources contained atomic terms that through post- classes directly matched with the search terms, thus reflecting
coordination would allow to express the terms), the BioPortal documentation that mappings are not always
bidirectional.
x 5 resources do not make the distinctions in terminology
made by the IASP (AP3, details in Table 3), Table 5 quantifies the appropriateness of the mappings on
the basis of our methodology. The ‘B’ and ‘T’ following the
x 11 resources provide textual definitions for at least resource names in Table 5 indicate whether the resource
some of the classes (AP2, AP4), exhibits mappings bi-directionally resp. only incoming. B-
x 3 resources exhibit inappropriate homonymy for some mappings are only counted once in the totals. Mappings are
of the search terms (AP5), qualified as being excluded (‘Excl.’) from the analysis because
of either ambiguity or missing information on the side of the
x more than half of the resources exhibit for at least some classes mapped to (‘T?’) or being in the realm of the 22 source
of the search terms a hierarchy which on the basis of the classifications (’S?’). ‘Correct’ mappings result from (1) the
face value of the preferred terms is composed of automatic assignment of the adequacy assessment for pairs of
disjoint classes (AP9, details in Table 4), source and target classes with identical non-ambiguous
preferred terms (’SAME’), and the manual verification of (2)
x none of the representational artifacts cover the domain
classes with synonymous preferred terms, i.e. lexical variants
delineated by the IASP search terms adequately when
or descriptions (‘VARIANT’) and (3) classes with ambiguous
taking all assessment parameters into account.
3
ICBO 2014 Proceedings
preferred terms. Erroneous mappings (‘ERROR’) are brought TABLE V. CORRECTNES OF DIRECT CLASS MAPPINGS
about by (1) automatic determination of mapping to or from Error Correct Excl.
inactive classes (‘OBSO’) and manual verification of (2a)
mapping to or from classes with ambiguous meaning
HOMONYM
DISAMBIG.
% WRONG
VARIANT
Representational
(‘HOMONYM’), and (2b) inappropriate mappings between
WRONG
TOTAL
Artifacts
SAME
classes with unambiguous meanings (‘WRONG’). Table 6
OBSO
provides insight in the accuracy of the methods applied in the
T?
S?
BioPortal to create mappings, i.e. whether on the basis of the ACGT-MO T 1 1 100
UMLS Concept Unique Identifiers (‘cui’), the LOOM AI-RHEUM T 1 1 100
BDO B 3 20 5 3 31 10.7
algorithm (‘loom’) or both. COSTART B 31 12 44 40 44 171 34.4
CRISP B 7 4 17 2 19 21 70 36.7
CSSO B 1 2 17 8 3 31 10.7
IV. DISCUSSION CTCAE B 4 19 4 4 31 14.8
During ‘The Consensus Workshop: Convergence on an GALEN B 5 9 1 7 6 1 3 32 50
HIMC-ICD09 T 2 9 11 100
Orofacial Pain Taxonomy’, held March 30 – April 1, 2009, HIMC-LOINC T 3 3 0
Miami, Florida, which was attended by representatives from all HL7 T 4 2 6 0
major pain institutions, it was concluded that an adequate HOM-CLINIC T 3 3 0
treatment of the ontology of pain together with an appropriate HOMERUN-UHC T 3 3 0
HP B 7 2 19 2 4 34 30
terminology, is mandatory to advance the state of the art in ICD10 T 1 2 3 6 16.7
diagnosis, treatment and prevention [13]. ICD10CM B 2 7 6 4 19 13.3
ICPC2P B 1 6 21 48 1 10 87 9.21
As a first step, it was proposed to study the terminology IFAR T 2 2 0
and ontology of pain as currently defined. The ontological LOINC T 6 3 9 0
MEDDRA T 15 122 137 100
aspects have since then been covered in [4], and the underlying MESH B 2 8 5 44 13 4 1 16 93 19.7
principles thereof been applied, for instance, in the definition MP B 6 6 13 29 8 6 1 7 76 36.8
of new pain-related disease entities and classifications [14, 15]. NCIT B 9 9 4 43 23 3 1 23 115 24.2
NCIt-Activity T 2 2 4 50
NDFRT B 5 10 8 60 1 1 28 113 27.4
TABLE IV. GROUPING OF THE SEARCH TERMS IN DISJOINT UPPER NDF-RT T 2 4 7 24 8 1 1 2 49 37
CLASSES IN THE HIERARCHY OF THE REPRESENTATIONAL ARTIFACTS NIFSTD B 2 2 17 4 2 27 16
OMIM B 3 3 18 9 2 35 18.2
Representational PDQ B 3 4 4 8 3 1 11 34 31.8
SNOMEDCT
Grand Total
SOPHARM
WHO-ART
RH-MESH
ICD10CM
B 2 11 8 1 3 25 61.9
COSTART
Artifact →
MEDDRA
PHARE
NIFSTD
ICPC2P
NDFRT
PHARE
GALEN
OMIM
CTCAE
MESH
SYMP
CRISP
T 2 1 3 0
SNMI
Search Term
DOID
CSSO
PMA
ICNP
NCIT
PDQ
BDO
RCD
MP
HP
Grouping
RCD B 9 10 5 36 24 4 1 11 100 27.3
Allodynia 7 8 4 11 8 11 15 10 5 79
Disease or Finding 6 4 4 8 8 9 11 10 5 65 RH-MESH T 12 86 98 100
Function / Process 4 4 RPO T 2 2 4 50
Meta / Top 1 3 2 4 10
Analgesia 4 5 7 9 1 6 7 6 8 3 6 11 5 6 9 9 102 SNOMEDCT B 6 150 3 2 15 4 1 4 185 88.3
Body part 1 1 SOPHARM B 6 8 13 29 4 6 1 10 77 40.9
Discipline 3 3
Disease or Finding 3 2 3 1 1 1 7 2 9 3 3 9 9 53
SYMP B 7 14 55 17 18 111 22.6
NON-pain disorder 4 4 SYN T 2 2 4 50
Technique / Therapy
Function / Process
3
6
1 3
3
2
2
9
11
TRAK T 2 1 3 0
Meta / Top 2 3 2 1 2 1 11 WHO-ART B 34 7 10 18 17 86 59.4
Pharm. Eff./Endpoint 1 6 1 2 10
Dysesthesia 4 9 1 11 8 5 22 60
Grand Total 78 148 41 241 177 23 19 238 965 37.7
Adverse event 1 1 % of mappings 27.67 45.70 26.63
Disease or Finding 2 8 1 8 8 2 13 42
Function / Process 2 2
Meta / Top 1 1 3 1 9 15
Hyperalgesia
Body part
4 7
1
8 1 11 8 5 8 3 14 19 6 29 10 4 13 150
1 The analysis performed here is another response to the
Discipline
Disease or Finding
4
3 3 4 1 8 8 5 6 1 12 19 3 15 10 4 1 103
4
workshop’s recommendations with the goal to obtain more
NON-pain disorder
Function / Process 4 2
12 12
6
insight in how pain assessment terminology is dealt with in
Meta / Top
Hyperesthesia 4 8 5
3
8 1 11
2 2
8
2 1 14
18 5 14 4
24
86
representational artifacts such as widely used classification
Body part 1 1 systems, terminologies, and ontologies. At the same time, it
Disease or Finding 3 7 4 4 1 8 6 18 2 10 4 67
Function / Process 2 2 provided an opportunity to assess the usability of the NCBO
Meta / Top
Hyperpathia
1 1 4
4
3 2
10
1 4
6 23
16
43 BioPortal for a task of this nature, and the appropriateness of
Disease or Finding
Function / Process
4 8 3 14
2
29
2 the principles and methods applied in the BioPortal to present a
Meta / Top
Hypoalgesia 4 1 8
2
10
1 9
17 10 4 13 67
12 unified, highly standardized and ontology-like view on
Body part
Disease or Finding
1
3 1 8 8 13 10 4 1 48
1 resources which are qua structure and underlying design
NON-pain disorder 12 12 principles very different.
Meta / Top 2 4 6
Hypoesthesia 4 7 1 11 8 8 5 20 4 68
Body part 1 1
Disease or Finding 3 6 1 8 8 6 2 11 4 49 A. Are Resources in the BioPortal intrinsically flawed
Function / Process 2 2
Meta / Top
Paresthesia 7 4 2 4
1
8
3
8 1 11
2
8 8 5
1 9
18 5 15 3
16
107
As can be inferred from Table 2 and Table 3, all retrieved
Adverse event
Body part 2
1
2
1
4
resources, with – at first sight – the exception of MeDDRA and
Discipline 1 1 SNOMED CT, seem to perform quite poorly in terms of
Disease or Finding 6 2 2 2 7 4 1 8 8 6 1 18 2 11 3 81
Function / Process 2 2 coverage of the domain. Of course, some resources might have
Meta / Top
Grand Total
1 1 1 4 3 2 1
7 24 12 2 8 7 9 39 5 16 16 15 72 31 35 40 3 5 3
1 4 18
6 56 60 38 164 39 24 26 762 been designed with a specific purpose in mind and pain
4
ICBO 2014 Proceedings
assessment terminology therefor being out of their scope. It is TABLE VI. MAPPING SOURCES
however hard to imagine for what sort of purpose a term such Result cui cui, loom loom Grand Total
as paresthesia might be relevant and dysesthesia not: if one is
Error 29 20 218 267
present, all should be present. An exception is analgesia in the WRONG 5 4 69 78
sense of a procedure rather than of a symptom: there would OBSO 24 16 108 148
indeed be no place for any of the other terms in procedure HOMONYM 41 41
terminologies. Although there are indeed a few resources Correct 50 23 368 441
retrieved for which analgesia is the only term matched, these SAME 2 9 230 241
VARIANT 48 14 115 177
resources are not restricted to procedures. Some resources turn DISAMBIG. 23 23
out to exhibit a better coverage when spelling variants are used Excluded 22 17 218 257
in the queries, but not to the extent that it can explain the S? 31 31
overall lack of coverage. T? 22 17 187 226
Grand Total 101 60 804 965
Some resources, such as COSTART, MeSH and WHO- % Wrong 36.71 46.51 37.20 37.71
ART, suffer from the lack of discrimination between terms in
pairs such as hypoalgesia/hypesthesia, hyperalgesia/hyper-
esthesia, dysesthesia/paresthesia and analgesia/hypoalgesia. First there is the observation that through the mappings, 16
This was also found in SNOMED CT but only for classes that additional resources were discovered that contain classes which
were labelled ‘inactive’ thus reflecting that these mistakes map directly to classes which were retrieved by means of the
made in earlier versions were corrected afterwards. search terms. This can in part be explained by the absence of
the search terms in the synonym set of these additional classes,
15 resources exhibit through the eyes of the BioPortal a but upon further inspection, it turns out that in case of in total
backbone structure which at least can be frowned upon (Table 255 mappings for RH-MeSH and MeDDRA, as well as for
4). How can analgesia be a kind of nervous system (possible) resources which according to the syntax of the URIs
(COSTART), communication disorder (DOID - Human of the classes mapped to might be named ‘HOMERUN-UHC’,
Disease Ontology), or pharmacogenomics (PHARE)? How can ‘HOM-CLINIC’, ‘HIMC-LOINC’ and ‘HIMC-ICD09’, the
paresthesia be a kind of peripheral nervous system (OMIM), URIs returned by the annotator do not resolve at all [8]. The
hyperalgesia a kind of adrenal adenoma (WHO-ART) or former 4 resources are also not listed on the BioPortal webpage
neuroscience (CRISP)? One can assume sloppy design on the as being resources it contains, yet classes from them show up
side of the authors of these resources, or violation of the in the mapping results. In case of SNOMED CT, mappings are
principle that preferred terms should have face validity [10]: primarily involving classes which are marked as ‘inactive’.
thus in COSTART ‘nervous system’ might not mean nervous
system, but rather symptom related to the nervous system. Or, A second observation is that – after excluding these 255
and this leads to the next section, perhaps the BioPortal mappings as well as two others for which the meaning of the
represents the structure of these resources erroneously? source class could not be disambiguated – still almost 38% of
the mappings are inaccurate. There is no significant difference
B. Is the BioPortal itself, or are some design or quality in accuracy between mappings produced using LOOM or
UMLS CUIs alone. However, when both the LOOM and CUI-
assurrance principles behind it, intrinsically flawed?
methods suggest a mapping, the error rate increases to over
That something wasn’t right with the representation of WHO- 46%, thus almost the equivalent of flipping a coin.
ART in the BioPortal was noted by Ruttenberg in 2011 and as
such acknowledged by BioPortal staff who traced the issue C. Limitations
down to be caused by the WHO-ART source codes, but
nevertheless decided nothing to do about it at that time [16]. The work reported on here bears certain limitations. Although
And apparently never since: the version of WHO-ART that the data demonstrate (1) that the domain of pain assessment
showed up in the work reported about in this paper was version terminology is poorly covered in the BioPortal resources, (2)
‘2013AB’ which was uploaded to the BioPortal, according to that the way in which the BioPortal organizes the retrieved
the summary page, February 18, 2014, indeed without any classes hierarchically using the subclass relation is debatable,
attention to the known issues. The data presented here and (3) that the techniques used to map these classes between
demonstrate further that it is not just WHO-ART of which the resources are not quite adequate, no generalizations can be
representation in the BioPortal is problematic with respect to made to other domains. A further limitation is that the data
the semantics of the subclass relationship, but also 14 other were retrieved using the BioPortal website rather than the
resources that were retrieved on the basis of the search terms REST services. Perhaps these services offer better ways to
(Table 2, AP9). filter inadequate data, but if that were the case, one could
wonder why such filters are not used on the website.
Another indication that the BioPortal could benefit from
some quality assurance introspection comes from the finding Assessment of the correctness of the suggested hierarchy
that for 8 of the 27 resources retrieved the Annotator returned and the mappings was carried out with the quality criteria of
‘UMLS:OrphanClass’ as ancestor for 40 of the classes matched the OBO Foundry and adherence to the principles of
directly (Table 2, AP8). Ontological Realism in mind, neither of which are universally
accepted [17] yet gaining considerable attraction [18]. Thus it
Also the mapping results provide serious evidence in the is quite conceivable that reviewers outside the Foundry would
direction that quality improvement is required. report lower error rates, for instance by finding it perfectly
5
ICBO 2014 Proceedings
acceptable that the ‘concept’ of analgesia as a pharmaceutical ACKNOWLEDGMENT
effect in some drug is considered equivalent to the ‘concept’ of The work described is funded in part by grant 1R01DE021917-
analgesia as a procedure performed by an anesthesiologist or as 01A1 from the National Institute of Dental and Craniofacial
a state of a patient brought about by such procedure. At the Research (NIDCR). The content of this paper is the
other hand, since the review here was based by first flagging responsibility of the author and does not necessarily represent
results that for sure require manual evaluation (see the official views of the NIDCR or the NIH.
methodology) it might very well be that certain mapping- or
ancestor records were erroneously not flagged. In that sense,
the error rates presented here could very well be – modulo REFERENCES
mistakes made by sloppiness of the reviewer – the best case [1] IASP Subcommittee on Taxonomy. Pain terms: a list with definitions
scenario. Another limitation is that this study does point out the and notes on usage. Pain. 1979;6(3):249-52.
kind of mistakes and how to find them semi-automatically, but [2] Merskey H, Bogduk N. Classifications of chronic pain: Description of
is not conclusive on whether the root cause is in the source chronic pain syndromes and definition of pain terms. Report by the
International Association for the Study of Pain Task Force on
systems, the BioPortal, or a combination of both. Taxonomy. Seattle: IASP Press; 1994.
[3] International Association for the Study of Pain. IASP Pain Taxonomy.
V. CONCLUSION AND RECOMMENDATIONS 2012; Available from: http://www.iasp-pain.org/Content/Navigation
Menu/ GeneralResourceLinks/PainDefinitions/default.htm.
Without doubt, studies such as this one could not be carried out [4] Smith B, Ceusters W, Goldberg LJ, Ohrbach R. Towards an Ontology of
without a resource such as the BioPortal, or would require a lot Pain. In: Okada M, editor. Proceedings of the Conference on Logic and
more time and effort. Evenly without doubt, the BioPortal Ontology. Tokyo: Keio University Press; 2011. p. 23-32.
made it possible to reach the objectives of this study which [5] Ceusters W. An information artifact ontology perspective on data
were to find out (1) whether the sources in the BioPortal collections and associated representational artifacts. Stud Health
provide a more adequate view on pain assessment terminology Technol Inform. 2012;180:68-72.
– the answer being no, and (2) to what extent the BioPortal [6] Noy NF, Shah NH, Whetzel PL, Dai B, Dorf M, Griffith N, et al.
BioPortal: ontologies and integrated data resources at the click of a
itself is a useful instrument in determining whether (1) is mouse. Nucleic Acids Research. 2009 July;1:37.
indeed the case – the answer being yes. As a side effect, this
[7] Jonquet C, Shah NH, Musen MA. The open biomedical annotator.
study raises serious questions about the quality assurance Summit on translational bioinformatics. 2009;2009:56-60.
principles employed in the design and management of the [8] Ceusters W. Supplementary data to Pain Assessment Terminology in the
BioPortal, more specifically (1) about the quality of the NCBO BioPortal: Evaluation and Recommendations.2014: Available
resources the BioPortal accepts for inclusion – it might seem from: www.referent-tracking.com/SupplementarydatatoICBO2014.docx.
unfair to criticize a lack of clear best practice policies in the [9] Whetzel PL, Team N. NCBO Technology: Powering semantically aware
investigated resources while not distinguishing their different applications. J Biomed Semantics. 2013 Apr 15;4 Suppl 1:S8.
semantic expressivity, the point being however that the [10] Cimino JJ. Desiderata for controlled medical vocabularies in the twenty-
BioPortal itself does not allow for such distinctions and first century. Methods of Information in Medicine. 1998;37(4-5):394-
403.
‘promotes’ all resources as ontologies, (2) the suitability of
representing the hierarchy of these resources by means of the [11] Cimino JJ. In Defense of the desiderata. Journal of Biomedical
Informatics. 2006;39(3):299-306.
subclass relation, and (3) about certain house-keeping
[12] Schulz S, Jansen L. Formal ontologies in biomedical knowledge
operations. Quality seems thus far not to have been much of a representation. Yearbook of medical informatics. 2013;8(1):132-46.
concern to the BioPortal scientific community, as witnessed by [13] Ohrbach R, List T, Goulet J, Svensson P. Recommendations from the
the presence of only one paper in Pubmed that addresses the International Consensus Workshop: Convergence on an Orofacial Pain
topic [19]. Furthermore, although the BioPortal does indeed Taxonomy. Journal of Oral Rehabilitation. 2010.
offer a mechanism to users to make notes on the quality of [14] Nixdorf D, Drangsholt M, Ettlin D, Gaul C, de Leeuw R, Svensson P, et
BioPortal content [6], it doesn’t seem to be used much: the al. Classifying orofacial pains: a new proposal of taxonomy based on
BioPortal homepage displays a list of the 5 last notes ontology. Journal of Oral Rehabilitation. 2012;39(3):161-9.
submitted, of which the last three were submitted 7 months [15] Schiffman E, Ohrbach R, Truelove E, Look J, Anderson G, Goulet J-P,
et al. Diagnostic Criteria for Temporomandibular Disorders (DC/TMD)
prior to writing this paper, all three about a ‘request’ issued by for Clinical and Research Applications: Recommendations of the
user rboden – noted in the name of ‘Jesus’ as contact person – International RDC/TMD Consortium Network and Orofacial Pain
to add the following new term ‘We need someone with Special Interest Group. Journal of Oral and Facial Pain and Headache.
qualifications’. It is a bad sign that spam of this kind, whether 2014;28(1):6-27.
unnoticed or noted but not acted upon, is accepted. [16] Ruttenberg A, Ferguson R. Something amiss in translation of
WHOART. 2011 [cited 2014 April 15]; Available from:
For the BioPortal to become an instrument which is useful https://mailman.stanford.edu/pipermail/bioontology-support/2011-
for other purposes than determining that its content is of poor April/003124.html.
quality the following suggestions are in order: (1) do not accept [17] Brochhausen M, Burgun-Parenthoine A, Ceusters W, Hasman A, Leong
resources that violate standard subsumption principles, (2) TY, Musen M, et al. Discussion of “Biomedical Ontologies: Toward
Scientific Debate”. Meth Information in Medicine. 2011;50(3):217-36.
display for each resource quality metrics, rather than mere
quantity metrics, for instance the extent to which they follow [18] Schulz S, Balkanyi L, Cornet R, Bodenreider O. From concept
representations to ontologies: A paradigm shift in health informatics?
the principles of ontological realism or the OBO Foundry, and Healthcare Informatics Research. 2013;19(4):235-42.
(3) provide better documentation about the methods and [19] He Z, Ochs C, Agrawal A, Perl Y, Zeginis D, Tarabanis K, et al. A
algorithms used to present hierarchies and mappings, and about family-based framework for supporting quality assurance of biomedical
the internal quality assurance principles. ontologies in BioPortal. AMIA Symp proceedings. 2013;2013:581-90.
6