=Paper=
{{Paper
|id=None
|storemode=property
|title=Recommending Concepts to Experts: An Exploration of Recommender Techniques for
Collaborative Ontology Engineering Platforms in the Biomedical Domain
|pdfUrl=https://ceur-ws.org/Vol-897/session4-paper18.pdf
|volume=Vol-897
|dblpUrl=https://dblp.org/rec/conf/icbo/WalkSTNNM12
}}
==Recommending Concepts to Experts: An Exploration of Recommender Techniques for
Collaborative Ontology Engineering Platforms in the Biomedical Domain==
https://ceur-ws.org/Vol-897/session4-paper18.pdf
Recommending Concepts to Experts: An Exploration of
Recommender Techniques for Collaborative Ontology
Engineering Platforms in the Biomedical Domain
Simon Walk 1∗, Markus Strohmaier 1,2 , Tania Tudorache 2 , Natalya F. Noy 2 ,
Csongor Nyulas 2 and Mark A. Musen 2
1
Knowledge Management Institute, Graz University of Technology, Inffeldgasse 21a/II, 8010 Graz, Austria
2
Stanford Center for Biomedical Informatics Research, 1265 Welch Road, Stanford, CA 94305-5479, USA
ABSTRACT One way of augmenting users in collaborative systems is to pro-
Biomedical ontologies such as the 11th revision of the Internatio- vide them with adequate support and guidance for contributing their
nal Classification of Diseases and others are increasingly produced expertise (Ling et al., 2005). In systems such as Wikipedia (Cosley
with the help of collaborative ontology engineering platforms that faci- et al., 2007), recommender techniques are already used to help coor-
litate cooperation and coordination among a large number of users dinate collaborative tasks and support users in identifying articles to
and contributors. While collaborative approaches to engineering bio- work on. In the context of collaboratively engineering biomedical
medical ontologies can be expected to yield a number of advantages, ontologies, no such tools exist yet.
such as increased participation and coverage, they come with a num- The main focus of this paper is to explore recommender techni-
ber of novel challenges and risks. For example, they might suffer from ques for collaborative ontology engineering platforms in the bio-
low participation, lack of coordination, lack of control or other rela- medical domain. The paper is structured as follows: In Section 2 we
ted problems that are neither well understood nor addressed by the will discuss related work on collaborative engineering of biomedical
current state of research. In this paper, we aim to tackle some of ontologies as well as existing work on recommender techniques. In
these problems by exploring techniques for recommending concepts Section 3 we will map different recommender techniques to the bio-
to experts on collaborative ontology engineering platforms. In detail, medical ontology engineering domain. In Section 4 we will provide
this paper will (i) discuss different recommendation techniques from a short introduction to an exemplary collaborative ontology engi-
the literature (ii) map and apply these categories to the domain of neering project from the biomedical domain: the ICD-11 project.
collaboratively engineered biomedical ontologies and (iii) present pro- In Section 5 we will present results from three proof-of-concept
totypical implementations of selected recommendation techniques as recommender implementations. In section 6, we will conclude by
a proof-of-concept. discussing our approaches and point to future work.
The overall contributions of this paper are a high level mapping
1 INTRODUCTION of recommender techniques to collaborative ontology engineering
In the field of biomedical research, an increasing number of onto- platforms in the biomedical domain, and a proof-of-concept in the
logies are created collaboratively by a large group of people. form of implementations in the context of the ICD-11 project.
Examples include biomedical ontologies such as the Gene Onto-
logy (GO), the Ontology of Biomedical Investigations (OBI), the
National Cancer Institute Thesaurus (NCI) or the 11th revision of
2 RELATED WORK
the International Classification of Diseases (ICD-11). While col- 2.1 Collaborative Authoring & Ontology Engineering
laborative approaches to engineering biomedical ontologies can be In the context of collaboration platforms, many factors are known
expected to yield a number of advantages, such as increased parti- to influence the motivation and activity of individuals. For example,
cipation and coverage, higher acceptance or improved quality, they we know that transparent and well defined goals can affect groups
come with a number of novel challenges and risks. For example, and their performance. Making contributors aware of the utility of
recent research on collaborative authoring environments indicates their contributions represents another important factor (Ling et al.,
that the quality of collaboratively constructed products depends on 2005). Restructuring the payoff function, i.e. reducing the costs
the number of active participants, the ability to direct qualified and increasing the benefits of contributions, has also been identi-
participants to relevant content, amongst other factors (Kittur and fied as a potential intervention to increase participation (Cabrera and
Kraut, 2008). In addition, collaborative ontology engineering pro- Cabrera, 2002).
jects might suffer from a lack of coordination, lack of control, low An increasing number of biomedical ontologies, such as the
quality and other related problems that are neither well understood Gene Ontology, the National Cancer Institute Thesaurus, or the
nor addressed by the current state of research. These problems hin- ICD-11, are created using collaborative ontology engineering plat-
der progress and have the potential to jeopardize success of future forms. Requirements for collaborative ontology engineering plat-
ontology engineering projects in the biomedical domain. To tackle forms have been discussed by, for example, (Noy and Tudorache,
these challenges, new approaches for coordinating work and for 2008). Examples of existing platforms include OntoEdit, different
supporting contributors are needed. forms of Wikis such as Wiki@nt and OntoWiki or WebProtégé
(Tudorache et al., 2011). While many tools put an emphasis on col-
∗ To whom correspondence should be addressed: s.walk@student.tugraz.at laboration, we know little about how to effectively coordinate and
1
Walk et al
shape collaborative ontology engineering projects. iCAT Analytics rich textual content however might impair the overall usefulness of
(Pöschko et al., 2012) represents a first attempt to provide a detailed content-based recommendations.
analysis of collaborative ontology engineering processes.
3.2 Recommending concepts via collaborative filtering
2.2 Recommender systems
The intuition behind collaborative filtering is to find concepts or
The main objective of recommender technology is to provide per- items based on similar user behavior. This is accomplished by
sonalized suggestions that help an individual, or a group of indi- identifying behavioral patterns or usage patterns of users, and by
viduals, to find objects or items of interest (Burke et al., 2011). grouping them according to their similarity (Sarwar et al., 2001;
Historically, recommendations were used on e-commerce websites Goldberg et al., 1992).
to enhance impulsive buying behavior of customers. Usage patterns are patterns that define the interest of a user for
In the literature, a distinction between three basic recommenda- items or concepts. They either can be explicitly entered information
tion strategies can be identified: item/content based, collaborative such as ratings or implicit measures deducted from the amount of
filtering, and knowledge-based recommender techniques (Burke previously viewed, bought, or changed items by a single user. In the
et al., 2011). While content based recommender strategies focus context of collaboratively engineered biomedical ontologies, usage
on recommending items that are similar with regard to their con- patterns could be defined by grouping different behavior of users on
tent, collaborative filtering strategies focuses on recommending a collaborative platform such as adding, editing or even moving or
items that are similar with regard to behavioral patterns of similar deleting a concept, property or individual. Notes can be used as an
users. Knowledge-based recommender strategies focus on identif- indication for interest as well as viewing patterns or viewing times.
ying similar items using background/domain knowledge. Similarity for collaborative filtering is usually calculated by iden-
In the context of collaborative authoring systems, such as Wiki- tifying users with common interests which can be done by calcu-
pedia, recommender systems can be useful not only to help finding lating the similarity between their usage patterns. In collaborative
items of interest, but also to increase participation (Cosley et al., systems, interest is often modeled by explicit item rankings ente-
2007). While ontologies have been used as source of domain know- red by the users. Since this kind of information is typically not
ledge for generating recommendations (Sieg et al., 2010), applying available in biomedical contexts (users do not rate their favorite
recommender techniques to recommend concepts to experts is a concepts), other features, such as the number of times a concept has
novel problem. been viewed or changed or even other properties assigned to users
and concepts, can be used. To calculate similarity, a series of dif-
3 RECOMMENDING CONCEPTS TO EXPERTS ferent similarity measures, including Pearson correlation or cosine
In the following, we aim to explore how the three identified recom- similarity (Sarwar et al., 2001), is available.
mender strategies map onto collaborative ontology engineering Potentials & Limitations for collaborative filters are closely tied
platforms in the biomedical domain. to the extent that usage data is available. Collaborative filtering
approaches are particularly prone to the early phases of collabo-
3.1 Recommending concepts based on content rative ontology engineering projects, where little data about user
interactions is available. However, once sufficient data is collected,
The intuition behind content based recommender techniques is to
collaborative filtering approaches can recommend concepts that are
find and identify similar items or concepts by calculating and com-
not necessarily related content-wise, but through other usage pattern
paring similarity between content-related features of each concept.
based characteristic.
Content in an ontology can be defined as features of concepts, tex-
tual properties such as titles and descriptions, notes and discussions
or ratings. In the context of collaboratively engineered biomedical
ontologies, these properties can be term names, textual definitions 3.3 Recommending concepts using domain knowledge
of the concepts, clinical descriptions such as related/affected body The intuition behind knowledge based recommender systems is to
parts, synonyms, signs and symptoms, investigation findings such find similar concepts based on specific domain knowledge. They
as lab activities or measures needed to diagnose a disease or even represent a sub-class of content based recommender systems and
treatment plans. differ from them by using domain knowledge to create rules to
Similarity for content based recommender systems is usually determine the best item or concept to recommend, instead of simple
calculated on a common set of features or properties that all items properties.
or concepts share, using similarity or correlation measures such Domain knowledge is specific knowledge extracted from the envi-
as Pearson correlation, cosine similarity or the Jaccard coeffici- ronment of the system or the system itself. The biggest challenge
ent. Other textual similarity measures that could be used include in creating knowledge based recommendations is identifying via-
the Levenshtein distance, or simple overlap of textual properties ble domain knowledge, that will produce good results when used to
of concepts. Depending on the environment, different similarity calculate similarity. Recommendations can be produced by traver-
measures can yield different results when presented with the same sing along the edges in an ontology to identify related sub, super
input. or sibling concepts. In addition, linkages between ontologies can be
Potentials & Limitations for content based recommender systems exploited to generate knowledge based recommendations as well.
are closely tied to the properties and content of the concepts in Similarity for knowledge based recommender techniques can be
the ontology. An advantage of content-based recommendations is calculated using properties that could either be actively collected,
that they can be generated even in the absence of social usage by querying users for input, or implicitly by analyzing previous
data (i.e. they do not suffer from the ramp-up problem). A lack of behavior of a user.
2
Recommending Concepts to Experts
Fig. 1. Example of a collaborative ontology engineering platform: The iCAT user interface
Potentials & Limitations for knowledge based recommender 5.1 Content-based concept recommendations
systems are mostly related to the problem of distinguishing betw- We used real data excerpts, extracted from the ICD-11 and its log of
een basic content and domain knowledge. In addition, not every changes, to demonstrate how content based recommender systems
domain knowledge property will provide an equal basis for good can be applied to collaborative ontology engineering platforms in
recommendations. An advantage of knowledge based recommender the biomedical domain.
techniques is that, at least in some way, they are less dependent on For all users U and the set of all concepts C, we extracted their
the quantity of content and contributions. previously changed concepts Cu ⊆ C, together with all words from
the title and the words included in the definition of a concept c ∈
4 THE ICD PROJECT AND iCAT Cu .
In the following, we will briefly introduce the ICD-11 project. We Before doing so, we performed three additional tasks: (i) stop
will use the project later as an example to illustrate the adoption word (e.g.: is, as, and, so etc.) removal, (ii) stemming, a mechanism
of recommender techniques for collaborative ontology engineering used in natural language processing to reduce words to their stem
platforms in the biomedical domain. The International Classification and (iii) data cleaning, i.e. we have removed special characters from
of Diseases is a taxonomy maintained by the World Health Organi- the textual properties of the concepts. For this example, we used the
zation and is updated to a new revision around every decade. It is stop word list available from the Natural Language Toolkit.
used worldwide for monitoring health related expenses, to inform For similarity calculations, we used cosine similarity, as there is
policy makings, and to collect disease statistics. ICD-10 and all evidence that it provides good results for existing collaborative envi-
other predecessors of the ICD-11 were created by selected interna- ronments (Adomavicius and Tuzhilin, 2005). Results range from 0
tional experts; the production process was closed to the public. For (= completely unsimilar) to 1 (= identical).
ICD-11, the WHO decided on a more open, collaborative approach.
This new approach allows experts all over the world to contribute to ~ LB = {disease : 906, skin : 125, contact : 33, acute : 97}
W
ICD-11, using a web based collaboration platform called the ICD-11 ~ AR = {disease : 386, skin : 34, contact : 0, acute : 39}
W
Collaborative Authoring Tool (iCAT, as depicted in Figure 1) (Tudo- ~ RC = {disease : 272, skin : 841, contact : 399, acute : 65}
W
rache et al., 2010). There are currently around 100 international ~LZ1 = {disease : 1, skin : 1, contact : 0, acute : 0}
V
experts working on ICD-11. ~L56 = {disease : 0, skin : 1, contact : 0, acute : 1}
V
Next, we will present a number of proof-of-concept implementati- ~Z20 = {disease : 1, skin : 0, contact : 1, acute : 0}
V
ons of recommender techniques aiming to demonstrate how recom-
Table 1. W ~ u and V
~c (left) displaying excerpts of processed word-count
menders could be applied to collaborative ontology engineering
lists (right) from users and concepts used for cosine similarity calculations.
platforms in the biomedical domain.
W~ u represents all words and their respective number of appea-
5 VALIDATION: PROOF OF CONCEPT ~c collects all
rances in the title and definition of all d ∈ Cu and V
To study the general feasibility of recommending concepts to words and word counts per concept c ∈ C. Table 1 shows excerpts
experts, we implemented three selected recommendation techniques ~ and V
of W ~ depicting word lists for the users LB, AR and RC
for the ICD-11 project as a proof-of-concept. In the following we as well as excerpts of the concepts LZ1 (“LZ1 Impairment of nor-
will illustrate the implications of different recommender techniques mal functioning resulting from skin disease”), L56 (“Other acute
through examples using actual interaction data obtained from sele- skin changes due to ultraviolet radiation”) and Z20 (“Contact with
cted ICD-11 users. We will refer to these users as LB, AR and RC and exposure to communicable diseases”), after stemming and stop
from here on. word removal.
3
Walk et al
Cosine similarity was calculated for every pair cos(W ~ u, V
~c )
n
where u ∈ U and c ∈ C and stored in the user-concept similarity X
O(i, j) = N (k, j) + N (k, j) ∗ M (i, k) (1)
matrix MU,C (see Table 2). k=0,k6=i
LZ1 L56 Z20 The final results are illustrated in Table 6. Collaborative filtering
LB 0, 792159 0, 170571 0, 721471 recommends concepts for a user based on the concepts that similar
AR 0, 762570 0, 132542 0, 700839 users found interesting. In our example, the user LB and RC have a
RC 0, 809721 0, 659126 0, 488160
high similarity rating of 0, 5, and user RC has contributed to DBS
Table 2. The user-concept similarity matrix MU,C depicts the similarity
many times. Hence, the concept DBS is recommended (similarity
between users and concepts. The higher a value, the more similar a concept
is to previously changed concepts of the user. of 21) to LB.
H40.1 BP N CS XII DBS
In Table 2, recommendations for LB can be generated from LB - - - 21
MU,C by suggesting those concepts that have the highest simila- AR 0 15,96 59,85 -
rity values. In our example, this approach would recommend LZ1 RC 3 - - -
and Z20. Table 6. User-concept similarity matrix OU,C holds the similarity results
according to Equation 1 for the collaborative filtering approach. The higher
5.2 Collaborative filtering-based concept the similarity the likelier it is, that the user is interested in that concept.
recommendations
To demonstrate collaborative filtering-based recommendations, we
5.3 Knowledge-based concept recommendations
used log data from iCAT to calculate similarity between all user
u ∈ U . In this approach, two users are similar if they have modified Our final proof-of-concept implementation will assume that users
similar concepts. The concepts c ∈ C that have been changed by u are most likely interested in concepts that are ontologically related
from November 2009 to 30th August 2011, are denoted as sets Cu to the ones they have already shown interest in. The more a concept
(as seen in Table 3). For the users LB, AR and RC these concepts is interlinked or referred to by previously changed concepts in the
are H40.1 (“Primary open-angle glaucoma”), BP N CS (“Benign ontology, the more related it is to the interests of that specific indi-
proliferations, neoplasms and cysts of the skin”), XII (“Diseases vidual. An excerpt of the ICD-11 ontology, represented as directed
of the skin”) and DBS (“De Barsy syndrome”). graph using the is-a relationships, is depicted in Figure 2.
CLB = {H40.1, BPNCS, XII}
CAR = {DBS}
CRC = {BPNCS, XII, DBS}
Table 3. The set Cu (left) represents an excerpt of the set of all concepts
(right) modified by u from November 2009 to 30th August 2011 that was
used for calculating similarity between users.
Based on this matrix, we used the Jaccard coefficient to calculate
similarity between users based on the set of all concepts cu for all
u ∈ U resulting in a user-user similarity matrix MU,U (see Table 4)
as Mi,j = J(ui , uj ).
Fig. 2. Representation of an ICD-11 excerpt as a directed graph. Nodes
LB AR RC
refer to concepts while edges represent isA relationships. Dotted lines
LB 1 0 0,5
indicate modified concepts.
AR 0 1 0,33
RC 0,5 0,33 1
We assumed that user LB has only changed the concept B57.1
Table 4. The user-user similarity matrix MU,U lists the Jaccard coefficient
similarity values between all pairs of users.
(“Acute Chagas disease without heart involvement”). To explore
related concepts, we followed links in the ontology until either a
We introduced an arbitrary threshold minSimilarity set at predefined depth level was reached or enough highly interlinked
0.0001 which excludes user pairs with very little similarity. Based concepts were discovered. Table 7 depicts all values for expansi-
on that modified table, we start to count the number of changes done ons of concepts B57 (“Chagas’ disease”), B57.2 (“Chagas disease
by uj on every single concept c ∈ Cuj and store the results in the (chronic) with heart involvement”), B57.3 (“Chagas disease (chro-
matrix NU,C (see Table 5). nic) with digestive system involvement”), SC1 (“Selected Cause is
Remainder of certain infectious and parasitic diseases in the Con-
H40.1 BP N CS XII DBS densed and Selected Infant and child mortality lists”), SC2 (“Sele-
LB 2 1 12 0 cted Cause is Trypanosomiasis”) and M ortality (“Tabulation list
AR 0 0 0 20 for mortality”).
RC 0 12 45 14
Next we traversed along all paths, as shown in Figure 2 and
Table 5. NU,C , the user-concept change count matrix lists the number of
Table 7, from all previously edited concepts, counting the number of
changes done by every user u ∈ U to every concept c ∈ C.
encounters of each concept. The higher the number of encounters,
The values for the user concept similarity matrix OU,C are the more weight it will receive in a ranking of concepts for user u.
calculated as depicted in Equation 1.
4
Recommending Concepts to Experts
Rk. Content Based Score Collaborative Filtering Score Knowledge Based Score
1 L02.9 ’Cutaneous abscess, furuncle 0.381792 II Neoplasms 9.120824 ’Selected Cause is Remainder of 42
and carbuncle, unspecified’ certain infectious and parasitic dise-
ases in the Condensed and Selected
General mortality lists’
2 L02.8 ’Cutaneous abscess, furuncle 0.372091 VI ’Diseases of the nervous system’ 9.119071 ’Ectodermal dysplasia syndromes’ 34
and carbuncle of other sites’
3 ’Chronic ulcer skin’ 0.359489 XI ’Diseases of the digestive 8.155308 ’Chromosomal disorders affecting 28
system’ the skin ’
4 ’Congenital skin anomaly other’ 0.359119 E09-E1B ’Diabetes mellitus’ 8.136958 ’Genetic, chromosomal and develo- 24
pmental disorders affecting the skin
’
5 ’Pediculosis/skin infestation other’ 0.356631 V ’Mental and behavioural disor- 8.117054 XII ’Diseases of the skin’ 23
ders’
6 ’Fear of skin disease other’ 0.350870 IX ’Diseases of the circulatory 8.106256 ’Genetic syndromes affecting nails’ 19
system’
7 ’Malignant neoplasm of skin’ 0.345841 A15 ’Respiratory tuberculosis, 8.100946 ’Tabulated - Other diseases of the 18
bacteriologically and histologically skin and subcutaneous’
confirmed’
8 ’Dysplasia syndromes with 0.338079 I21 ’Acute myocardial infarction’
8.058171 L20-L30 ’Dermatitis and eczema’ 17
skin/mucosae involvement’
9 ’Tabulated - Other diseases of the 0.333487 H25 ’Senile cataract’ 7.100532 ’Dysplasia syndromes with prema- 17
skin and subcutaneous’ ture ageing appearance’
10 ’Tabulated - Other malignant neo- 0.327885 VII ’Diseases of the eye and 6.137702 ’Parasitic infestations affecting the 16
plasms of skin’ adnexa’ skin’
Table 8. Ranked concept recommendations for the user RC according to our three different recommender techniques. The higher the score, the better the
concept is ranked for recommendation. Every approach provides different results regarding level of detail, scope and sub-domain.
Depth B57.1 B57.2 B57.3 B57 SC1 SC2 Mortality Burke, R. D., Felfernig, A., and Gker, M. H. (2011). Recommender systems: An
1 1 0 0 1 0 0 0 overview. AI Magazine, 32(3), 13–18.
2 1 0 0 2 1 1 0 Cabrera, A. and Cabrera, E. F. (2002). Knowledge-Sharing dilemmas. Organization
3 1 0 0 2 2 2 2 Studies, 23(5), 687–710.
4 1 0 0 2 2 2 3 Cosley, D., Frankowski, D., Terveen, L., and Riedl, J. (2007). Suggestbot: using intel-
Table 7. A listing of the number of encounters on different depth levels for ligent task routing to help people find work in wikipedia. In Proceedings of the 12th
international conference on Intelligent user interfaces, IUI ’07, pages 32–41, New
concepts CLB of user LB when path traversing Figure 2 to find most
York, NY, USA. ACM.
interlinked concepts for calculating knowledge based recommendations.
Goldberg, D., Nichols, D. A., Oki, B. M., and Terry, D. B. (1992). Using collaborative
filtering to weave an information tapestry. Commun. ACM, 35(12), 61–70.
6 DISCUSSION & FUTURE WORK Kern, R., Körner, C., and Strohmaier, M. (2010). Exploring the influence of tagging
The final recommendations for RC (see Table 8) produced by motivation on tagging behavior. In Proceedings of the 14th European conference on
Research and advanced technology for digital libraries, ECDL’10, pages 461–465,
our different proof-of-concept recommendation algorithms differ Berlin, Heidelberg. Springer-Verlag.
significantly with regard to level of detail, scope and sub-domain. Kittur, A. and Kraut, R. E. (2008). Harnessing the wisdom of crowds in wikipedia: Qua-
Recently or currently browsed concepts could be included in the lity through coordination. In CSCW ’08: Proceedings of the ACM 2008 conference
similarity calculations to increase the scope of the generated recom- on Computer supported cooperative work, pages 37–46. ACM.
Ling, K., Beenen, G., Ludford, P., Wang, X., Chang, K., Li, X., Cosley, D., Frankowski,
mendations. While we believe that concept recommender systems
D., Terveen, L., Rashid, A. M., Resnick, P., and Kraut, R. (2005). Using social
could represent a useful instrument to augment user experience psychology to motivate contributions to online communities. Journal of Computer-
on collaborative ontology engineering platforms in the biomedical mediated Communication, 10(4).
domain, we have not performed evaluations of our implementations Noy, N. F. and Tudorache, T. (2008). Collaborative ontology development on the (
yet. Understanding what kinds of recommender techniques are use- semantic ) web. Nature Biotechnology.
Pöschko, J., Strohmaier, M., Tudorache, T., and Musen, M. A. (2012). Pragmatic analy-
ful in what kind of contexts, including a more sophisticated analysis sis of crowd-based knowledge production systems with iCAT Analytics: Visualizing
of user-behavior similar to (Kern et al., 2010), represent important changes to the ICD-11 ontology. In Proceedings of the AAAI Spring Symposium
next steps for future research. We believe that understanding the uti- 2012: Wisdom of the Crowd. Accepted for publication.
lity of recommender systems to steer and augment user activity in Sarwar, B., Karypis, G., Konstan, J., and Reidl, J. (2001). Item-based collabora-
tive filtering recommendation algorithms. In Proceedings of the 10th International
collaborative ontology engineering projects represents an exciting
Conference on World Wide Web, pages 285–295, New York, NY, USA. ACM.
avenue for future biomedical research. This work is relevant for the Sieg, A., Mobasher, B., and Burke, R. (2010). Ontology-based collaborative
future design of collaborative ontology engineering platforms, and recommendation. Computing.
for operators of such systems as well as for users and contributors. Tudorache, T., Falconer, S. M., Nyulas, C. I., Noy, N. F., and Musen, M. A. (2010). Will
Semantic Web Technologies Work for the Development of ICD-11? In Proceedings
of the Ninth International Semantic Web Conference.
REFERENCES Tudorache, T., Nyulas, C., Noy, N. F., and Musen, M. A. (2011). WebProtégé: A
Adomavicius, G. and Tuzhilin, A. (2005). Toward the next generation of recommender Distributed Ontology Editor and Knowledge Acquisition Tool for the Web. Semantic
systems: A survey of the state-of-the-art and possible extensions. IEEE Trans. on Web Journal, 11-165.
Knowl. and Data Eng., 17, 734–749.
5