This work presents some experiments in letting humans annotate citations according to CiTO, an OWL ontology for describing the function of citations. We introduce a comparison of the performance of different users, and show strengths and difficulties that emerged when using that particular model to characterise citations of scholarly articles.
The mere existence of a citation might not be enough to capture the relevance
of the cited work. For instance, some simple questions arise: is it correct to
count negative and positive citations in the same way? Is it correct to give
selfcitations the same weight of others? Is it correct to give a survey the same weight
of a seminal paper, by only counting the number of times it has been cited? A
more effective characterisation of citations opens interesting perspectives that
go beyond the quantitative evaluation of research products, as highlighted in [
To this end, the first issue to address is to identify a formal model for
characterising the nature of citations in a precise way – i.e. a citation model. The
citation model has to capture the citation functions, i.e. “the author’s reasons for
citing a given paper” [
Our work is positioned within the field of “Semantic Web and Cognition”.
In particular, the goal of this paper is to analyse weaknesses and strengths of
a particular citation model, studying how it has been used (and misused) by
the users for the annotation of citations. The model under investigation is CiTO
(Citation Typing Ontology)3 [
The paper is then structured as follows. In Section 2 we introduce previous works on classification of citations. In Section 3 we present our experimental setting and results: we go into details of the analysis performed by the humans and discuss the outcomes. Finally we conclude the paper sketching out some future works in Section 4. 2
Teufel et al. [
These works actually represent some of the sources of inspiration of CiTO
(the Citation Typing Ontology) developed by Peroni et al. [
In order to assess how CiTO is used to annotate scholarly articles, we compared the classifications performed by humans on a set of citations. The role of CiTO in such a process was obviously prominent. We in fact used the experiment to study the effectiveness of CiTO, to measure the understandability of its entities, and to identify some possible improvements, extensions and simplifications.
Our goal was to answer to the following four research questions (RQs): 1. How many CiTO properties have been used by users during the test? 2. What are the most used CiTO properties? 3. What is the global inter-rater agreement among users? 4. What are the CiTO properties showing an acceptable positive agreement between users?
4 ACL Anthology Network: http://clair.eecs.umich.edu/aan/index.php.
The test bed includes some scientific papers encoded in XML DocBook, containing citations of different types. The papers are all written in English and chosen among those published in the proceedings of the Balisage Conference Series (devoted to XML and other kinds of markup). We automatically extracted citation sentences, through an XSLT transform, from all the papers published in the seventh volume of the proceedings, which are freely available online5 . The XSLT transform is available at http://www.essepuntato.it/2013/citalo/xslt.
We took into account only those papers for which the XSLT transform
retrieved at least one citation (i.e. 18 papers written by different authors). The
total number of citations retrieved was 377, for a mean of 20.94 citations per
paper. We then filtered all the citation sentences that contain verbs (extends,
discusses, etc.) and/or other grammatical structures (uses method in, uses data
from, etc.) that carry explicitly a particular citation function. We considered
that rule as a strict guideline as also suggested by Teufel et al. [
The test was carried on, through a web interface, by five users, all academic but not necessarily expert in Computer Science (the main area of the Balisage Conference). None of them was an expert user of CiTO. Each user processed each citation sentence separately, with its full context window, and had to select one CiTO property for that sentence. Users could also revise their choices and perform the experiments off-line. There was no time constraint and users could freely access the CiTO documentation. We used R7 to load the data and elaborate the results. All the data collected are available online at http://www.essepuntato.it/2013/aic2013/test.
The experiments confirmed some of our hypotheses and highlighted some unexpected issues too. The first point to notice is that our users have selected 34 different CiTO properties over 40, with an average of 22.4 properties per user (RQ1). Moreover a few of these properties have been used many times, while most of them have been selected in a small number of cases, as shown in Table 1 (RQ2). There were 6 properties not selected by any user: compiles, disputes, parodies, plagiarizes, refutes, and repliesTo.
These data show that there is a great variability in the choices of humans.
In fact only 3 citations (out of 104) have been classified with exactly the same
5 Proceedings of Balisage 2011: http://balisage.net/Proceedings/vol7/cover.html.
6 The context window [
The results on the core CiTO properties were also confirmed by a slightly different analysis. We filtered only the 23 citations on which the users used at most two properties, as mentioned earlier in table Table 2. The k value on that subset of citations showed a moderate positive agreement between humans (k = 0.55, with 5 raters over 23 subjects). We had also moderate and high local positive agreement (i.e. k > 0.66) for 10 of the 15 properties used. The 5 properties showing an high positive agreement are citesAsDataSource (k = 0.77), citesAsPotentialSolution (k = 0.88), citesAsRecommendedReading (k = 0.7), credits (k = 0.74, that was not included in the core set mentioned above), and includesQuotationFrom (k = 0.74); the properties showing a moderate positive agreement are citesAsRelated (k = 0.6), citesForInformation (k = 0.4), critiques (k = 0.49), obtainsBackgroundFrom (k = 0.49), and usesMethodIn (k = 0.49). 3.3
One of our findings was that some of the properties were used only few times or
not used at all. This result can depend on a variety of factors. First, the authors
of the articles in our dataset, which are researchers on markup languages, use
a quite specific jargon so the citation windows resulted not easy to interpret
with respect to citations. Second, the positive or negative connotation of the
properties was difficult to appreciate. For instance, the fact that the
properties carrying negative judgements (corrects, derides, disagreesWith, etc.) are less
frequent than the others supports the findings of Teufel et al. [
Although we think the intended audience of the research articles one chooses
for such an experiment may bias the use of some properties, we also believe
that some properties are actually shared among different scholarly domains.
The property citesForInformation is a clear example. As expected, it was the
most used property, being it the most neutral of CiTO. This is in line with
the findings of Teufel et al. [
One of the reasons for having a low positive agreement in total (i.e. k = 0.16)
could be the high number of properties (40) defined in CiTO. To test this, we
mapped the 40 CiTO properties into 9 of the 12 categories identified by Teufel et
al. [
We also asked humans informally what were the cognitive issues they experienced during the test. Some of them highlighted that it was easy to get lost in choosing the right property for a citation because of the large number of possible choices. In addition, they also claimed that supporting the documentation of CiTO with at least one canonical example of citation for each property could be useful to simplify the choice. 4
The main conclusion for this paper is that classifying citations is an extremely difficult job also for humans, as demonstrated in our experiments on the properties of CiTO. The human analysis we presented herein gave us important hints on the understanding and adoption of CiTO, still showing some uncertainty and great variability. The identified strengths and weaknesses will be used to further improve the ontology, together with experiments on a larger set of users, decreasing the number of possible choices (for instance by using only the CiTO properties showing more agreement among humans).