Introduction
Fundamental functionalities of intelligent textbooks, such as content enrichment
[
23, 21, 9
] and personalization [
35, 31, 2
], exploit the knowledge contained in the
text to select relevant content and organise it according to a structure that
re ects the prerequisite relations (PR) among concepts [
17, 19
]. PRs are utterly
relevant in education since they establish which concepts are needed by a student
to understand a further concept without loss of information or misconceptions
[
20, 15, 1, 14
]. Despite their importance, PRs recognition, both in manual and
automatic form, still presents many open challenges. PR manual annotation is
in many cases a common way to achieve a PR-enriched intelligent textbook, but
is also an essential step for developing automatic textbook content modelling.
Systems for automatic PR learning (e.g. [
22, 1
]) often rely on manually annotated
gold standards [
29, 34, 15
] for evaluating or training machine learning algorithms.
Agreement rates [
6
] are often used to estimate annotations' reliability [
8
], but
these measures are conditioned by many factors, including the design of the
annotation task and the subjectivity of the phenomenon to be labelled [
5, 28
].
PR-annotations rarely achieve high agreement values unless the annotation task
is extremely guided [
18
]. A broad literature discusses causes of disagreement in
annotation [
12, 11, 7
], however this has not been fully explored for PRs. Our goal
Copyright c 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
is to investigate this matter and highlight possible causes of errors and issues, in
order to provide a way to improve PR modelling for intelligent textbooks. This
is useful especially when increasing the size of the dataset is not a practical way
to reduce the impact of errors (as in case of high demanding and time-consuming
annotation tasks like PR).
2
Experimental Study, Results and Discussion
Our analyses are conducted on PRET dataset described in [
4
], obtained by
asking ve domain experts to manually annotate PRs between the concepts
mentioned in a computer science textbook chapter[
10
]. Experts had to read the
text and in parallel annotate PRs among the encountered concepts. We stressed
this point since our annotation goal is to model the textbook content rather
than elicit the PR of a domain from the expert's mind. For the purpose of this
paper, in order to investigate the variability of annotations, we performed an
incontext annotation check, showing to each expert a subset of her/his annotations
and asking to con rm or revise their annotation, after reading again the text.
This process involved only \rare" annotations (concept pairs annotated only by
him/her) since errors are known to be more probable among them [
13
]. In the
end, the manual annotation resulted in 25 concept pairs annotated by all ve
experts, 46 annotated by four experts, 83 by three, 214 by two and 477 by only
one annotator, for a total of 845 unique pairs.
2.1 Interpretation Variability and Annotation Errors. This section deals
with the research question, i.e. which are the main causes of disagreement and
errors. Our results cannot be generalised but are discussed in the light of the
literature. During the annotation check mentioned above, we asked experts to
specify the motivation for their revisions, among the following: a) Not a
Concept : at least one concept of the pair is not a domain term; b) Background
Knowledge: the PR derives form the expert's domain knowledge since it's not
expressed in the text; c) Too Far : the concepts are too distant in the path; d)
Annotation Error : mistake due to distraction; e) Wrong Direction: the concepts
should be reversed; f ) Co-Requisites : there is no dependency relation between
the two concepts. The table in Fig.1 reports the distribution of these labels in the
revised pairs. Not a Concept is the most recurrent problem for all annotators:
common-usage terms like channel and system were added as domain concepts,
but then revised. As it will be discussed below, this error does not seem mostly
a trivial one, but an interpretation error commonly observed for educational
concepts [
33
], whose boundaries verge on subjectivity [
5
]. Other error types are
unevenly distributed: Too Far seems again related to interpretation and
subjectivity issues, that may arise from the expert questioning his/her annotation style
(e.g., annotating as PR also indirectly related concepts), Background Knowledge
might be again accounted as annotation style error, however in this case it is not
due to subjectivity, but to wrong interpretation of the task (i.e., the tendency
to infer from other sources PRs that are not explicitly written in the text). The
last three types seem all due to distraction. While distraction is inevitable (thus
some kind of revision could be a good practice), subjectivity and annotation
style errors are more tricky, but the possibility of revision could allow to obtain
a more reliable annotation [
26, 24
]. Concept interpretation error will be analysed
below more in depth since it is particularly critical for intelligent textbooks, even
independently of PR relations, as discussed also in [
33
].
Concept Interpretation. Despite the guidelines, the notion of concept was faced
in di erent ways, using a ne vs coarse-grained interpretation. To understand
the role played by concepts on agreement, we performed Pearson correlation
between root and leaf nodes (those with zero in- and out-degree respectively)
in annotations' graphs [
16
]. Our intuition is that graphs with high root nodes
correlation suggest an agreement on what is intended as core knowledge, i.e.
Primary Notions (PN) for the domain; similarly, correlation on leaves, re ecting the
paths' Learning Outcomes (LO), indicates a common interpretation of concepts
granularity. We noticed that revision helped to harmonise the annotations: while
prior the revision we obtained on average a stronger correlation between root
nodes (0.49) than leaves (0.29) (p < 0:05), on the revised graphs the correlations
became similar (0.47 for roots and 0.46 for leaves with p < 0:05).
2.2 Semantic and Lexical Relationships in PRs. We investigate here if
there are any semantic relations and linguistic patterns that can be identi ed as
frequently occurring in PRs with high vs low agreement. As Fig.2 shows, lexical
relations are the most frequent semantic type among PRs with high agreement,
covering more than 60% of cases, while Functional Relations are more common
among PRs with low agreement. This could be due to the fact that taxonomies,
by de nition, exploit a\dependency relation" to classify elements. On the other
hand the Functional Relation is highly a ected by the presentation of concepts in
the text, which entails a text interpretation and a subsequent low agreement due
to subjectivity. Such distributions highlight the need for annotation guidelines
with clear examples of PRs not involved in lexical relations, since those types
might be harder to identify along a text and could give raise to disagreement.
2.3 Holistic Nature of the Annotation Process. Previous work [
29, 34,
36
] mostly addressed PR annotation and evaluation as a pairwise combination
of concepts; we also used this approach [
3
]. However, we believe that such
approach over-simpli es the annotation and may result in misinterpretations of
the relations contained in the text. Indeed, semantic annotations often result in
(directed) graphs [
27, 25, 30
] in which each path is an interpretation of a relation
that arises from reading the whole text (this explains their holistic nature) and
should be evaluated accounting for those peculiarities. The commonly adopted
pairwise evaluation of PRs misses the interdependence between concepts involved
in a PR path and does not take into account the pedagogical characteristics of
the annotated graph as a whole. Temporal relation processing may represent an
interesting ground of comparison for PR, since precedence relation also shows
a transitive and sequential nature. Researchers in both elds encounter similar
limitations using traditional performance metrics used in information retrieval,
e.g. precision, recall [
32
]. A common scenario in both elds is when three items
A; B; C (concepts or events) are annotated by a rater such that A < B and
B < C, but another rater identi es the relation A < C: in such cases,
traditional agreement metrics will fail to identify A < C as a shared relation, even if
it is an implicit consequence of the other two relations [
32
]2. This suggests that
agreement could be computed by considering transitive edges and path
similarity in the two graphs. However, it is worth nothing that this approach involving
transitive closure might result in considering as PR some relations that are too
far in a path. This is an open issue, as well as the selection of proper metrics
to compare PR graph similarities. In our current work we are testing metrics
that work at di erent levels, among them Graph Edit Distance (GED), Vertex
Edge Overlap (VEO), PageRank, inter-agreement in learning pahts based on
the transitive characteristic of the PR (for details see [
16
]). We believe this
research direction is a relevant contribution wrt the current approaches in order
to compute not just agreement between annotators, but also between automatic
methods for knowledge graph creation, highly useful in intelligent textbooks.
Conclusions In this paper we investigated di erent forms of disagreement on
PR annotation that a ect PR processing and management, and consequently
textbook modelling. Our aim is thus to contribute to the modelling of PR
structure in textbooks by developing a subjectivity-aware PR coding protocol. The
analysis also opens the study of future research on using fuzzy PRs for
structuring knowledge behind text instead of yes/no (binary) PRs.
2 < indicates both the temporal relation before and the prerequisite relation