=Paper=
{{Paper
|id=None
|storemode=property
|title=Hierarchical Text Classification for Supporting Educational Programs
|pdfUrl=https://ceur-ws.org/Vol-835/paper3.pdf
|volume=Vol-835
|dblpUrl=https://dblp.org/rec/conf/iir/JuRMV12
}}
==Hierarchical Text Classification for Supporting Educational Programs==
https://ceur-ws.org/Vol-835/paper3.pdf
Hierarchical Text Classification for Supporting
Educational Programs
Qi Ju∗, Chiara Ravagni?†, Alessandro Moschitti∗, and Giampiero Vaschetto?
∗DISI, University of Trento, Italy
{qi,moschitti}@disi.unitn.it
?Centro Studi Erickson, Italy
{chiara.ravagni,giampiero.vaschetto}@erickson.it
†University of Nuremberg, Germany
Abstract. More than two decades have passed since the first design of the
CONSTRUE system [2], a powerful rule-based model for the categorization
of Reuters news. Nowadays, statistical approaches are well assessed and they
allow for an easy design of text classification (TC) systems. Additionally, the
Web has emphasized the need of approaches for digesting large amount of
textual information and making it more easily accessible, e.g., thorough hier-
archical taxonomies like Dmoz or Yahoo! categories. Surprisingly, automated
approaches have not proved yet to be indispensable for such categorization
processes. This suggests that the role of TC might be different from simply
routing documents to different topical categories.
In this paper, we provide evidence of the promising use of TC as a support for
an interesting and high level human activity in the educational context. The
latter refers to the selection and definition of educational programs tailored
on specific needs of pupils, who sometime require particular attention and
actions to solve their learning problems. TC in this context is exploited to
automatically extract several aspects and properties from learning objects,
i.e., didactic material, in terms of semantic labels. These can be used to
organized the different pieces of material in specific didactic program, which
can address specific deficiencies of pupils. The TC experiments, carried out
with state-of-the-art algorithms and a small set of training data, show that
automatic classifiers can easily derive labels like, didactic context, school
matter, pupil difficulties and educative solution type.
Keywords: hierarchical text classification, information management applications,
e-learning
1 Introduction
The last two decades have seen an impressive development of methods for automated
text categorization (TC) [7]. This has been mainly due to the combination of two
important factors: (i) the exponential development of the Web, requiring for effective
methods of information access and management; and (ii) the enhancement in theory
and practice of machine learning methods, which constitute the bases of TC.
Despite the success of the TC research, it is still not clear if such technology
should be devoted to the design of topical categorization systems as very famous
Web hierarchical categorization systems are currently manually maintained, e.g.,
Dmoz or Yahoo! categories. On the other hand, TC also regards the association of
semantic labels that go beyond the simple routing of information to the most ap-
propriate user feeds. Indeed, this kind of task inevitably suffers from errors in Recall
and/or in Precision. Different would be the approach and results, if the outcome
of the TC system were cooperatively used as a tool to organize the information in
different and creative ways. In this respect, TC would be seen as a tool similarly
to search engines, rather than an end-to-end system forced to demonstrate a very
high accuracy.
In this paper, we report on our experience with the e-Value project, whose
aims are the reorganization or combination of educational materials in different
pedagogical contexts. The Erickson Research Centre has been cataloging a large set
of published educational materials in smaller units, according to the SCORM (2004)
standards, Shareable Content Object Reference Model1 . These documents are used
for the creation of novel and specific didactic product as follows: (i) school classes
are evaluated about target cognitive processes; (ii) processes in which pupils have
difficulties are detected and recorded in a huge database (DB) of normative data
along with the results of its elaboration; (iii) The Decision Support System (DSS)
chooses the proper didactic material for the class according to the DB content.
The above steps require: (a) to identify cognitive processes involved in pupils’
learning; (b) to divide the didactic materials in smaller parts (learning objects); and
(c) classify such objects according to their bibliographic characteristics and to the
cognitive processes involved, which depends on the user context (e.g., age, class,
special situations). An automatic classifier can be used for easing and speeding up
the last step. It can provide a rough classification, which can constitute the starting
point for the work of expert catalogers.
The use of the classifier would reduce the cataloging costs, both in terms of
time and human resources. Indeed, any educational material, being part of a book,
article or best practice, needs to be read and evaluated by experts, before being
assigned to the proper categories; this process takes a huge amount of time. As an
alternative model, the classifier can perform a first approximate categorization and
after, the experts can refine it. The clear advantage is that materials pertaining
to a certain subject can be directly assigned to its experts (working in that field),
thus improving the accuracy of classification and avoiding the burden to exchange
materials among the different experts.
However, the above scenario could be realized only if the adopted multi-class
classifier (MCC) performed accurate hierarchical categorization. Given the novelty
of the intended taxonomy, it is not simple to predict if MCC can deploy the needed
accuracy. For this purpose, we have:
– designed a new taxonomy that meets the organization needs of e-Value;
– defined an annotation procedure and produced an initial datasets of 122 docu-
ments, organized in 112 categories (of course the documents are repeated in the
hierarchy); and
– implemented an MCC, which exploits state-of-the-art TC models such as, Sup-
port Vector Machines, structured in binary flat categorizers.
The preliminary experiments on the overall hierarchy of 112 nodes show promising
results, ranging from a Micro-F1 of above 95% for the first level to about 70% on
the whole hierarchy. This outcome is rather promising and enables future research
in the use of TC for the efficient implementation of educational programs.
In the reminder of this paper, Section 2 describes the tackled task in more detail,
Section 3 reports on our results and Section 4 derives the final conclusions.
2 Automatic Support to the e-Value project
The main objective of the e-Value project is to design, develop and test a multimedia
platform (consisting of a set of web applications), which integrates the evaluation
of various learning abilities and the application of didactic processes. These can
benefit from automatic methods for classifying the didactic material used in such
1
http://www.adlnet.gov/capabilities/scorm/scorm-2004-4th
Root
C1 C2 C3 C4
C11 C12 C13 C21 C22 C23 C24 C31 C32 C41 C42 C43 C44 C45
C121 C122 C123 C124 C231 C232 C233 C311 C312 C321 C322
C2321 C2322 C2323 C2324
Fig. 1. Hierarchical Categorization Scheme of e-Value (only category with at least 1 train-
ing documents are present
processes. The next sections describe the problem in more detail and suggest how
a TC system can be used in such context.
2.1 e-Value Framework
The framework includes different interconnected processes:
– standard evaluation procedures and dynamic assessment of learning abilities of
pupils;
– collection of normative data, e.g., educational material and pupils’ evaluations;
– continuous data flow, i.e., the related database is continuously updated and the
normative data currently available is integrated and compared with the new
arriving data; and
– qualitative and quantitative evaluation of the collected data.
The educational material is used for defining didactic products, which address
specific action (intervention). It consists of books, CD-ROMs, collections of articles,
etc. The e-Value project aims at both using independently and jointly the materials
above.
Designing an intervention often requires the use of units taken from several
books or CD-ROMs but including the entire sources is very ineffective, considering
that only some small parts will be used. To enable more flexibility in the creation
of training programs, the material collections are divided into basic training units,
called learning objects, which can be reassembled in a flexible way. This requires
to analyze the materials to be used in the interventions and selecting the portion
involved in the target cognitive processes.
2.2 A framework use-case
A use of the framework is illustrated by the following example. In a school context
some classes are evaluated with respect to targeted cognitive processes. The tests
may reveal that some of the pupils have difficulties in certain processes. Thus,
the test results are recorded (building a large database of normative data) along
with some elaboration of them, i.e., basic data statistics. Then the DSS chooses
the proper didactic material for the class by proposing different material to pupils
requiring attention and quick intervention. For this purpose the educational team
need to:
– identify every cognitive process that can be involved in learning. At the moment,
this has been restricted to mathematics and reading-writing (with linguistic
skills and metaphonetics);
!"#$%&'())")(&*$+,&*#$-#&+
!"# .(."##(,&+ !$$# %&'()*+*,*-.(# !$/01# 2345&33.*+.#
!""# 678*,(#9&,,:.+)(+;.(# !$$"# <,*=(,&# !$/0># ?*'&+;&#
!"$# 678*,(#45.@(5.(# !$$""# A.@(# !$/0B# A(9.7.#C8(95('&#
!"$"# ?5.@(5.(#!,(33&#D# !$$"$# 6.,,(=(# !$//# !(,7*,*EF8@&5.#5(;.*+(,.#
!"$$# ?5.@(5.(#!,(33&#DD# !$$$# ?5*)*+9(# !$/1# !(,7*,*EF8@&5.#5&,('.G.#
!"$0# ?5.@(5.(#!,(33&#DDD# !$$$"# H*+&@(# !$/># !(,7*,*EA(44*5'.#-*4*5;.*+.#
!"$/# ?5.@(5.(#!,(33&#DI# !$0# J=.,.'K#,.+-8.3'.7L&# !$/B# !(,7*,*E!(,7*,*#,&''&5(,&#
!"$1# ?5.@(5.(#!,(33&#I# !$0"# M&33.7*# !$/N# ?5*=,&@E3*,G.+-#
!"0# 678*,(#3&7*+9(5.(#9.#"#-5(9*# !$0""# O&+*@.+(;.*+&# !$/P# !(4(7.'K#9.#*5.&+'(53.#+&,,*#34(;.*#
!$# !"#$%&'())")(&*$+/"#$'("+ !$0"$# !('&-*5.;;(;.*+&# !$/Q"R# !*3'58.5.3'&@.#9.#5.)&5.@&+'*#7*+G&+;.*+(,.#
!$"# M&''*E375.''85(# !$0"0# D9&+'.).7(;.*+&# !$/Q""# <&*@&'5.(#&87,.9&(#S4.(+(T#
!$""# ?5&5&C8.3.'.# !$0"/# O&).+.;.*+&# !$/Q"$# %.385(#9.#-5(+9&;;&#-&*@&'5.7L&#
!$"""# ?5&5&C8.3.'.#-5()*#@*'*5.# !$0"1# ?*,.3&@.(# !$/Q"0# %.385(#9.#-5(+9&;;&#).3.7L&#
!$""$# ?5&5&C8.3.'.#G.38*#34(;.(,.# !$0"># J55.77L.@&+'*#,&33.7(,&# !$/Q"/# M&#'5(3)*5@(;.*+.#-&*@&'5.7L&#
!$""0# U&*5.&#.+-&+8&# !$0$# %*5)*E3.+'(33.# !$1# J,'5*#
!$"$# O&7*9.).7(# !$0$"# !*+7*59(+;&# !0# !"#$%&'())")(&*$+-(#0")(&*$+"10**(+
!$"$"# M&''&5&# !$0$$# 6'58''85(#9&,,(#)5(3&# !0"# V26#
!$"$$# 6.,,(=&# !$0$0# J+(,.3.#-5(@@('.7(,&# !0""# J8'.3@*#
!$"$0# ?(5*,&# !$0$/# J+(,.3.#,*-.7(# !0"$# W9.'*#
!$"$/# F*+#4(5*,&# !$00# F(55(;.*+&# !0"0# I.3'(#
!$"$1# H5(3.EV5(+*# !$00"# !*@45&+3.*+(77*+'*# !0"/# ?3.7*@*'5.7.'K#
!$"0# !*@45&+3.*+&# !$00$# ?5*98;.*+(77*+'*# !0"1# 6.+95*@ .#O*X+#
!$"0"# ?(5*,&# !$/# %('&@('.7(# !0"># J,'5*#
!$"0$# H5(3.# !$/"# F8@&5*# !0$# O6J#
!$"00# V5(+*# !$/""# ?5*7&33.#3&@(+'.7.# !0$"# D4&5(''.G.'K#
!$"/# !*@4.'(;.*+&# !$/"$# !*+'&--.*# !0$$# O.3,&33.(#
!$"/"# !,&''&5&# !$/"0# ?5*7&33.#45&E3.+'(''.7.# !0$0# O.3-5().(#
!$"/$# 6.,,(=&#+*+#*5'*-5().7L&# !$/"/# ?5*7&33.#,&33.7(,.#.+'(''.7.# !0$/# O.37(,78,.(#
!$"/0# ?(5*,&#+*+#*5'*-5().7L&# !$/$# !(,7*,*#E#45*7&33.#9.#=(3&# !0$1# !*@=.+(;.*+ .#O6J#9.G&53.#E#(,'5*#
!$"//# !F*+#4(5*,&# !$/$"# 6&-+.#9&,,&#*4&5(;.*+.# !0$># F&338+#O6J#
!$"1# Y5'*-5().(# !$/$$# H(''.#+8@&5.7.# !/# !"#$%&'())")(&*$+#(2&+.(+(*#$'3$*#&+
!$"1"# Y4(5*,&# !$/$0# U(=&,,.+&# !/"# ?*'&+;.(@&+'*#
!$"1$# Y)5(3.# !$/$/# !(,7*,*#(#@&+'&# !/$# A&784&5*#
!$"10# Y=5(+*# !$/0# !(,7*,*#E#+8@&5.#+('85(,.# !/0# O.9(''.7(#.+3&-+(@&+'*#
!$"># 6'&385(#'&3'*# !$/0"# J,-*5.'@.#9.#7(,7*,*#375.''*# !//# D+'&5G&+'*#,*-*4&9.7*#
!$">"# ?.(+.).7(;.*+&# !$/0$# D+7*,*++(@&+'*#9.#+8@&5.# !/1# D+'&5G&+'*#43.7*,*-.7*#
!$">$# U5(375.;.*+&# !$/00# %8,'.4,.# .G.3*5.#
%.+.@*#!*@8+&#%8,'.4,*#&## # #
%(33.@*#!*@8+&#
!$">0# A&G.3.*+&# !$/0/# O&+*@.+('*5&#
! # #
Table 1. Description of the different categories of the hierarchy in Figure 1.
!"#"$%&' ()*+,%-.' ("/0%-.' 1)"2+/+.,' 3"2*$$' 4&'
5&' 67' &8' 9:7;<&' &:9999' 9:=&;6'
5<' ;9' <9' 9:=9;7' 9:=>99' 9:=<87'
56' ;&' &=' 9:=>99' &:9999' 9:=?;;'
5;' 6=' &?' &:9999' 9:77<;' 9:=6?>'
' ' ' ' ' '
@+2).' ' ' 9:=<99' 9:=>76' 9:=677'
@*2).' ' ' 9:=<;<' 9:=>>&' 9:=67<'
'
Fig. 2. Performance for the first level
– divide the didactic materials in smaller parts (learning objects). This because
the use of the entire books or CD-Rom would be unfeasible, considering that
just a few exercises need to be applied. Thus the whole material has to be
checked by experts to be subdivided in learning objects. The latter are then
used to design the formative offer, in place of the entire material, obtaining a
more personalized and individualized learning.
– Categorize the materials according to their bibliographic characteristics and,
most importantly for the fruition of the materials, to features of the involved
cognitive processes, e.g., the age, class and special situations of the target pupils
etc.
– Porting the material from paper or optical media to an electronic format (pdf
or swf) so that it can be reassembled online and offline.
In the last phase the application of an automatic classifier can provide significant
benefits to the whole process as explained in the following section.
2.3 Classification Task
To meet the need of the e-Value project, we have defined a new taxonomy as well
as the annotation procedure and initial datasets. Our hierarchical categorization
scheme is shown in Figure 1, whose more descriptive labels are reported in Table
1. The materials have to be classified according to four macro-categories, and then
divided into a structure of sub-categories of 4 levels. Each category is meaningful
for a correct description of the materials, from both administrative perspective
(e.g., in which educational context should be applied) and subject/cognitive process
viewpoint (e.g. Mathematics – Number – Lexical and semantic processes instead of
Mathematics – Basic processes of calculus – Numerical facts). The Macro-categories
are: C1 – School and class (referring to the ages 5 – 14); C2 – Subject/cognitive
process (referring to the subjects of mathematics, linguistics, phonetics, reading-
writing abilities); C3 – Pupils’ situation (for the cases of special needs or particular
situations); and C4 – Type of material (or the normal didactic usage in the class,
or for pupils with special situation or greater difficulties in the subject).
Such automatic classification could improve the manual categorization costs, in
terms of both time and human resource. Each piece of educational material, being
part of a book, article or best practice, needs to be read and evaluated by experts,
before being assigned to the proper categories, and this process takes a huge amount
of time. Therefore, the use of an automatic classifier could significantly reduce the
time required to read and evaluate the materials. Of course, experts will need to read
part of the material in any case to refine and validate the output of the classifier.
However, the materials pertaining to a certain subject can be directly routed to the
experts of such field, thus improving the categorization accuracy.
!"#"$%& '()*+%,- '"./%,- 0("1*.*-+ 2"1)$$ 34
54 67 48 9:7;&4 4 9:<4;6
5& ;9 &9 9:<9;7 9:<= 9:<&87
56 ;4 4< 9:<= 4 9:<>;;
5; 6< 4> 4 9:77&; 9:<6>=
544 = 4 9 9 9
54& 68 4= 9:<666 9:<666 9:<666
546 > 4 9 9 9
5&4 4& = 4 9:7 9:777<
5&& 49 6 9:; 9:888> 9:=
5&6 4 4 9:<;4& 9:<;4& 9:<;4&
5&; &9 44 4 4 4
5&= 9 4 9 9 9
564 & 9
56& 6< 4< 9:<= 4 9:<>;;
5;4 &6 44 4 9:<9<4 9:<=&;
5;& 64 4& 9:7=>4 4 9:<&64
5;6 &= 7 9:777< 4 9:<;4&
5;; 49 8 4 9:888> 9:7
5;= 9 4 9 9 9
? ? ?
@*1(- 9:<48& 9:<48& 9:<48&
@)1(- 9:8;97 9:86> 9:86&=
Fig. 3. Performance for the second level
3 Experiments
The aim of our evaluation is to demonstrate that state-of-the-art TC methods can be
applied to learn hierarchical classifiers for our e-Value taxonomy. This task is made
complex by two different aspects: (i) in addition to topic labels such as, Euclidean
Geometry, Problem Solving or Geometric Transformation, the taxonomy also con-
tains semantic characterization such as Story Development or Story Understanding,
whose characterization using simple terms seems harder; and (ii) given the novelty
of the taxonomy, we could only produce a small dataset, which makes the learning
of classification functions more difficult. To deal with and analyze such problems,
we experimented with hierarchy subsets, defined according to the hierarchy’s levels,
ranging from 1 to 4 (the maximum depth of our hierarchy). The deeper the level,
the more difficult TC is.
3.1 Setup
One major drawback of machine learning and thus of TC based on it is the need
of training data, i.e., a set of documents manually classified into the referring tax-
onomy. This data is difficult to find and/or to produce as it requires human labor.
Given the novelty of our taxonomy defined in Figure 1, no previous data was avail-
able. Thus, we set an annotation procedure (with only one annotator) of the didactic
material available in the Erickson’s database. We randomly selected 60 documents
and we classified each of them according to all the 112 nodes of the taxonomy. This
led to a dataset of 122 documents (repetitions are considered).
We randomly divided the above data in training and test set by taking care that
for each document all its repetitions were all put either in the training or in the test
set. The training data was used to learn the set of 112 binary classifiers, one for each
category, following the one-vs-all schema. The output of the multi-class classifier is
the merged set of the individual binary classifier decisions. Although simple, this is
considered a state-of-the-art approach [5, 3]. We used default SVM parameters as the
small training data prevented to apply any reasonable parameterization approach.
We used a bag-of-term representation (string separated by space and punctuation)
without applying any feature selection, stop list or lemmatization. Although, we are
!"#"$%& '()*+%,- '"./%,- 0("1*.*-+ 2"1)$$ 34 !"#"$%& '()*+ '",- .("/*,*0+ 1"/)$$ 23
5464 6& 47 78999: 789 789&49 45353 5 3 6 6 6
5466 67 ; 7 7 7 45355 7 3 6 6 6
546& 47 ; 7 7 7 45357 7 3 6 6 6
546< 47 = 4 789 78:= 453&5 6 3 6 6 6
546= > & 78&&&& 78&&&& 78&&&& 45383 6 5 6 6 6
5646 = 4 7 7 7 45385 6 3 6 6 6
564< 7 4 7 7 7 45387 6 3 6 6 6
564= 7 6 7 7 7 45393 5 3 6 6 6
5649 6 4 7 7 7 45533 & 3 3 3 3
5664 > 6 78< 4 78=:4< 45535 : 5 6;& 3 6;8<3&
5666 > & 78999: 78999: 78999: 45553 : 7 6;999< 6;999< 6;999<
56&6 6 4 7 7 7 45755 6 3 6 6 6
56<4 4 6 7 7 7 45757 5 3 6 6 6
56<6 46 47 4 78< 78=:4< 4575& 3 3 6 6 6
56<& 4 < 7 7 7 45&33 3 5 6 6 6
5&64 7 & 7 7 7 45&53 8 7 6 6 6
5&66 67 > 78: 78:::; 78:&9; 45&55 & 8 6 6 6
5&6& 4 9 7 7 7 45&57 3 5 6 6 6
5&6< 49 < 4 4 4 45&5& 7 9 3 6;399< 6;5=8<
5&6= = 4 7 7 7 45&73 3 & 6 6 6
45&75 3 3 6 6 6
45&77 6 5 6 6 6
? ? ?
@*1(- 78;=<= 78:6=4 78:;<= >*/(0 6;=&76 6;97:8 6;<5<7
@)1(- 786;;& 7869;> 7869&4 >)/(0 6;37:& 6;35== 6;33&<
(a) third level (b) fourth level
Fig. 4. Performance for the third and fourth level. Categories with no document in the
test set and the categories of upper levels are not reported.
confident that the latter may relevantly improves our models. We used the classical
log(T F ) ∗ IDF weighting scheme and normalized vectors.
The performance is provided by means of Micro- and Macro-Average F1, evalu-
ated from our test data over all 112 categories. Additionally, the F1s of the binary
classifiers are reported. For measuring the performance of different hierarchical lev-
els, only the nodes up to the target level are considered, e.g., for the first level, we
only measure the Micro/Macro F1 of C1, C2, C3 and C4.
3.2 Results and Discussion
Table 2 reports the performance on the first level. We note that for each category
there are about 40 documents for training. These seem to be enough as the accuracy
of the individual categories as well as the overall Micro/Macro F1 is exceptionally
high. This is not completely surprising as most documents are repeated in the above
four categories.
Table 3 illustrates the results for the second level. We note that when the training
documents are more than 20, very good results can be achieved. Low performance is
shown for C11 and C13, which are trained with less than 7 documents. Additionally,
they have only one test document, this means that their accuracy cannot really be
estimated. The situation of C31 is even worse as it has no test documents. In this
case, we do not report any accuracy in the related row. It should also be noted
that, since we use one-vs-all schema, the accuracy of C1,..,C4 is the same as before.
Thus, from now on, we will not report the accuracy of previously reported binary
classifiers.
Table 4 shows the performance on levels 3 and 4. Again the few training doc-
uments available for the classifiers prevent to achieve a reasonable F1. There are
some good cases such as C124 and C322 but also bad cases such as C122 and C123.
The latter two refer to Primaria Classe II and Primaria Classe III, respectively,
which have large overlap with the other classes, i.e., I, IV and V. For separating
such categories, the simple bag-of-words may not be enough.
4 Conclusions
In this paper, we have described an interesting and new semantic classification
problem in the context of the educational framework of the e-Value project. We
have defined a new hierarchical taxonomy, which is promising for improving the
production cycle of educational systems. To test the feasibility of the approach, we
have also built a corpus annotated according to the above taxonomy. Such data
was used for training an MCC based on SVMs. The results show that when there
is a reasonable amount of training documents the classifiers can deploy remarkably
high accuracy. On the other hand, the F1 of lower level categories is highly affected
by data scarceness. Some categories would probably require the definition of more
expressive features to better model their separation.
Possible solutions are also provided by previous work, which shows more ad-
vanced TC models, e.g., [6], in which global dependencies between hierarchical nodes
are encoded in a gradient descendent learning approach. They experimented with
Reuters Volume 1 (RCV1) 2 on a subhierarchy only containing 34 nodes. Other rel-
evant work such as [4] and [1] uses a rather different datasets and a different idea of
dependencies based on the feature distributions over the linked categories. Finally,
[3] experiment with models similar to ours achieving state-of-the-art on RCV1.
Acknowledgements
The research described in this paper has been partially supported by the Italian
Project e-Value (PAT) and by the European Community’s Seventh Framework
Programme (FP7/2007-2013) under the grants #231126: LivingKnowledge –
Facts, Opinions and Bias in Time, #247758: EternalS – Trustworthy Eternal
Systems via Evolving Software, Data and Knowledge, and #288024: LiMoSINe –
Linguistically Motivated Semantic aggregation engiNes.
References
1. Dumais, S.T., Chen, H.: Hierarchical classification of web content. In: Belkin, N.J., In-
gwersen, P., Leong, M.K. (eds.) Proceedings of SIGIR-00, 23rd ACM International
Conference on Research and Development in Information Retrieval. pp. 256–263.
ACM Press, New York, US, Athens, GR (2000), http://research.microsoft.com/
~sdumais/sigir00.pdf
2. Hayes, P.J., Weinstein, S.P.: Construe/Tis: a system for content-based indexing of a
database of news stories. In: Rappaport, A., Smith, R. (eds.) Proceedings of IAAI-90,
2nd Conference on Innovative Applications of Artificial Intelligence. pp. 49–66. AAAI
Press, Menlo Park, US (1990)
3. Lewis, D.D., Yang, Y., Rose, T., Li, F.: Rcv1: A new benchmark collection for text
categorization research. The Journal of Machine Learning Research (5), 361–397 (2004)
4. McCallum, A., Rosenfeld, R., Mitchell, T.M., Ng, A.Y.: Improving text classification
by shrinkage in a hierarchy of classes. In: ICML. pp. 359–367 (1998)
5. Rifkin, R., Klautau, A.: In defense of one-vs-all classification. J. Mach. Learn. Res. 5,
101–141 (December 2004), http://dl.acm.org/citation.cfm?id=1005332.1005336
6. Rousu, J., Saunders, C., Szedmak, S., Shawe-Taylor, J.: Kernel-based learning of hi-
erarchical multilabel classification models. The Journal of Machine Learning Research
(7), 1601–1626 (2006)
7. Sebastiani, F.: Machine learning in automated text categorization. ACM Computing
Surveys 34(1), 1–47 (2002)
2
trec.nist.gov/data/reuters/reuters.html