This paper describes the results of a work, carried out by RAI within the framework of the project \La Citta Educante", aiming at creating statistical models for automatic document categorization and named entity recognition, both acting in the educational eld and in Italian language. The taxonomy used for the documents categorization is the Scienti c Disciplinary Sector taxonomy (SSD) used in Italy to organize the disciplines and thematic areas of higher education. The actual statistical models were created with the Apache OpenNLP libraries. The obtained results showed fairly good accuracy with SSD document classication and some signi cant improvement of categorization precision for Named Entities compared to the state of the art.
With the increase of the number of multimedia contents that are produced
everyday by a wide range of applications and devices, users can now access huge
amount of data in wide-world localized archives and repositories. As an
example, in the educational domain, teachers can now circulate their lectures through
online video repositories, like e.g. VideoLectures.NET.1 E ective and
comprehensive annotations are needed in order to quickly and easily access those
contents. In the literature many approaches can be found that aim at addressing
the problem of video analysis and retrieval. Thought, almost all of the proposed
research targets speci c and restricted domains such as sports and news
broadcasting [
\La Citta Educante" project2 aims to develop new models of learning and
teaching by exploring, developing and evaluating innovative technologies for
knowledge extraction, management and sharing. These include content based
video analysis, segmentation and summarization [
This section describes the theoretical background on which automatic video annotation was performed. The assumption is that video annotation can be shifted to a language processing problem by analyzing the text automatically derived by the speech content of broadcast material. Two problems are approached, i.e. document categorization and named entity recognition from spoken documents. To address these problems, the Apache OpenNLP library3 was used. 2.1
RAI has a multi-year experience in automatic metadata extraction in the news
domain and developed technologies used internally also for automatic news
categorization. In this context a proprietary classi cation schema already adopted
in manual documentation is used [
De nition of the Categorization System. In order to de ne the most
appropriate classi cation schema various options were considered, including the
names of the subjects of study in various degrees of education. As an
alternative, the organization of the contents of Skuola.net,4 a portal that keeps and
shares teaching materials, notes and news addressed to both middle/high school
and university students, was considered. Starting from this analysis, it can be
observed that topics are more and more speci c as far as the level of education
increases. However grouping in areas does not follow a coherent set of criteria, as
it may be appropriate to re ect the goals of the di erent institutes. In the case
of Skuola.net, topics are mapped to the speci c faculties and examinations that
meet the greatest interest of the portal users. This latter approach is certainly
interesting, as it represents a categorization that takes into account the point
of view of the nal user. Though, in the absence of a speci c model, it would
be preferable to adopt a set of neutral categories with respect to the goals of
particular communication operators. Thus, we decided to adopt the Scienti c
Disciplinary Sectors (SSD) classi cation system,5 which is used in Italy for the
organization of higher education as the reference for the subject classi cation.
The SSD, de ned by the Italian Ministry of Education, University and Research
(MIUR), is organized hierarchically on four levels. There are 14 areas at the top
level, 88 macro-sectors at the second level, 188 competition sectors at the third,
and 367 scienti c-disciplinary sectors at the last level. Hierarchical classi cation
allows the attribution of one (or more) category(s) to an appropriate degree of
speci city/generality. We limited the work to the rst two levels of the
hierarchy, as the number of remaining classes is comparable to what already done for
journalism in RAI [
edge sharing through interoperability and data exchange. For this purpose, the World Wide Web Consortium (W3C) de ned a number of formal languages that allow the development of systems for the integration of heterogeneous data sources, such as those from multimedia archives, social networks and open data. 4 http://www.skuola.net/ (last accessed September 2017) 5 http://attiministeriali.miur.it/anno-2015/ottobre/dm-30102015.aspx (last accessed September 2017) Among them, the Simple Knowledge Organizational System (SKOS)6 represents the state of the art for describing structured vocabularies, such as glossaries, classi cation systems, and taxonomies. SKOS was designed with the idea of being easily extensible in order to allow connection and sharing of knowledge between di erent databases. The main SKOS elements are: { Concepts, which are elementary information units, representing ideas, meanings, objects, or events at the basis of a system of knowledge organization. Concepts can be grouped into collections and/or organized according to a particular schema (ConceptScheme); { Labels and Notations, which are a set of words or phrases, even in di erent languages, that describe a concept; { Relationships, such as hierarchies and equivalences, between concepts within the same dictionary (Semantic Relations) and between concepts de ned in di erent dictionaries (Mapping Relations).
Some examples of taxonomies implemented in SKOS include the multilingual thesaurus of the European Union (Eurovoc),7 and the thesaurus of the Wikipedia categories. As part of \La Citta Educante", the Scienti c Disciplinary Sectors (SSD) taxonomy has been implemented in SKOS through an iterative process de ned by the following steps: { De nition of concepts. A SKOS concept was created for each SSD area and macro-sector and identi ed by the corresponding alphanumeric code de ned by the ministerial ordinance. For example the code 12/F indicates the macro-sector F in area 12 of the SSD; { De nition of the main scheme. In order to group all the concepts of the ontology, a top level SKOS ConceptScheme was de ned to which all the SSD areas are related by the hasTopConcept/topConceptOf property of the SKOS standard; { Concept description. Each concept was described with a label (prefLabel) based on the name (Italian and English) of the corresponding denomination de ned by the ministerial decree. For example, the 'Diritto processuale civile' and 'Civil procedural law ' labels were assigned to the macro-sector 12/F, respectively for the Italian and English languages; { De nition of intra-schema hierarchy. Hierarchical relationships between areas and macro-sectors were de ned using the broader and narrower properties of the SKOS standard; { De nition of inter-schema relationships. In order to promote interoperability with other classi cation schemes, each SSD concept was mapped with at least one concept of the Eurovoc thesaurus and of DBpedia categories. Eurovoc was selected as it allows a unique classi cation of documents in the European Union's institutional databases irrespective of the language used in the documents themselves. DBpedia was chosen as the central hub 6 https://www.w3.org/2004/02/skos/ (last accessed September 2017) 7 http://eurovoc.europa.eu/ (last accessed September 2017) of the Linked Open Data (LOD) network. Furthermore, in order to ensure interoperability with the RAI archives, each SSD concept was mapped to the classi cation schema used for documentation of the RAI's archives. Inter-schema relationships were de ned according to the following criteria: { The SKOS hasExactMatch relates concepts identi ed by the same label to the considered ontologies. For example, the SSD macro-sector ('Civil Procedure Law ') was mapped exactly to the 'Civil procedure' DBpedia category; { The SKOS hasNarrowMatch property (hasBroadMatch) relates more indepth (generic) concepts to the considered ontologies. For example, the SSD macro-sector 'International Law, European Union, Comparison, Economy, Markets and Navigation' hasNarrowMatch 'Business law ', 'Comparative law ' , 'European Union law ', and 'International law ' by DBpedia; { The SKOS hasRelatedMatch property relates concepts that share one or more features but whose mapping do not fall into any of the previous cases. For example, the SSD macro-sector 'Astronomy, Astrophysics, Physics of Earth and Planets' hasRelatedMatch 'Planetary science' by DBpedia. An excerpt from the ontology is shown in Fig.1. 2.2
For the creation of the named entity recognition (NER) models we adopted a
semi-supervised approach, a technique commonly used in machine learning when,
given a large dataset, only a subset has annotations. The complete manual
annotation of the entire dataset is a long and expensive process not exempt from
human mistakes. E.g., Xue et al [
This section describes the experiments that we undertook to demonstrate the
e ectiveness of the presented methods. The reference dataset includes 2,020
episodes of TGR Leonardo, a science newscast produced by RAI and
broadcasted daily from Monday to Friday on the RAI3 channel. Each episode, lasting
approximately 10 minutes, features news about technology, health, economy,
environment and society. The format is that of a traditional newscast. Each episode
was automatically segmented and transcribed into elementary news stories using
the RAI ANTS system [
This section describes the guidelines adopted at the annotation stage as well as
the composition and organization of the reference dataset. Assigning SSD areas
and macro-sectors requires a good understanding of the hierarchy of categories
and a careful listening and evaluation of content. The hierarchy of categories
needs to be well-known as some words can have wider or narrower meaning
with respect of common usage. For example the term pedagogy does not simply
indicate the educational aspect of children/adolescents (as often intended) but
of people of all ages. To create the ground-truth categories we assigned one or
two as a maximum areas/macro-sectors to each news item, according to their
importance with respect to the news item topic. For example, if a news item talks
about the use of information technology in secondary schools, the news item
would have been annotated by 'History, philosophy, pedagogy and psychology '
and 'Mathematics and informatics ' as, respectively, primary and secondary SSD
area. All detected news items were manually annotated by a group of 10 people.
The total number of annotated news is 6,608, corresponding to approximately
243 hours of audio-visual material. The size of this dataset is in line with those
adopted in similar works [
A 10-fold cross validation was used to assess the performance of the categorization task. Fig.4 shows the results on the average precision, recall, and F-measure for the classi cation according the SSD areas. The categorization model achieves good performance in terms of precision.
Analyzing average recall values, a more varied situation is noted. In particular, for the areas most represented in the reference dataset, a good balance between precision and recall is obtained. Conversely, for the areas that are least represented in the dataset, low recall values are obtained. This may depend on the use of a not enough large learning dataset (e.g. insu cient annotations for a given area), or on the interchangeability of the area with other areas with similar topics. This phenomenon is con rmed by the analysis of the confusion matrix shown in Fig.5. The matrix rows show the expected areas. The matrix columns count the deduced areas (i.e. those generated automatically by the categorizer). The element in position (i,j) indicates the percentage of news items belonging to area i that have been categorized with the area j. The matrix diagonal shows the percentage of news items ranked correctly for each area, while the extra-diagonal elements contain misclassi cation errors. It can be noted that the matrix has the maximum at the diagonal for most of the considered areas (highlighted in green). Areas subject to many categorization errors are highlighted in red. For these areas, errors are mostly distributed along the matrix columns, con rming good precision with respect to lower recall. Some areas tend to be confused more than others (e.g., Medical Sciences with Biological Sciences and vice versa), in accordance with intrinsic ambiguity (subject matter) or derivative (annotation errors) of the same topics.
Similarly, we evaluated the performance of the categorization of the SSD macro-sectors. Table 1 reports the top-5 classi cation scores. As expected, global performance is worse than performance of the SSD area categorizer. This might be attributable to di erent causes, including the small size of training examples for some macro-sectors, the higher number of categories (i.e. macro-sectors) to be recognized, and the higher degree of interchangeability among them. 3.3
NER process was evaluated in terms of precision for each of the entity categories considered. Recall was omitted from the analysis because of the huge amount of time required to assess it (since it requires having a dataset fully annotated). This omission is considered acceptable because in our application scenario the Recall is less signi cant. When doing searches on large amount of data in fact, users usually obtain a lot of matching documents and desire that they are not false positives in order not to waste their time, this means that high precision is desirable. For sure low recall excludes from the search potentially interesting documents but this is less important for not really speci c searches made on big indexes. The dataset of 6,600 TG Leonardo news items was randomly split into four sets of equal size, each of them assigned to one annotator for evaluation. Precision was assessed manually, counting the occurrences of the following cases: { Correct identi cation and classi cation. Entity correctly identi ed in the text and attributed to the right category. { Correct identi cation but incorrect classi cation. Entity correctly identi ed in the text but attributed to a wrong category.
{ Wrong identi cation. Entity mistakenly identi ed.
The results obtained are shown in Table 2. For calculation purpose, the Named
Entities found were considered as statistically independent although present with
multiple occurrences in the same document, e.g. the word "Italia" found twice
as a location in the same text counted as two correct occurrences and not just
one. This method is justi ed by the fact that identi cation and classi cation are
carried out on the basis of context analysis limited to the surrounding words and
not with simple match on a vocabulary. As a rst consideration of data
analysis, it is noted that the obtained precision values are aligned between the four
datasets, con rming a substantial validity and concordance of assessments over
the four annotators. The categorization precision values are over 70% (i.e. more
than double a random grading process characterized by a uniform probability
distribution) for each of the test datasets. These are therefore good results, even
more signi cant when compared with the state of the art [
This paper described the results of a work carried out by RAI, aimed to create statistical models for automatic document categorization and named entity recognition, both acting in the educational eld and in Italian language. The taxonomy used for the documents categorization is the Scienti c Disciplinary Sector
LOC PER ORG LOC PER ORG LOC PER ORG LOC PER ORG taxonomy that, as part of the work, has been coded using SKOS to get a formal XML/OWL ontology. The reference ground-truth for document categorization has been created by annotating a dataset of automatic speech transcriptions derived from RAI's scienti c newscasts. Named entity recognition models were generated to be suitable for automatically transcribed text without punctuation and capital letters. Document categorization performance was calculated in terms of precision, recall and F-measure. Named entity recognition was evaluated in terms of precision of both entity detection and entity classi cation, distinctly for locations, people and organizations. The obtained results showed fairly good accuracy with SSD document classi cation and some signi cant improvement of categorization precision for Named Entities compared to the state of the art. Next step for RAI is to test the trained OpanNLP models both in the context of the \La Citta Educante" and for an internal archive search and retrieval experimental service. As an example, locations automatically extracted within speech transcriptions could be used to enrich archive metadata and provide auxiliary information for applications such as georeferencing, data mashups and map visualization.
Acknowledgments. This work was carried out within the project \La Citta Educante" (CTN01 00034 393801) of the National Technological Cluster on Smart Communities cofunded by the Italian Ministry of Education, University and Research - MIUR.