English. We present a project aimed at studying the evolution of students' writing skills in a temporal span of 15 years (from 2001 to 2016), analysing in particular the impact of neo-standard Italian. More than 2,500 essays have been transcribed and annotated by teachers according to 28 different linguistic traits. We present here the annotation process together with the first data analysis supported by NLP tools.
In this work, we present an extensive study on the
evolution of high-school students’ writing skills,
taking into account essays spanning 15 years
(from 2001 to 2016). In particular, we are
interested in tracking the presence of expressions
and constructions typical of neo-standard Italian
1See the open letter signed by around 600 University professors at http:// gruppodifirenze.blogspot.it/2017/02/ contro-il-declino-dellitaliano-scuola. html.
The Italian neo-standard is the current linguistic register in Italy, in which forms previously considered colloquial have become widely accepted in the national language.
We analyse more than 2,500 essays written by students from different high-schools in the Autonomous Province of Trento during the exit exam (the so-called Maturita`). The study is the outcome of a project comprising different steps: i) digital acquisition and transcription of thousands of essays balancing their distribution across school years and school types; ii) computer-assisted annotation of some linguistic traits of interest; iii) diachronic analysis of the traits. While the first step has been carried out by the Istituto provinciale per la Ricerca e la Sperimentazione educativa (IPRASE), we led steps ii) and iii), which are discussed in the next sections. Beside an in-depth and diachronic study of the evolution of students’ writing skills, a major contribution of this paper is also the release of the corpus in the form of embeddings and n-grams. 2
The staff of IPRASE have digitized and transcribed essays stored in the archives of 21 secondary schools located in different areas of Trentino Province. These areas include both the two major cities, Trento and Rovereto, but also other communities in the valleys (Val di Fiemme, Val di Non, Valsugana) and Riva del Garda. Nine different types of schools were involved: liceo classico, liceo scientifico, liceo artistico, liceo linguistico, liceo musicale e coreutico, liceo delle scienze umane, istituto tecnico tecnologico, istituto tecnico economico and istituto professionale. Six school years were chosen between 2000-2001 and 2015-2016, thus having a temporal span of 15 years for a total of 2,544 essays and almost 1.5 million words. Table 2 shows the distribution of essays per year with the corresponding number of words. These essays are of the so-called type B, that requires students to write a short essay or a newspaper article. Students can choose between 4 areas: artistic-literary, socio-economic, technicalscientific, historical-political. For each area, a title is given together with a set of reference materials. For example, students writing an essay of type B with historical-political content in 2014 were asked to comment some excerpts from Hannah Arendt, Ghandi and Martin Luther King about violence and non-violence in the XX Century.
Due to privacy reasons, we are not allowed to
distribute the full texts of the corpus. However,
we release both word vectors and n-grams of the
essays. We build three types of embeddings with
300 dimensions: the GloVe embeddings based
on linear bag-of-words contexts
Around 20 teachers have been involved in the
annotation of essays using the CAT platform
We briefly present in Table 2 the traits that the teachers had to mark on each essay. The goal of the annotation is to detect the presence of linguistic traits that were deemed relevant to diachronically study style and complexity evolution by IPRASE experts and teachers. This approach is therefore rather different from the standard essay correction that is usually performed by teachers, and for this reason the training phase was particularly relevant.
The list of traits to include in the project was
mainly inspired by the work of
In the first Table column, we mark traits that
were identified in a fully automatic way (A), those
that were annotated semi-automatically (S), and
the manual ones (M). For those marked with S,
we pre-processed the essays using the Tint NLP
tool
After the initial training phase, the average annotation time for each essay through the web interface was 30 minutes. We roughly estimate that the same task would take at least one hour on a standard Word document. Another advantage of using 4A complete version of the annotation guidelines (in Italian) is available at this link: http://bit.do/erd9P 5http://derivatario.sns.it/ the CAT interface was the possibility to have all annotations in a consistent format, easily export them to compute statistics and make comparisons.
We present here an analysis of some traits of
interest. We focus in particular on traits that are, at
least in part, automatically annotated and counted
(marked with A or S in Table 2), because the work
of those requiring a manual annotation is still in
progress. For each trait we compute the observed
relative frequency per 10,000 words. This
normalization has allowed us to have more easily
comparable and legible numbers. Furthermore, we
calculate the Gulpease index to monitor writing
complexity
To extract reliable measures of students’ language use, we removed from the texts the quotations present in the essays citing the reference material provided together with the topic. This preprocessing step was performed by adopting the FuzzyWuzzy package6, a Java fuzzy string matching implementation, and the Stanford CoreNLP quote annotator7. These tools allow us to recognize text reuse both when it is explicitly signaled by quotes and when there is no overt signal. The average percentage of quotations within the corpus is 1.9% but it varies a lot among the essays, reaching up to 46% of the content in some cases. The following is an example taken from an essay about the pursuit of happiness in 2010 for the socio-economic area. The snippet in bold, containing one of the complex connectives of trait 16, was automatically removed: La riflessione di Zygmunt Bauman sembra essere una risposta: “L’incertezza e` l’habitat naturale della vita umana, sebbene la speranza di sfuggire ad essa sia il motore delle attivita` umane.”
After removing quotations, we obtain the following results for the automatically annotated traits:
presence of semantically generic and polysemic
words, which are frequently used in neo-standard
Italian (Fig. 1). In particular, lemmas ‘fare’,
‘dire’, and ‘cosa’ (to make, to say, thing) show
a decrease in occurrence in the last two school
6https://github.com/xdrop/fuzzywuzzy
7https://stanfordnlp.github.io/
CoreNLP/quote.html
years considered
out a verbal predicate are a typical feature of news
style and juvenile writing, to make the text
dramatic and concise
Trait 16 - Complex Connectives. The lack of complex connectives is another indicator of neostandard Italian. As shown in Figure 2, ‘nondimeno’ is never used by students and also ‘qualora’ and ‘giacche´’, used mostly in liceo classico, disappear in the last two school years from all the essays. ‘Affinche´’ is adopted in all school types with the only exception of liceo artistico, in which complex connectives are barely used.
Trait 17 - Punctuation. Over the last two school
years considered in our analysis, there has been an
overall decline in the use of punctuation with the
exception of question marks (see Figure 3). The
frequent use of question marks is inherited from
the style of news
grammatical rule8, the euphonic ‘d’ should be
introduced only when the conjunction ‘e’ or the
preposition ‘a’ are followed by a word starting
with the same vowel: e.g., ed ecco, ad andare.
However, this rule is not followed in the essays and
the presence of ‘d’ between two different vowels
is higher than the one between the same vowels
(33.8 versus 17.6 of relative frequency). Besides,
while the disappearance of this trait is considered
a characteristic of neo-standard Italian
Gulpease. We computed the Gulpease index to see whether there has been a decrease of complexity, i.e. an increase in readability, over time. Contrary to our expectations, the average readability of essays has slightly decreased in the last two years considered, with a drop of 1.8 points, bringing it below 50. This corresponds to texts that are quite difficult to read for a person with a medium school degree (diploma di scuola media in the Italian school system), but not too challenging for a person with a high school degree. Moreover, values do not change much across different school types.
These preliminary analyses show that the impact of neo-standard Italian is multi-faceted and, while some traits confirm that students’ language is getting simpler and less formal (e.g. overall decline of punctuation), some others seem to contradict this finding (e.g. decline in the use of ‘fare’, ‘dire’, ‘cosa’). Also the differences across school types are not clear-cut and consistent. 5
While several works in the past have focused on
the creation and analysis of corpora to study
students’ mistakes, their writing quality and their rate
of progress over the year
German, the KoKo corpus of argumentative essays
to study pupils’ writing competences
As for Italian, a relatively small number of
studies has been carried out with various goals.
The projects TIscrivo (2011-2014) and TIscrivo
2.0 (2014-2017)9 have been launched to
investigate the writing skills of primary schools and
lower secondary schools in Southern Switzerland
Compared to previous works, our analysis is different in several ways. First, none of the previous studies considers a text span of 15 years. Then, the traits to be annotated are different: we do not focus on mistakes, but on indicators of neostandard Italian. Finally, our interest lies also in the annotation workflow, studying how NLP can support the identification of such traits and implementing the necessary processing modules to speed up annotation.
9http://dfa-blog.supsi.ch/tiscrivo/ 6
In this work, we have presented a project aimed at tracking the evolution of students’ writing skills over time. The goal of this work was not only to introduce the corpus collection and annotation activities, but also to show how this kind of projects can benefit from NLP by speeding up annotation and increasing data consistency. In the future we will complete the analysis of all the traits for a more comprehensive view of the role of neostandard Italian in students’ essays. We will also use some of the manual annotations to train new NLP modules performing the same task automatically.
We would like to thank Chiara Motter from IPRASE for coordinating the corpus transcription, and the high-school teachers for annotating the essays.