English. This paper reports first results of a wider study devoted to exploit the potentialities of a NLP-based approach to the analysis of a corpus of reflective writings on teaching activities. We investigate how a wide set of linguistic features allows reconstructing the linguistic profile of the texts written by the Italian teachers and predicting whether are reflective.
Since 2014, the “National Institute for Documentation, Innovation and Educational Research” (INDIRE) manages for the Ministry of Education (MIUR) the induction program of the Italian Newly Qualified Teachers (NQTs), i.e. the induction phase of teachers professional development that aims to support teachers in their transition from their initial teacher education into working life in schools. Experimented for the first time in 2014, it became effective starting in 2015 with the DM 850/2015.1 The program involves all new hiring teachers from primary to secondary school for a total of 130,000 NQTs committed in the last 3 years. The underlying theoretical framework developed by INDIRE, MIUR and University of Macerata is based on the alternation of laboratorial and traditional classroom activities with documentation and reflection activities. The purpose is “to influence practices through a process that alternates between moments of immersion and distancing, which are actualised in When I teach and When I reconsider my teaching to think of what happened” (Magnoler et al., 2016). An on-line environment developed and managed by INDIRE2 was set up to support teachers to reflect about and document their educational and professional activities (see Figure 1) during the induction program. All evidences of the instructional tasks (surveys, writing tasks, lesson plans, instructional materials, etc.) are collected in the e-portfolio and printed by the teachers for the final exam. An yearly monitoring of teachers activities is carried on by INDIRE to assess the effectiveness of the whole induction program, as well as of the single instructional tasks. It is aimed to modify, whenever needed, the program in order to improve stakeholders’ scaffolding to the newly qualified teachers and lastly teachers’ professional development.
In this paper, we report first results of an
ongoing study devoted to investigate the
potentialities offered by Natural Language Processing
methods and tools for the analysis of the NQTs
eportfolio. We consider in particular the documents
written by the 26,526 teachers hired in the 2016/17
school year. Many protocols (or models) have
been proposed to assess reflection in teachers
writing, e.g.
Our contribution: i) we collect a corpus of reflective writings manually annotated by experts in the learning science domain and classified with respect to different types of reflectivity; ii) we detect a wide set of linguistically phenomena, characterizing the collected writings; iii) we report the first results of an automatic classification experiment to assess which features contribute more in the automatic prediction of reflexivity. 2
Within the teaching and teacher education domain, a very large amount of studies have been dedicated to conceptualization and analysis of teachers reflection and teachers’ reflective practice. Dewey (1933), Van Manen (1977), Schon (1984; Schon (1987; Schon (1991), Mezirow (1990) are among the main references. The attention on reflective thinking in the teachers education field has increased starting from the 80s as a reaction to the overlay technical view of teaching. Scholars have intensely studied reflection as a concept, detected more levels and types of reflection, how it works during and after professional teachers’ practice, its role and purpose in teachers’ professional development, and how it can be embedded in the curriculum of teachers preparation or professional development, and which techniques may be used to promote it (groups of discussion, readings, oral interview, action research projects, writing tasks, etc). In his seminal work “How we think”, Dewey provides the most shared definition of reflective thinking as applied in the educational field: reflection may be seen as an “active, persistent, and careful consideration of any belief or supposed form of knowledge in the light of the grounds that support it and the further conclusions to which tends”. Hence, reflection is a systematic process of thinking that happens only if related to actual experiences, and includes observation of conditions and references to different pieces of knowledge, (i.e. references to previous experiences, domain knowledge, common sense knowledge, etc.), in order to respond to a dilemma (Mezirow, 1990). Teachers’ educators have extensively employed writing tasks, such as writing structured or unstructured journals, portfolios, essays, blogs, open-ended questions to foster reflection both in pre-service and experienced teachers. Operational definitions of reflectivity proposed to develop schemes for assessing it are focused on identifying the presence of “reflective content” in teachers’ writing, or how deep the reflection is.
Based on these premises, we are currently developing a reflection assessment schema suitable to describe properly the peculiarities of the Italian teachers’ reflective writings written in the framework of the 2016/17 induction program. The schema designed so far, reported in Table 1, was devised according to the following criteria: a writing is reflective if it i) makes direct references to experienced teaching activity, ii) involves several topics (content/pedagogical knowledge) and references to previous experiences, classroom management, learners needs, iii) includes premises analysis (theoretical, context-related, personal) iv) debates a problem (a dilemma), a doubt, v) has an output: it sums up what was learned, sketches future plans, gives a new insight and understanding for immediate or future actions. 3
The corpus of NQTs reflective writings is part of the wider collection of documents written by the 26,526 teachers engaged in the 2016/17 INDIRE induction program. The whole corpus includes all texts written in two of the seven activities of the e-portfolio: Didactic Activity 1 and 2 (DA) for a total of 265,200 texts. During these two activities, teachers were supported by guiding questions designed by INDIRE experts to help them to understand the consistency of the planned and acted I contenuti presentati sono stati acquisiti e gli alunni intervistati si sono dimostrati soddisfatti dell’intervento e del parere personale che hanno potuto esprimere sull’argomento di discussione.
Per rispondere alla domanda circa la possibilita` di migliorare l’attivita` affrontata, diro` innanzitutto che ritengo sempre possibile migliorare le proprie prestazioni. Sono convinta che l’esperienza sia una grande alleata e che, col tempo, si cresca, ci si arricchisca e si migliori.
Credo che la scelta piu` efficace sia stata quella della valutazione tra pari. In particolare, durante la fase della premiazione del concorso di poesia, un alunno per classe si e` recato nell’altra scuola e ha tenuto un discorso introduttivo alla premiazione, nonche´ gestito la stessa in autonomia. Questo, a mio avviso, ha fatto sentire gli studenti i veri protagonisti del loro lavoro e ha favorito la motivazione, intrinseca ed estrinseca. Le consegne sono sempre state fornite in modo chiaro, ma hanno necessitato diverse ripetizioni per essere assimilate.
In realta`, mi sono accorta che solo pochi di loro erano capaci di dare una spiegazione adeguata (anche dal punto di vista formale) e soprattutto non riuscivano a trovare esempi calzanti se non con l’aiuto del libro di testo. Questo momento di ricognizione ha portato via quasi il doppio del tempo che avevo previsto, ma e` comunque stato molto utile per accelerare il loro compito di ricerca durante l’analisi del nuovo testo proposto. Li ho stimolati a chiarire ogni dubbio e grazie anche alle loro domande credo che gli argomenti siano stati davvero appresi da tutti gli studenti, anche da chi di solito ha piu` difficolta` o da chi normalmente partecipa meno. E` stata una lezione che li ha molto coinvolti nonostante si trattasse di una lezione piuttosto “tradizionale”, perche´ mi hanno detto che questo sarebbe servito loro anche per lo studio di altre materie e soprattutto in vista dell’esame. ence domain according to the reflectivity annota5 short texts as answers to 5 different groups of tion scheme described in Section 2 (see Table 2). questions. The first 4 groups provide guidance for The agreement between the three annotators was teachers to write general reflections only on the calculated using the Fleiss’ kappa test and we obdesign of their teaching activity; the fifth group is meant to guide NQTs towards an overall reflection on their whole teaching experience, i.e. both the design and the real teaching activity, also including classroom assessment techniques.
We focused here on the answers to this latter group of questions that were devised in order to encourage teachers to reflect on the following issues: i) differences and similarities between the designed and achieved activities, ii) the most effective choices adopted, also including classroom assessment techniques, iii) how the activity could be improved, iv) the role played by the tutained a k=0.66, i.e. substantial agreement.
No reflection Rhetoric Reflection Radical reflection TOTAL n. answers 185 35 217 36 473 n. sent.
348 91 609 149 1,197 39,936 tated for different types of reflectivity. 4
tor and documentation practices.
We considered
The annotated corpus was tagged by the part-ofin particular a subset of this group of answers that speech tagger described in Dell’Orletta (2009) and were annotated by 3 experts in the learning scidependency-parsed by the DeSR parser (Attardi et al., 2009). This allowed to extract a wide set of multilevel features, i.e. raw text, lexical, morpho-syntactic and syntactic, fully described by Dell’Orletta et al. (2013). They was used to reconstruct the linguistic profile of reflective writings and to carry out a first classification experiment aimed at predicting whether a text is reflective. 4.1
Table 3 shows a selection of the features that vary significantly i) between reflective and nonreflective answers (column Reflectivity) and ii) among the different types of reflectivity we considered (column Types of Reflectivity)3. The analysis of variance was computed in the first case using the Wilcoxon Rank-sum test for paired samples, while in the second case we used the KruskalWallis test since we aimed to assess the different distribution of features in the 4 classes.
In both cases, features from all levels of analysis
resulted to be significant. If we consider the first
ten most discriminative features, reflective
writings resulted to be longer in terms of number of
words and sentences, they are characterized by
longer sentences and by a lower Type/Token
Ratio; they contain an higher number of verbal heads
and of embedded complement ‘chains’ (governed
by a nominal head). Interestingly, they mostly
contain linguistic phenomena typically related to
syntactic complexity, for example they are
characterized by i) an higher use of verbal
modification (e.g. higher % of adverbs, of auxiliary and
modal verbs), ii) more complex verbal predicate
structures (e.g. higher average verbal arity,
calculated as the number of instantiated dependency
links sharing the same verbal head), iii) more
extensive use of subordination (e.g. higher % of
subordinate clauses also embedded in deep chains),
iv) features related to a non canonical word
order (e.g. higher % of pre-verbal objects and
postverbal subjects), v) longer dependency links and
higher parse trees, two features related to sentence
length. On the contrary, non reflective NQTs’
answers contain an higher level of lexical
complexity: they have an higher Type/Token Ratio, a lower
percentage of “Fundamental words”, i.e. very
frequent words according to the classifica
3The full list of ranked features is contained in Appendix.
If we focus on the linguistic profile of the different types of reflective writings, we can observe that answers annotated as Reflection and Radical reflection are mostly characterized by features typically related to structural complexity. This is particular the case of Radical reflection answers that are longer in terms of number of sentences and words; they have more complex verbal predicates (e.g. an higher % of adverbs and of an implicit mood such as gerundive that can be more ambiguous with respect to the referential subject), more complex use of subordination (e.g. average length of ‘chains’ of embedded subordinate clauses), long distance constructions (length of dependency links), non canonical constructions (post-verbal subject). The higher % of demonstrative pronouns and determiners can be related to one of the most representative characteristic of reflection, i.e. the direct reference to real life. On the contrary, they contain a simpler use of lexicon, e.g. a lower Type/Token ratio and an higher percentage of “Fundamental words”. 4.2
Table 4 reports the results of the automatic
classification experiment we devised in order to predict
whether a text is reflective. We built a classifier
based on LIBLINEAR
A very competitive baseline was computed: it exploits the distribution of unigrams of words (Unigrams). As it can be seen, the model that uses all the considered features resulted to be the best Avg sentence length Avg number of sentences Avg number of words Type/token ratio (100 token) % of “Fundamental words” of BIV % of “High usage words” of BIV % of “High availability words” of BIV % of adjectives % of possessive adjectives % of adverbs % of prepositions % of demonstrative pronouns % of demonstrative determiners % of determinative articles % of subordinative conjunctions % of sentence boundary punctuation % of auxiliary verbs % of modal verbs % of verbs – subjective mood % of verbs – infinitive mood % of verbs – gerundive mood % of verbs – indicative mood % of verbs – third person singular % of verbs – third person plural % of verbs – imperfect tense % of dependency types – auxiliary % of dependency types – object % of dependency types – preposition % of dependency types – subordinate clause % of dependency types – subject Avg number of verbal heads Avg number of embedded complement chains Length of ‘chains’ of embedded subordinate clauses (avg) Maximum length of dependency links (avg) Parse tree depth (avg) Arity of verbal predicates (avg) % of pre-verbal objects % of post-verbal subject % of subordinate clauses in post-verbal position types of reflective texts and average value of feature distribution in the different types of reflective texts. Ranking positions with p <0.001 are marked in italics and with p <0.05 in boldface. one. On the contrary, the model relying on very simple types of features (raw text features) that capture how much teachers have written achieves the worst results. We also carried out a very preliminary experiment to classify the three different types of reflective writings but it produced unsatisfactory results due to the unbalanced distribution of answers in the reflective classes. As expected, a balanced experiment yielded very low accuracies since we used very few data. 5
We reported first results of a on-going study devoted to reconstruct the linguistic profile of a corpus of reflective writings by Italian newly recruited teachers that we collected for the specific purpose of this paper. We are currently enlarging
F1 0 58.4 tive writings using different models of features. the corpus with new
manually annotated data to improve the accuracy of the automatic classification of different types of reflectivity. multilayer perceptron. Proceedings of Evalita’09, Evaluation of NLP and Speech Tools for Italian ,
Reggio Emilia, December.
B. Larrivee 2008. Development of a tool to assess teachers’ level of reflective practice. Reflective
Practice, Vol. 9(3).
P. Magnoler, GR. Mangione, MC. Pettenati, A. Rosa,
PG. Rossi. 2016. Induction models and teachers professional development. Journal of e-Learning and Knowledge Society, Vol. 12(3).
J. Mezirow. 1990. Fostering critical reflection in adulthood: a guide to transformative and emancipatory learning. Jossey-Bass Publishers.
N. Qazi and K. Raza. 2012. Effect of Feature Selection, SMOTE and under Sampling on Class Imbalance Classification. Proceedings of the 2012 UKSim 14th International Conference on Modelling and Simulation, pp. 145-150.
D.A. Schon. 1984. The Reflective Practitioner: How
Professionals Think In Action. Basic Books.
D.A. Schon. 1987. Educating the Reflective Practi
tioner. Jossey-Bass.
D.A. Schon. 1991. The reflective turn: Case studies in and on educational practice. Teachers College
Press.
GM. Sparks-Langer, GM. Simmons, M. Pasch, A.
Colton, A. Starko. 1990. Reflective pedagogical thinking: How can we promote it and measure it?
Journal of Teacher Education, Vol. 41(5).
T. D. Ullmann. 2015. Automated detection of reflection in texts. A machine learning based approach.
The Open University.
T. D. Ullmann. 2015. Keywords of written reflection - a comparison between reflective and descriptive datasets. Proceedings of the 5th Workshop on Awareness and Reflection in Technology Enhanced
Learning.
M. Van Manen. 1977. Linking Ways of Knowing with
Ways of Being Practical. Curriculum Inquiry, Vol.
6(3).
H.C. Waxman et al. 1987. Images of Reflection in
Teacher Education. Summaries of papers presented at a National Conference on Reflective Inquiry in Teacher Education, Houston. Avg sentence length Avg number of sentences Avg number of tokens Type/token ratio (first 100 lemma) Type/token ratio (first 200 lemma) % of “Fundamental words” of BIV % of “High usage words” of BIV % of “High availability words” of BIV Lexical density % of adjectives % of possessive adjectives % of adverbs % of negative adverbs % of determiners % of demonstrative determiners % of indefinite determiners % of prepositions % of articles % of demonstrative pronouns % of personal pronouns % of relative pronouns % of determinative articles % of subordinative conjunctions % of single commas or hyphens % of numbers % of sentence boundary punctuation % of verbs % of auxiliary verbs % of modal verbs % of verbs – subjective mood % of verbs – infinitive mood % of verbs – gerundive mood % of verbs – indicative mood % of verbs – third person singular % of verbs – third person plural % of verbs – imperfect tense % of syntactic roots % of dep–auxiliary % of dep–nominal/clausal argument % of dep–indirect complement % of dep–locative complement % of dep–temporal complement % of dep–nominal/clausal modifier % of dep–relative modifier % of dep–object % of dep–preposition % of dep–subordinate clause % of dep–subject Avg number of verbal heads Avg number of embedded complement chains Length of ‘chains’ of embedded subordinate clauses (avg) Length of dependency links (avg) Maximum length of dependency links (avg) Parse tree depth (avg) Arity of verbal predicates (avg) % of verbal roots % of verbal roots with explicit subj % of finite complement clauses % of infinite complement clauses % of pre-verbal objects % of post-verbal subject % of subordinate clauses in post-verbal position texts and ii) different types of reflective texts and average value of feature distribution in the different