In times of increasingly personalized educational content, designing a data-driven platform which o ers the opportunity to create content for di erent use cases is arguably the only solution to handle the massive amount of information. Therefore, we developed the software "Machina Callida" (MC) in our project CALLIDUS (Computer-Aided Language Learning: Vocabulary Acquisition in Latin using Corpus-based Methods). The main focus of this research project is to optimize the vocabulary acquisition of Latin by using a data-driven language learning approach for creating exercises. To achieve that goal, we were facing problems concerning the quality of externally curated research data (e.g. annotated text corpora) while curating educational materials ourselves (e.g. predened sequences of exercises). Besides, we needed to build a user-friendly interface for both teachers and students. While teachers would like to create an exercise or test and use them (even as printed out copies) in class, students would like to learn on the y and right away. As a result, we o er a repository, a le exporter for various formats and, above all, interactive exercises so that learners are actively engaged in the learning process. In this paper we show the work ow of our software and explain the architecture focusing on the integration of Arti cial Intelligence (AI) and data curation. Ideally, we want to use AI technology to facilitate the process and increase the quality of content creation, dissemination and personalization for our end users.
adaptable (or not even digital) for teachers and split into a vast amount of different items like textbook, exercise book, vocabulary book etc. that all learners have to buy separately, if needed [7, p. 194f.]. On top of that, most of the teaching materials only refer to the initial stage of language acquisition, in which Latin original texts do not yet matter [21, p. 133]. Although the companies are also providing teachers with reading books for intermediate learners containing sections of selected Latin original texts, teachers are still in continuous need of adaptable texts and exercises for these advanced stages. In addition, although the curricula o er a standardized canon of Latin authors [20, p. 45], it still includes a wide range of di erent texts compared to the available time in Latin classes. What is more, teachers prefer to use texts whose vocabulary is covered as much as possible by the basic vocabulary already acquired by the students, since the comprehensibility of the text can be considerably limited if less than 95% of words are known [25, p. 352].
As a consequence, teachers may choose texts from a large pool of Latin authors, but without supporting material they rarely do, because they lack the time to prepare texts and exercises independently. Instead, they often fall back on ready-made materials that are quality-tested but rarely t the needs of the learning group. This situation results in a kind of dilemma: Many teachers would like to enrich their lessons with further authors and support their students individually in their language acquisition with (personalized) exercises, but they do not feel up to the challenge of selecting and adapting materials to their students' needs [24, p. 115/117].
This brief outline of the problem shows the need to develop a platform that
allows teachers (and students) to create needs-based exercises for authentic Latin
texts. Furthermore, for a good user experience it is necessary that the process of
generation is fast and easy to handle, that the generated exercises are ready to
use (analogically and digitally) or share, and that they are well curated for later
reuse. These requirements are illustrated in three exemplary use cases which
have been modeled loosely following the guidelines of Cockburn [
Use Case 1: The teacher needs 2: The teacher does 3: The teacher wants exercises based on au- not have enough to support his/her thentic Latin texts time to prepare an students in a personexercise manually alized way to enable individual learning Primary actor Stakeholders Scope
As a teacher, I want As a teacher, I As a teacher, I want to select a section of search the reposi- an overview of how the work to be read. tory for at least one my students perform I want to compare matching exercise. I in an exercise. I want this section to the want to combine dif- to be able to see used core vocabulary ferent search terms at a glance what for getting an overview in an extended mistakes are made of the amount of un- search, e.g. Latin most often so that known words. Then, I text passage, exer- I know what to fowant to set the param- cise type, linguistic cus on when createters of the intended focus, popular ex- ing the next exerexercise: type of exer- ercises, vocabulary. cise. I would also cise and linguistic focus Then, I want to use like a recommenda(speci c lemmata, syn- the exercise in class tion as to which extactic structures, mor- (with smartphones, ercise to select next phology, context-based tablets or interactive if there already is a meaning, word equiva- whiteboard), to em- suitable exercise in lents). After getting a bed it in a learning the database. preview, all selections platform for later can be easily changed, use or to send it if I think that, e.g.,the to the students for exercise is too di cult. their homework.
Repetition and deep- Repetition and deep- Zone of ening of vocabulary ening of vocabulary (linguistic) knowledge in context knowledge (individ- opment ually) student of proximal develeach Precondi- Teachers are presented Teachers are pre- Students generate tion with an option to gen- sented with an data about their erate new exercises. option to browse individual progress. exercises from an The data can be existing database. tracked and analyzed automatically.
Minimal The generated exercise The database con- Many students Guaran- can be exported. tains exercises and have completed the tees can be searched. same (or similar) exercises.
Success The generated exercise The search for a Teachers receive Guaran- can be shared and is matching exercise helpful suggestions tees stored in a database is supported by for choosing the next that is easily accessible advanced ltering. exercise. to end users. Popular and wellcurated exercises are marked.
The teacher invokes The teacher decides Students have just the exercise generation to use a ready-made completed an exersetup. exercise. cise and now should attempt another one.
The teacher picks the The teacher picks The students regisoption of generating a the option of search- ter with the software new exercise. ing the database. and go through the given exercise.
The teacher chooses a The teacher selects The teacher receives text passage from a a single or multiple an evaluation about wide range of Latin au- lters or uses the the performance thors. extended search op- (percentage, ertion. ror types) of each student.
The teacher compares The teacher eval- The teacher also gets the words of the text uates the results. a recommendation with the used core vo- Depending on the which parameters cabulary and changes results, the teacher to set for the next the section accordingly changes the search exercise or which (go to step 2) or pro- terms / lters (go to exercise to select ceeds to set the param- step 2) or decides to from the database. eters of the exercise. use one of the given
exercises.
The teacher decides on The teacher uses the The students get the exercise format, the exercise in class or their new exercise linguistic focus and the disseminates it using and work on it (go instruction statement. a link, so that stu- to step 1).
dents may use their own mobile devices.
The system presents a preview. The teacher either exports the exercise to a printable format or shares it digitally or tries other parameters (go to step 4) or even changes the section (go to step 2).
Automatic parsing and evaluation: the developer's point of view In order to help teachers create high-quality educational content, we provide support for each of the necessary steps in our software at https://korpling. org/mc. 2.1
Selection of text (Use Case 1)
Many Latin text editions are proprietary and thus do not comply with the FAIR
data principles [
The vocabulary lter has to be targeted at one of several reference
vocabularies. These are essentially lemmatized word frequency lists derived from
textbooks [
Text complexity, on the other hand, does not directly relate to a student's previous knowledge, but to an intrinsic comparison between multiple Latin texts. In our case, it is a combination of well-known operationalizations of the presumed degree of di culty that readers may face when approaching a text, e.g. lexical density [19, p. 61]. This helps teachers to determine the suitability of a given text passage (or corpus) with regard to their students' linguistic competence. The major strength of such measures does not reside in their inherently awed approximation of actual complexity, but in enabling a formalized linguistic comparison that goes beyond mere counting of words and integrates syntax, morphology and semantics [11, p. 607]. By combining information about vocabulary and text complexity, teachers can signi cantly accelerate and improve their choice of texts, thus curating better educational content for their students.
Focus on speci c linguistic phenomena (Use Case 1) Once teachers have committed themselves to a suitable text passage, they may still not know the exact target of a potential exercise. Therefore, we o er a keyword in context (KWIC) view to explore collocations and the speci c usage of a particular word [18, p. 97]. The super cial token-based display is enriched by morpho-syntactic information, e.g. part of speech and dependency links. Therefore, teachers can qualitatively inspect usage patterns on multiple linguistic levels as needed.
A major problem in this approach is that most Latin texts are not curated
as treebanks with scienti c annotations, but rather just as plain text. In other
words, we lack the key prerequisite to provide a rich KWIC view. To compensate
for this shortcoming, we use an AI-driven dependency parser [
Design of interaction / learning setting (Use Case 1)
Therefore, we o er teachers the
possibility to choose from a range of
existing exercises with the same type of interaction, so it is easier for them to
maintain a certain level of consistency, even in longer learning sequences.
Furthermore, some of the exercise formats may be considered part of the same line
of progression, e.g. clozes can be solved with a visible pool of boxes using Drag
and Drop (easy, see Fig. 2) or by typing characters into blank text elds (more
di cult). Besides, the same basic technology and layout can be used to produce
di erent exercises, e.g. Drag and Drop works for both the cloze and matching
format. In this regard, the usage of a large common framework (H5P [
However, user-speci c quantitative evaluation is not enough. In order to increase students' learning success, they also need adaptive qualitative feedback. A prerequisite for that is the detection and classi cation of errors: the integrated binary evaluation of H5P can be used as a basis to categorize various error types, e.g.: Did the student fail to give any answer at all? Did the student actually provide the correct answer, but with minor typing mistakes? Did the student make obvious grammatical mistakes? If so, are they related to morphology, vocabulary or syntax? Depending on the speci c type of error, suitable feedback needs to be generated. Our main objective here is to provide deeper support for teachers and students in order to optimize the learning progress towards a speci c goal, e.g. being able to read texts from a speci c corpus. A good approach in that case may be to create exercises for this corpus and use the students' performance as an objective for reinforcement learning [13, p. 2094]. The AI model should then learn to utilize suitable pedagogical actions (e.g. distributing exercises for learning) to maximize a student's performance on the test exercise dataset for a corpus. 3
Next steps: Learning Analytics and semantic analysis
For the future integration of Learning Analytics in our software, we have already
built a prototype that evaluates a learner group's performance across multiple
dimensions, e.g. working speed, interaction type, accuracy and performance gain
over time. A large part of this analysis is most suitable for groups, which is
why it is probably useful for teachers. Individuals, on the other side, would
need a stronger emphasis on their development over time, which is harder to
track because it would require them to use the software as their main source of
language learning. Therefore, speci c milestones are to be reached in the next
months:
{ summarize group performances as an indicator that helps teachers to
readjust their general didactic strategy, e.g. by focusing more heavily on certain
linguistic phenomena
{ analyze results for individual students over time and suggest the most
suitable exercises for them considering their personal characteristics, i.e. learning
style, thematic priority and particular weaknesses
Apart from improving the quality of the existing work ow, we also consider
increasing its quantity, e.g. by adding new linguistic phenomena: Semantics is
currently underrepresented in our automatic analyses, which makes it hard for
teachers to group their educational content around a certain topic. This could
be alleviated by integrating representation learning as an independent feature:
Unsupervised machine learning, in the form of Contextual Word Embeddings
like those provided by BERT [