Expert Distribution Similarity Model: Feedback methodology for non-imitation based handwriting practice. Olivier Dikken1[0000−0002−7455−3471] , Bibeg Limbu2[0000−0002−1269−6864] , and Marcus Specht1,2[0000−0002−6086−8480] 1 TU Delft 2 Leiden Delft Erasmus center for education and learning, TU Delft Abstract. Learning fine psychomotor skills such as handwriting is a te- dious endeavour which requires close supervision of the teacher to master. However, the increasing number of students in classes means less time a teacher can allocate for each student. This adversely affects the devel- opment of handwriting in students. Sensor-based technologies can help address this problem, as they are capable of providing feedback to the student whilst the teacher is not present during the student’s writing. While there are multiple sensor-based applications to date for handwrit- ing practice, such applications provide feedback in only for simple tracing over practice tasks. In this paper, we present a conceptual methodology using AI and sensors, for providing feedback in non-tracking tasks that do not have a single correct solution and allow larger variations. Keywords: Psychomotor skills · Calligraphy · Feedback · Sensors 1 Introduction Learning handwriting can be tedious and difficult [1]. Handwriting is a funda- mental but complex psychomotor skill that is universally taught to students all over the world. Numerous hours of tiresome practice is required to develop and improve handwriting. Marquardt et al. [2] stated that more than 30% of girls and more than 50% of boys have problems acquiring fluid and legible handwriting. They identified lack of practice of psychomotor skills as one of the causes behind the problem, to which they suggested “special writing motor skills training” and “more time for assistance in class”. However, increasing the number of students in classes along with curriculums that favour handwriting less due to digitisation, leads to fewer time teachers and students can dedicate to teach/learn handwrit- ing. This often leads to students practicing by themselves and teachers only having access to the final static image of the hand writing practice sessions for providing feedback. Feedback provided in such cases often ignore psychomotor aspects of hand-writing learning, i.e. handwriting parameters (HWP ), such as pressure and tilt of the pen [3]. This in-turn leads to minimal focus on students’ learning process which results in development of improper psychomotor skills Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2 Dikken et al. and often, internalisation of those techniques which can be difficult to forget and also hampers their progress [4], [5]. Therefore, feedback on the HWP which can lead to proper psychomotor skills development for handwriting [6] is crucial during practice. In this paper, we present a conceptual solution for providing real-time feedback to students on their psychomotor performance based on an expert model, when a teacher is not present to provide real-time feedback. It also provides teachers with information about the students’ psychomotor per- formance in addition to the final static image which can help teachers provide more effective summative feedback to students. The conceptual solution is ini- tially developed specifically with Roman characters in mind because from a prac- tical perspective, Roman characters are more accessible to us making it easier to find experts/students/teachers. However, our over-arching goal is to design a solution that focuses on time series path and will be applicable on various other similar domains. Furthermore we also briefly introduce the current state of the prototype. The conceptual solution presented in this paper is the next step to- wards finalising the prototype, so that it can provide feedback in non-tracing handwriting exercises. 2 Background Sensor-based technology have the potential to address the issue of insufficient feedback due to lack of teachers during students’ handwriting practice. Sen- sors have been previously used to track students HWP and provide guidance and feedback on tracing handwriting exercises [7], [8]. Tracing exercises are per- formed at imitation level of skills development by novices [9], according to Dave’s Taxonomy [10]. Sensors can, potentially, also be used to support the handwrit- ing practice in the successive levels of Dave’s Taxonomy (see Table 1). Practice at different levels of the model requires varying conditions and also varying de- gree of teachers’ involvement (see Table 1). The first level, i.e. imitation level of psychomotor development, is practised by trace-over exercises and the student needs to closely replicate the expert performance. Correspondingly, the teacher observes the student closely and provides as much support as possible. For exam- ple Limbu et. al, [7] in their prototype Calligraphy Trainer and loup-Escande et al. (2017) [8] focus on the imitation level of psychomotor development. They used sensors to provide continuous feedback on the HWP using an expert model, how- ever the later did not use auditory channels for any feedback. They used naive methods to identify errors which suffices at the imitation level, but only allows for variation of the temporal dimension (i.e. writing speed). In the manipulation level or higher, the student is not required to trace and copy the expert exactly, as long the minimum requirements are met. Therefore, there is no single correct answer: since variation is allowed in not only the temporal but also the spatial dimension (i.e. rotation, scale, aspect ratio), the system should take these ac- ceptable variations into account to provide feedback, for which new dissimilarity measures are needed. In addition, giving feedback for levels higher than Preci- sion requires context awareness about the creativity and artisticness for which Title Suppressed Due to Excessive Length 3 current state of the art AI technology is not yet capable of and therefore, better suited for a human expert to review. In the following sections, the prototype in its current state is presented along with the conceptual solution for providing feedback at the manipulation and precision levels. Daves’ level Teacher’s roles Student roles Dreyfus’ of psy- levels of chomotor expertise learning [9] [10] Imitation Observes the student closely Imitates the expert’s perfor- Novice to provide feedback and mance guidance. Manipulation Scaffolds the student. Performs the task with in- Advanced structions or reference model Beginner which is removed soon after Precision Provides no real-time feed- Performs with a certain de- Competent back but still provides sum- gree of precision and accu- mative feedback racy, with few errors Articulation Teacher’s role is faded into a Modifies movement or com- Proficient consultation role. bines various skills to fit novel requirements. NaturalizationTeachert is not needed any- Performs with little physical Expert more. or mental exertion. Table 1. Handwriting exercise conditions and required software functionality per level of psychomotor skills development according to Dave’s Taxonomy 3 Prototypical development to-date The prototype (in its current state) developed in the context of this project was developed with windows ink api and .Net framework. Only a single expert record- ing is used as a learning content and feedback is given by comparing the student performance data points to this single expert recording using tolerance thresh- olds to determine when the student deviates enough from the expert to consider it an error. Near real-time feedback is given over several modalities such as audio feedback on stroke speed or visual feedback on accuracy by changing the colour of the ink (see figure 1), saturation and the width of the stroke. The prototype provides summative feedback after completion of an exercise and is currently in the form of graphs showing the difference between student and expert data for selected features. Both type of feedback is based on euclidean distances measure- ments between student and expert data-points. Consequently, it only supports practising at the imitation level with basic real-time feedback, similar to Limbu, Jarodzka, Klemke, et al. [7] and Loup-Escande, Frenoy, Poplimont, et al. [8]. As seen in Figure 2, this method is unable to align the expert and students’ timeline when the scales are different, hence being unfit for handwriting practice at higher 4 Dikken et al. levels. Therefore, further development of this prototype to support manipulation level, or higher, requires matching student data-points with expert data-points using a more complex models for the expert data set and sequence alignment which is described below as a conceptual solution. Fig. 1. Prototype Tracing Exercise with near real-time feedback on accuracy using the ink colour Fig. 2. Prototype: batched feedback on HWP pressure 4 Conceptual solution As we mentioned above, the current prototype’s feedback doesn’t account for the variations that the students can make at manipulation and precision level Title Suppressed Due to Excessive Length 5 practice. To provide feedback at these levels the model needs to be able to com- pare the student data-points with the correct corresponding expert data-points for which an adequate methodological approach is needed. We propose a concep- tual methodology below which aims to provide near real-time and summative feedback at levels higher than imitation. This methodology has three main com- ponents: An Expert model, a Dissimilarity Measurement Model and an Error Classifier (see figure 3). Fig. 3. Overview of solution main components Expert Model The expert model needs to capture the acceptable variation of HWP, for which several experts need to make several recordings for a single target learning instance. When a student selects and starts an exercise, the relevant expert distribution is loaded from the expert model and used by the dissimilarity measurement model as a representation of the target performance and allowed variations. Dissimilarity Measurement Model The Dissimilarity Measurement Model computes the difference between the expert and the student in multiple dimen- sions, providing a dissimilarity score per data-point point. For this component, a sequence alignment method is needed to be able to compare the same target 6 Dikken et al. output sections. A well known sequence alignment method is Dynamic Time Warping. However, for aligning non-finished sequences, it is needed to split the target output into several sections using key-points (see [11]). Doing so allows for near real-time alignment of sub-sequences and therefore, enabling near real-time feedback. After matching data-points, a student’s output should be re-scaled and rotated to spatially match the expert output. The dissimilarity measure- ment model will make use of both low level and high level features. The low level features are used to have a detailed mapping from strokes and their prop- erties to the movements that created them. Higher level features are used to capture more context (e.g. if lines are parallel, if sizes of subsections are in pro- portion, if the average angle of near vertical lines is consistent. . . ). Different HWP are used by this model for real-time and summative feedback and there- fore, different error values are produced during real-time and batched sessions for summative feedback. This error values are then passed on to the corresponding Error classifiers (see Figure 3). Error Classifier To provide feedback, the system needs to first recognise the errors. An error classifier is needed to detect the presence and amplitude of mistakes in the student performance with respect to expert dataset. The error classifier component uses the output of the dissimilarity measurement component to classify subsections of the recording into known errors, and feeds its output (the identified errors) to the feedback module (see Figure 3). 4.1 Future Work To improve upon the current prototype and allow practice of the manipulation level, several steps in the methodology needs to be implemented. Firstly, the expert model in the current prototype is a single recording and should become a distribution inferred from a set of recordings. Secondly, dynamic time warp- ing starting with key-point detection for splitting the data sequence in sections should be implemented. Once the student and expert sequences can be correctly aligned the difference between student and expert data can be calculated for each feature, however, feature engineering needs to be performed to have more representative and meaningful features. Third, an error classifier needs to be trained to convert dissimilarity scores to known errors. Finally, the conceptual solution/methodology needs to be tested for its accuracy and performance. References [1] K. P. Feder and A. Majnemer, “Handwriting development, competency, and intervention,” Developmental Medicine & Child Neurology, vol. 49, no. 4, pp. 312–317, 2007. Title Suppressed Due to Excessive Length 7 [2] C. Marquardt, M. Diaz Meyer, M. Schneider, and R. Hilgemann, “Learning handwriting at school – a teachers’ survey on actual problems and future options,” Trends in Neuroscience and Education, vol. 5, no. 3, pp. 82–89, 2016, Writing in the digital age, issn: 2211-9493. doi: https : / / doi . org/10.1016/j.tine.2016.07.001. [Online]. Available: https://www. sciencedirect.com/science/article/pii/S2211949316300126. [3] T. Asselborn, T. Gargot, L. Kidziński, W. Johal, D. Cohen, C. Jolly, and P. Dillenbourg, “Automated human-level diagnosis of dysgraphia using a consumer tablet,” NPJ Digital Medicine, vol. 1, no. 1, pp. 1–9, 2018. [4] K. P. Feder and A. Majnemer, “Handwriting development, competency, and intervention,” Developmental Medicine & Child Neurology, vol. 49, no. 4, pp. 312–317, 2007. doi: https : / / doi . org / 10 . 1111 / j . 1469 - 8749 . 2007 . 00312 . x. eprint: https : / / onlinelibrary . wiley . com / doi / pdf / 10 . 1111 / j . 1469 - 8749 . 2007 . 00312 . x. [Online]. Available: https : / / onlinelibrary . wiley . com / doi / abs / 10 . 1111 / j . 1469 - 8749.2007.00312.x. [5] R. W. Barnes, “Surgical handicraft: Teaching and learning surgical skills,” The American Journal of Surgery, vol. 153, no. 5, pp. 422–427, 1987, Papers of the North Pacific Surgical Association, issn: 0002-9610. doi: https://doi.org/10.1016/0002-9610(87)90783-5. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0002961087907835. [6] M. Thorpe, Modern Calligraphy: Everything You Need to Know to Get Started in Script Calligraphy. St. Martin’s Publishing Group, 2013, isbn: 9781250016324. [Online]. Available: https://books.google.nl/books? id=GEBGAgAAQBAJ. [7] B. H. Limbu, H. Jarodzka, R. Klemke, and M. Specht, “Can you ink while you blink? Assessing mental effort in a sensor-based calligraphy trainer.,” eng, Sensors (Basel, Switzerland), vol. 19, no. 14, Jul. 2019, issn: 1424- 8220 1424-8220. doi: 10.3390/s19143244. [8] E. Loup-Escande, R. Frenoy, G. Poplimont, I. Thouvenin, O. Gapenne, and O. Megalakaki, “Contributions of mixed reality in a calligraphy learning task: Effects of supplementary visual feedback and expertise on cognitive load, user experience and gestural performance,” Computers in Human Behavior, vol. 75, pp. 42–49, 2017. [9] H. L. Dreyfus and S. E. Dreyfus, With athanasiou, t.(1986) mind over machine: The power of human intuition and expertise in the era of the computer. [10] R. Dave, Developing and writing behavioural objectives. Educational Inno- vators Press, 1970. [11] A.-T. Chiang, Q. Chen, Y. Wang, and M. R. Fu, “Motion Sequence Align- ment for A Kinect-Based In-Home Exercise System for Lymphatic Health and Lymphedema Intervention,” in 2018 40th Annual International Con- ference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, 2018, pp. 2072–2075.