Expert Distribution Similarity Model: Feedback methodology for non-imitation based handwriting practice.

Abstract

Learning ne psychomotor skills such as handwriting is a tedious endeavour which requires close supervision of the teacher to master. However, the increasing number of students in classes means less time a teacher can allocate for each student. This adversely a ects the development of handwriting in students. Sensor-based technologies can help address this problem, as they are capable of providing feedback to the student whilst the teacher is not present during the student's writing. While there are multiple sensor-based applications to date for handwriting practice, such applications provide feedback in only for simple tracing over practice tasks. In this paper, we present a conceptual methodology using AI and sensors, for providing feedback in non-tracking tasks that do not have a single correct solution and allow larger variations.

Keywords

• Psychomotor skills
• Calligraphy
• Feedback
• Sensors

-

Learning handwriting can be tedious and di cult [1]. Handwriting is a fundamental but complex psychomotor skill that is universally taught to students all over the world. Numerous hours of tiresome practice is required to develop and improve handwriting. Marquardt et al. [ 2 ] stated that more than 30% of girls and more than 50% of boys have problems acquiring uid and legible handwriting. They identi ed lack of practice of psychomotor skills as one of the causes behind the problem, to which they suggested \special writing motor skills training" and \more time for assistance in class". However, increasing the number of students in classes along with curriculums that favour handwriting less due to digitisation, leads to fewer time teachers and students can dedicate to teach/learn handwriting. This often leads to students practicing by themselves and teachers only having access to the nal static image of the hand writing practice sessions for providing feedback. Feedback provided in such cases often ignore psychomotor aspects of hand-writing learning, i.e. handwriting parameters (HWP ), such as pressure and tilt of the pen [ 3 ]. This in-turn leads to minimal focus on students' learning process which results in development of improper psychomotor skills Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). and often, internalisation of those techniques which can be di cult to forget and also hampers their progress [ 4 ], [ 5 ]. Therefore, feedback on the HWP which can lead to proper psychomotor skills development for handwriting [ 6 ] is crucial during practice. In this paper, we present a conceptual solution for providing real-time feedback to students on their psychomotor performance based on an expert model, when a teacher is not present to provide real-time feedback. It also provides teachers with information about the students' psychomotor performance in addition to the nal static image which can help teachers provide more e ective summative feedback to students. The conceptual solution is initially developed speci cally with Roman characters in mind because from a practical perspective, Roman characters are more accessible to us making it easier to nd experts/students/teachers. However, our over-arching goal is to design a solution that focuses on time series path and will be applicable on various other similar domains. Furthermore we also brie y introduce the current state of the prototype. The conceptual solution presented in this paper is the next step towards nalising the prototype, so that it can provide feedback in non-tracing handwriting exercises. 2

Background

Sensor-based technology have the potential to address the issue of insu cient feedback due to lack of teachers during students' handwriting practice. Sensors have been previously used to track students HWP and provide guidance and feedback on tracing handwriting exercises [ 7 ], [ 8 ]. Tracing exercises are performed at imitation level of skills development by novices [ 9 ], according to Dave's Taxonomy [ 10 ]. Sensors can, potentially, also be used to support the handwriting practice in the successive levels of Dave's Taxonomy (see Table 1). Practice at di erent levels of the model requires varying conditions and also varying degree of teachers' involvement (see Table 1). The rst level, i.e. imitation level of psychomotor development, is practised by trace-over exercises and the student needs to closely replicate the expert performance. Correspondingly, the teacher observes the student closely and provides as much support as possible. For example Limbu et. al, [ 7 ] in their prototype Calligraphy Trainer and loup-Escande et al. (2017) [ 8 ] focus on the imitation level of psychomotor development. They used sensors to provide continuous feedback on the HWP using an expert model, however the later did not use auditory channels for any feedback. They used naive methods to identify errors which su ces at the imitation level, but only allows for variation of the temporal dimension (i.e. writing speed). In the manipulation level or higher, the student is not required to trace and copy the expert exactly, as long the minimum requirements are met. Therefore, there is no single correct answer: since variation is allowed in not only the temporal but also the spatial dimension (i.e. rotation, scale, aspect ratio), the system should take these acceptable variations into account to provide feedback, for which new dissimilarity measures are needed. In addition, giving feedback for levels higher than Precision requires context awareness about the creativity and artisticness for which current state of the art AI technology is not yet capable of and therefore, better suited for a human expert to review. In the following sections, the prototype in its current state is presented along with the conceptual solution for providing feedback at the manipulation and precision levels.

Student roles

Dreyfus' levels expertise [ 9 ] of Daves' level Teacher's roles of psychomotor learning [ 10 ] Imitation

Observes the student closely Imitates the expert's perfor- Novice to provide feedback and mance guidance.

Manipulation Sca olds the student.

Prototypical development to-date

The prototype (in its current state) developed in the context of this project was developed with windows ink api and .Net framework. Only a single expert recording is used as a learning content and feedback is given by comparing the student performance data points to this single expert recording using tolerance thresholds to determine when the student deviates enough from the expert to consider it an error. Near real-time feedback is given over several modalities such as audio feedback on stroke speed or visual feedback on accuracy by changing the colour of the ink (see gure 1), saturation and the width of the stroke. The prototype provides summative feedback after completion of an exercise and is currently in the form of graphs showing the di erence between student and expert data for selected features. Both type of feedback is based on euclidean distances measurements between student and expert data-points. Consequently, it only supports practising at the imitation level with basic real-time feedback, similar to Limbu, Jarodzka, Klemke, et al. [ 7 ] and Loup-Escande, Frenoy, Poplimont, et al. [ 8 ]. As seen in Figure 2, this method is unable to align the expert and students' timeline when the scales are di erent, hence being un t for handwriting practice at higher levels. Therefore, further development of this prototype to support manipulation level, or higher, requires matching student data-points with expert data-points using a more complex models for the expert data set and sequence alignment which is described below as a conceptual solution. As we mentioned above, the current prototype's feedback doesn't account for the variations that the students can make at manipulation and precision level practice. To provide feedback at these levels the model needs to be able to compare the student data-points with the correct corresponding expert data-points for which an adequate methodological approach is needed. We propose a conceptual methodology below which aims to provide near real-time and summative feedback at levels higher than imitation. This methodology has three main components: An Expert model, a Dissimilarity Measurement Model and an Error Classi er (see gure 3).

Expert Model The expert model needs to capture the acceptable variation of HWP, for which several experts need to make several recordings for a single target learning instance. When a student selects and starts an exercise, the relevant expert distribution is loaded from the expert model and used by the dissimilarity measurement model as a representation of the target performance and allowed variations.

Dissimilarity Measurement Model The Dissimilarity Measurement Model computes the di erence between the expert and the student in multiple dimensions, providing a dissimilarity score per data-point point. For this component, a sequence alignment method is needed to be able to compare the same target output sections. A well known sequence alignment method is Dynamic Time Warping. However, for aligning non- nished sequences, it is needed to split the target output into several sections using key-points (see [ 11 ]). Doing so allows for near real-time alignment of sub-sequences and therefore, enabling near real-time feedback. After matching data-points, a student's output should be re-scaled and rotated to spatially match the expert output. The dissimilarity measurement model will make use of both low level and high level features. The low level features are used to have a detailed mapping from strokes and their properties to the movements that created them. Higher level features are used to capture more context (e.g. if lines are parallel, if sizes of subsections are in proportion, if the average angle of near vertical lines is consistent. . . ). Di erent HWP are used by this model for real-time and summative feedback and therefore, di erent error values are produced during real-time and batched sessions for summative feedback. This error values are then passed on to the corresponding Error classi ers (see Figure 3).

Error Classi er To provide feedback, the system needs to rst recognise the errors. An error classi er is needed to detect the presence and amplitude of mistakes in the student performance with respect to expert dataset. The error classi er component uses the output of the dissimilarity measurement component to classify subsections of the recording into known errors, and feeds its output (the identi ed errors) to the feedback module (see Figure 3). 4.1

Future Work To improve upon the current prototype and allow practice of the manipulation level, several steps in the methodology needs to be implemented. Firstly, the expert model in the current prototype is a single recording and should become a distribution inferred from a set of recordings. Secondly, dynamic time warping starting with key-point detection for splitting the data sequence in sections should be implemented. Once the student and expert sequences can be correctly aligned the di erence between student and expert data can be calculated for each feature, however, feature engineering needs to be performed to have more representative and meaningful features. Third, an error classi er needs to be trained to convert dissimilarity scores to known errors. Finally, the conceptual solution/methodology needs to be tested for its accuracy and performance.

References

K. P. Feder and A. Majnemer , \ Handwriting development, competency, and intervention," Developmental Medicine & Child Neurology , vol. 49 , no. 4 , pp. 312 { 317 , 2007 .

[2] C. Marquardt , M. Diaz Meyer, M. Schneider, and R. Hilgemann , \ Learning handwriting at school { a teachers' survey on actual problems and future options," Trends in Neuroscience and Education , vol. 5 , no. 3 , pp. 82 { 89 , 2016 , Writing in the digital age , issn: 2211 - 9493 . doi: https : / / doi . org/10.1016/j.tine. 2016 . 07 .001. [Online]. Available: https://www. sciencedirect.com/science/article/pii/S2211949316300126.

[3] T. Asselborn , T. Gargot , L. Kidzinski , W. Johal , D. Cohen , C. Jolly , and P. Dillenbourg , \ Automated human -level diagnosis of dysgraphia using a consumer tablet," NPJ Digital Medicine , vol. 1 , no. 1 , pp. 1 { 9 , 2018 .

[4] K. P. Feder and A. Majnemer , \ Handwriting development, competency, and intervention," Developmental Medicine & Child Neurology , vol. 49 , no. 4 , pp. 312 { 317 , 2007 . doi: https : / / doi . org / 10 . 1111 / j . 1469 - 8749 . 2007 . 00312 . x. eprint: https : / / onlinelibrary . wiley . com / doi / pdf / 10 . 1111 / j . 1469 - 8749 . 2007 . 00312 . x. [Online]. Available: https : / / onlinelibrary . wiley . com / doi / abs / 10 . 1111 / j . 1469 - 8749 . 2007 . 00312 .x.

[5] R. W. Barnes , \ Surgical handicraft: Teaching and learning surgical skills," The American Journal of Surgery , vol. 153 , no. 5 , pp. 422 { 427 , 1987 , Papers of the North Paci c Surgical Association, issn: 0002 - 9610 . doi: https://doi.org/10.1016/ 0002 - 9610 ( 87 ) 90783 - 5 . [Online]. Available: https://www.sciencedirect.com/science/article/pii/0002961087907835.

[6] M. Thorpe , Modern Calligraphy: Everything You Need to Know to Get Started in Script Calligraphy . St. Martin's Publishing Group , 2013 , isbn: 9781250016324 . [Online]. Available: https://books.google.nl/books? id=GEBGAgAAQBAJ.

[7] B. H. Limbu , H. Jarodzka , R. Klemke , and M. Specht , \ Can you ink while you blink? Assessing mental e ort in a sensor-based calligraphy trainer ., " eng , Sensors (Basel, Switzerland), vol. 19 , no. 14 , Jul . 2019 , issn: 1424 - 8220 1424- 8220 . doi: 10 .3390/s19143244.

[8] E. Loup-Escande , R. Frenoy , G. Poplimont, I. Thouvenin , O. Gapenne , and O. Megalakaki , \ Contributions of mixed reality in a calligraphy learning task: E ects of supplementary visual feedback and expertise on cognitive load, user experience and gestural performance," Computers in Human Behavior , vol. 75 , pp. 42 { 49 , 2017 .

[9] H. L. Dreyfus and S. E. Dreyfus , With athanasiou, t.( 1986 ) mind over machine: The power of human intuition and expertise in the era of the computer .

[10] R. Dave , Developing and writing behavioural objectives . Educational Innovators Press, 1970 .

[11] A.-T. Chiang , Q. Chen , Y. Wang , and M. R. Fu , \ Motion Sequence Alignment for A Kinect-Based In-Home Exercise System for Lymphatic Health and Lymphedema Intervention," in 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) , IEEE, 2018 , pp. 2072 { 2075 .