=Paper=
{{Paper
|id=Vol-2979/paper6
|storemode=property
|title=Expert Distribution Similarity Model: Feedback methodology for non-imitation based handwriting practice.
|pdfUrl=https://ceur-ws.org/Vol-2979/paper6.pdf
|volume=Vol-2979
|authors=Olivier Dikken,Bibeg Limbu,Marcus Specht
|dblpUrl=https://dblp.org/rec/conf/ectel/DikkenLS21
}}
==Expert Distribution Similarity Model: Feedback methodology for non-imitation based handwriting practice.==
https://ceur-ws.org/Vol-2979/paper6.pdf
Expert Distribution Similarity Model: Feedback
methodology for non-imitation based
handwriting practice.
Olivier Dikken1[0000−0002−7455−3471] , Bibeg Limbu2[0000−0002−1269−6864] , and
Marcus Specht1,2[0000−0002−6086−8480]
1
TU Delft
2
Leiden Delft Erasmus center for education and learning, TU Delft
Abstract. Learning fine psychomotor skills such as handwriting is a te-
dious endeavour which requires close supervision of the teacher to master.
However, the increasing number of students in classes means less time a
teacher can allocate for each student. This adversely affects the devel-
opment of handwriting in students. Sensor-based technologies can help
address this problem, as they are capable of providing feedback to the
student whilst the teacher is not present during the student’s writing.
While there are multiple sensor-based applications to date for handwrit-
ing practice, such applications provide feedback in only for simple tracing
over practice tasks. In this paper, we present a conceptual methodology
using AI and sensors, for providing feedback in non-tracking tasks that
do not have a single correct solution and allow larger variations.
Keywords: Psychomotor skills · Calligraphy · Feedback · Sensors
1 Introduction
Learning handwriting can be tedious and difficult [1]. Handwriting is a funda-
mental but complex psychomotor skill that is universally taught to students all
over the world. Numerous hours of tiresome practice is required to develop and
improve handwriting. Marquardt et al. [2] stated that more than 30% of girls and
more than 50% of boys have problems acquiring fluid and legible handwriting.
They identified lack of practice of psychomotor skills as one of the causes behind
the problem, to which they suggested “special writing motor skills training” and
“more time for assistance in class”. However, increasing the number of students
in classes along with curriculums that favour handwriting less due to digitisation,
leads to fewer time teachers and students can dedicate to teach/learn handwrit-
ing. This often leads to students practicing by themselves and teachers only
having access to the final static image of the hand writing practice sessions for
providing feedback. Feedback provided in such cases often ignore psychomotor
aspects of hand-writing learning, i.e. handwriting parameters (HWP ), such as
pressure and tilt of the pen [3]. This in-turn leads to minimal focus on students’
learning process which results in development of improper psychomotor skills
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
2 Dikken et al.
and often, internalisation of those techniques which can be difficult to forget
and also hampers their progress [4], [5]. Therefore, feedback on the HWP which
can lead to proper psychomotor skills development for handwriting [6] is crucial
during practice. In this paper, we present a conceptual solution for providing
real-time feedback to students on their psychomotor performance based on an
expert model, when a teacher is not present to provide real-time feedback. It
also provides teachers with information about the students’ psychomotor per-
formance in addition to the final static image which can help teachers provide
more effective summative feedback to students. The conceptual solution is ini-
tially developed specifically with Roman characters in mind because from a prac-
tical perspective, Roman characters are more accessible to us making it easier
to find experts/students/teachers. However, our over-arching goal is to design a
solution that focuses on time series path and will be applicable on various other
similar domains. Furthermore we also briefly introduce the current state of the
prototype. The conceptual solution presented in this paper is the next step to-
wards finalising the prototype, so that it can provide feedback in non-tracing
handwriting exercises.
2 Background
Sensor-based technology have the potential to address the issue of insufficient
feedback due to lack of teachers during students’ handwriting practice. Sen-
sors have been previously used to track students HWP and provide guidance
and feedback on tracing handwriting exercises [7], [8]. Tracing exercises are per-
formed at imitation level of skills development by novices [9], according to Dave’s
Taxonomy [10]. Sensors can, potentially, also be used to support the handwrit-
ing practice in the successive levels of Dave’s Taxonomy (see Table 1). Practice
at different levels of the model requires varying conditions and also varying de-
gree of teachers’ involvement (see Table 1). The first level, i.e. imitation level of
psychomotor development, is practised by trace-over exercises and the student
needs to closely replicate the expert performance. Correspondingly, the teacher
observes the student closely and provides as much support as possible. For exam-
ple Limbu et. al, [7] in their prototype Calligraphy Trainer and loup-Escande et
al. (2017) [8] focus on the imitation level of psychomotor development. They used
sensors to provide continuous feedback on the HWP using an expert model, how-
ever the later did not use auditory channels for any feedback. They used naive
methods to identify errors which suffices at the imitation level, but only allows
for variation of the temporal dimension (i.e. writing speed). In the manipulation
level or higher, the student is not required to trace and copy the expert exactly,
as long the minimum requirements are met. Therefore, there is no single correct
answer: since variation is allowed in not only the temporal but also the spatial
dimension (i.e. rotation, scale, aspect ratio), the system should take these ac-
ceptable variations into account to provide feedback, for which new dissimilarity
measures are needed. In addition, giving feedback for levels higher than Preci-
sion requires context awareness about the creativity and artisticness for which
Title Suppressed Due to Excessive Length 3
current state of the art AI technology is not yet capable of and therefore, better
suited for a human expert to review. In the following sections, the prototype in
its current state is presented along with the conceptual solution for providing
feedback at the manipulation and precision levels.
Daves’ level Teacher’s roles Student roles Dreyfus’
of psy- levels of
chomotor expertise
learning [9]
[10]
Imitation Observes the student closely Imitates the expert’s perfor- Novice
to provide feedback and mance
guidance.
Manipulation Scaffolds the student. Performs the task with in- Advanced
structions or reference model Beginner
which is removed soon after
Precision Provides no real-time feed- Performs with a certain de- Competent
back but still provides sum- gree of precision and accu-
mative feedback racy, with few errors
Articulation Teacher’s role is faded into a Modifies movement or com- Proficient
consultation role. bines various skills to fit novel
requirements.
NaturalizationTeachert is not needed any- Performs with little physical Expert
more. or mental exertion.
Table 1. Handwriting exercise conditions and required software functionality per level
of psychomotor skills development according to Dave’s Taxonomy
3 Prototypical development to-date
The prototype (in its current state) developed in the context of this project was
developed with windows ink api and .Net framework. Only a single expert record-
ing is used as a learning content and feedback is given by comparing the student
performance data points to this single expert recording using tolerance thresh-
olds to determine when the student deviates enough from the expert to consider
it an error. Near real-time feedback is given over several modalities such as audio
feedback on stroke speed or visual feedback on accuracy by changing the colour
of the ink (see figure 1), saturation and the width of the stroke. The prototype
provides summative feedback after completion of an exercise and is currently in
the form of graphs showing the difference between student and expert data for
selected features. Both type of feedback is based on euclidean distances measure-
ments between student and expert data-points. Consequently, it only supports
practising at the imitation level with basic real-time feedback, similar to Limbu,
Jarodzka, Klemke, et al. [7] and Loup-Escande, Frenoy, Poplimont, et al. [8]. As
seen in Figure 2, this method is unable to align the expert and students’ timeline
when the scales are different, hence being unfit for handwriting practice at higher
4 Dikken et al.
levels. Therefore, further development of this prototype to support manipulation
level, or higher, requires matching student data-points with expert data-points
using a more complex models for the expert data set and sequence alignment
which is described below as a conceptual solution.
Fig. 1. Prototype Tracing Exercise with near real-time feedback on accuracy using the
ink colour
Fig. 2. Prototype: batched feedback on HWP pressure
4 Conceptual solution
As we mentioned above, the current prototype’s feedback doesn’t account for
the variations that the students can make at manipulation and precision level
Title Suppressed Due to Excessive Length 5
practice. To provide feedback at these levels the model needs to be able to com-
pare the student data-points with the correct corresponding expert data-points
for which an adequate methodological approach is needed. We propose a concep-
tual methodology below which aims to provide near real-time and summative
feedback at levels higher than imitation. This methodology has three main com-
ponents: An Expert model, a Dissimilarity Measurement Model and an Error
Classifier (see figure 3).
Fig. 3. Overview of solution main components
Expert Model The expert model needs to capture the acceptable variation
of HWP, for which several experts need to make several recordings for a single
target learning instance. When a student selects and starts an exercise, the
relevant expert distribution is loaded from the expert model and used by the
dissimilarity measurement model as a representation of the target performance
and allowed variations.
Dissimilarity Measurement Model The Dissimilarity Measurement Model
computes the difference between the expert and the student in multiple dimen-
sions, providing a dissimilarity score per data-point point. For this component,
a sequence alignment method is needed to be able to compare the same target
6 Dikken et al.
output sections. A well known sequence alignment method is Dynamic Time
Warping. However, for aligning non-finished sequences, it is needed to split the
target output into several sections using key-points (see [11]). Doing so allows for
near real-time alignment of sub-sequences and therefore, enabling near real-time
feedback. After matching data-points, a student’s output should be re-scaled
and rotated to spatially match the expert output. The dissimilarity measure-
ment model will make use of both low level and high level features. The low
level features are used to have a detailed mapping from strokes and their prop-
erties to the movements that created them. Higher level features are used to
capture more context (e.g. if lines are parallel, if sizes of subsections are in pro-
portion, if the average angle of near vertical lines is consistent. . . ). Different
HWP are used by this model for real-time and summative feedback and there-
fore, different error values are produced during real-time and batched sessions for
summative feedback. This error values are then passed on to the corresponding
Error classifiers (see Figure 3).
Error Classifier To provide feedback, the system needs to first recognise
the errors. An error classifier is needed to detect the presence and amplitude of
mistakes in the student performance with respect to expert dataset. The error
classifier component uses the output of the dissimilarity measurement component
to classify subsections of the recording into known errors, and feeds its output
(the identified errors) to the feedback module (see Figure 3).
4.1 Future Work
To improve upon the current prototype and allow practice of the manipulation
level, several steps in the methodology needs to be implemented. Firstly, the
expert model in the current prototype is a single recording and should become
a distribution inferred from a set of recordings. Secondly, dynamic time warp-
ing starting with key-point detection for splitting the data sequence in sections
should be implemented. Once the student and expert sequences can be correctly
aligned the difference between student and expert data can be calculated for
each feature, however, feature engineering needs to be performed to have more
representative and meaningful features. Third, an error classifier needs to be
trained to convert dissimilarity scores to known errors. Finally, the conceptual
solution/methodology needs to be tested for its accuracy and performance.
References
[1] K. P. Feder and A. Majnemer, “Handwriting development, competency,
and intervention,” Developmental Medicine & Child Neurology, vol. 49,
no. 4, pp. 312–317, 2007.
Title Suppressed Due to Excessive Length 7
[2] C. Marquardt, M. Diaz Meyer, M. Schneider, and R. Hilgemann, “Learning
handwriting at school – a teachers’ survey on actual problems and future
options,” Trends in Neuroscience and Education, vol. 5, no. 3, pp. 82–89,
2016, Writing in the digital age, issn: 2211-9493. doi: https : / / doi .
org/10.1016/j.tine.2016.07.001. [Online]. Available: https://www.
sciencedirect.com/science/article/pii/S2211949316300126.
[3] T. Asselborn, T. Gargot, L. Kidziński, W. Johal, D. Cohen, C. Jolly, and
P. Dillenbourg, “Automated human-level diagnosis of dysgraphia using a
consumer tablet,” NPJ Digital Medicine, vol. 1, no. 1, pp. 1–9, 2018.
[4] K. P. Feder and A. Majnemer, “Handwriting development, competency,
and intervention,” Developmental Medicine & Child Neurology, vol. 49,
no. 4, pp. 312–317, 2007. doi: https : / / doi . org / 10 . 1111 / j . 1469 -
8749 . 2007 . 00312 . x. eprint: https : / / onlinelibrary . wiley . com /
doi / pdf / 10 . 1111 / j . 1469 - 8749 . 2007 . 00312 . x. [Online]. Available:
https : / / onlinelibrary . wiley . com / doi / abs / 10 . 1111 / j . 1469 -
8749.2007.00312.x.
[5] R. W. Barnes, “Surgical handicraft: Teaching and learning surgical skills,”
The American Journal of Surgery, vol. 153, no. 5, pp. 422–427, 1987,
Papers of the North Pacific Surgical Association, issn: 0002-9610. doi:
https://doi.org/10.1016/0002-9610(87)90783-5. [Online]. Available:
https://www.sciencedirect.com/science/article/pii/0002961087907835.
[6] M. Thorpe, Modern Calligraphy: Everything You Need to Know to Get
Started in Script Calligraphy. St. Martin’s Publishing Group, 2013, isbn:
9781250016324. [Online]. Available: https://books.google.nl/books?
id=GEBGAgAAQBAJ.
[7] B. H. Limbu, H. Jarodzka, R. Klemke, and M. Specht, “Can you ink while
you blink? Assessing mental effort in a sensor-based calligraphy trainer.,”
eng, Sensors (Basel, Switzerland), vol. 19, no. 14, Jul. 2019, issn: 1424-
8220 1424-8220. doi: 10.3390/s19143244.
[8] E. Loup-Escande, R. Frenoy, G. Poplimont, I. Thouvenin, O. Gapenne, and
O. Megalakaki, “Contributions of mixed reality in a calligraphy learning
task: Effects of supplementary visual feedback and expertise on cognitive
load, user experience and gestural performance,” Computers in Human
Behavior, vol. 75, pp. 42–49, 2017.
[9] H. L. Dreyfus and S. E. Dreyfus, With athanasiou, t.(1986) mind over
machine: The power of human intuition and expertise in the era of the
computer.
[10] R. Dave, Developing and writing behavioural objectives. Educational Inno-
vators Press, 1970.
[11] A.-T. Chiang, Q. Chen, Y. Wang, and M. R. Fu, “Motion Sequence Align-
ment for A Kinect-Based In-Home Exercise System for Lymphatic Health
and Lymphedema Intervention,” in 2018 40th Annual International Con-
ference of the IEEE Engineering in Medicine and Biology Society (EMBC),
IEEE, 2018, pp. 2072–2075.