This article presents a framework for a companion robot aimed at facilitating the learning of handwriting in children with dysgraphia. The framework is composed by a Wizard of Oz (WOZ) interface, a library of social behaviors and a perception system. The WOZ interface has been realized as a web service via Flask for cross-platform use, controlling the robot through ROS. A robot behaviors library has been designed by integrating robot gestures, facial expressions, and speech. The visual perception of the robot exploited a RGB-Depth camera to recognize humans in terms of their skeleton. The presented framework is a prototype that will be evaluated in the field during long-term child robot interactions. This framework is represents a basic building block in the development of a semi-autonomous system for the training of handwriting and fine motor skills.
Writing is an essential skill. Behind speech, it is the main means of describing things and expressing emotions. For most people, learning to write seems like a natural thing, but some children have real dificulties because of developmental coordination disorders (DCD).
DCD formaly called Dyspraxia[
To further study how assistive robotics can positively influence writing training in dysgraphic
children[
In a typical scenario (Fig 1), the robot stands with its fake tablet, together with a caregiver, the
child and her/his professional tablet, able to capture handwriting features. After the selection
by the caregiver of a handwriting activity, the robot will pretend to write on its fake tablet
with its hand. Corresponding signs will appear on the tablet. The child will then judge and
eventually correct the robot’s handwriting by writing next to it, providing a new sample. While
the robot learns from such writing examples, in the presented scenario the children will lead
the teaching: their self-esteem will be enhanced as they are no longer the ones being criticized.
Moreover, in order to improve the robot’s handwriting skills, the children should provide better
samples and, while doing so, they improves, too. In accord with this hypothesis, by interacting
with a non-judgmental robot having “writing dificulties”, children will feel responsible for
its improvements and its failures, motivating them on teaching, pushing them on providing
better samples and, ultimately, in training their own handwriting skills[
This article focuses on the development of: (1) a library of simple social behaviors as basic building blocks for the social interactions of a CoWriter robot; (2) a Wizard-of-Oz interface ofering the real-time control of the robot; (3) an embedded perception system aimed at capturing and logging children behaviors. The proposed system will have the double goal of experimenting and evaluating the behaviors of the robot while interacting with the children as well as providing caregivers with a robust, non-autonomous but ready-to-use handwriting training robotic system.
Luxai’s QTRobot (Fig 1) was chosen as preferred social robot platform, in the hope of having
expressive abilities in terms of social cues and, above all, broader facial expressions and better
stability[
To improve the quality of engagement in long-term child robot interactions[
In joint work with expert clinicians from the developmental psychiatry service from PitiéSalpêtrière medical hospital in Paris, we have established a library containing 78 behaviors grouped into 17 categories. These categories are divided into two tabs: the scenario and game tab and the reaction tab. In any case, to reduce alienating, repetitive behaviors and increase the robot’s vitality, we have included several sentences with similar meanings for the same behavior. The program selects one at random when running.
Perceiving and understanding the behaviors of interacting children is an essential skill that a socially aware autonomous robot should have. At the same time, capturing and exploiting in online or ofline analysis the behaviors of children while interacting can also refine and improve the interactive abilities of both the robot, by evaluating its behaviors, and the caregiver, by improving the social interplay. This can be done through the Nuitrack software, a 3D tracking middleware, a solution for skeleton tracking and gesture recognition. While integrated into the QTRobot APIs, through ROS messages, it was not integrated with the transformation management (TF) package. Consequently, Nuitrack data, including translations and rotation angles of 19 skeleton joints1 for each person perceived relative to the camera, were exploited and translated in terms of TFs transforms: each TF maintained the relationship between the perceived coordinates and the robot frame, unifiying them into a single Cartesian space.
To test the behaviors of the robot in real use-case experiences, a Wizard-of-Oz controller
has been built. The methodology behind a WoZ consists in the replacement of any automated
decision algorithm from the robot’s controller with the manual selection of the robot’s behavior.
During the interaction, the human ”wizard” will remotely choose through a graphical interface
the behavior that fits at its best the current state of the robot, of its surroundings and of the
current interaction with its human partners[
1Nuitrack skeleton: https://download.3divi.com/Nuitrack/doc/structtdv_1_1nuitrack_1_1Skeleton.html
The Figure 2 shows some screens from the implemented interfaces. When the interface opens, a login page asking the user to enter the child’s name. In the debug interface, buttons for each behavior will appear. Behaviors are divided into tabs, depending on the use case, and into diferent categories. The Caregiver interface Figure 2 is divided into two tabs: the games tab and the reaction tab. The games tab contains categories of behaviors that refer to the gaming scenario and on its advancement, as its beginning, its ending, its instructions2; the reaction tab contains the behaviors of the robot classified into categories identified by images. A double click on the category name opens a menu with the correspondents robot’s behaviors. A behavior can be sent to the robot and executed by a further click on its name.
The Figure 3 shows us a comparison between the actual posture of a person interacting with the robot in the selected scenario and his skeleton as perceived by the robot’s 3D camera. The proposed perception system is able to track in real-time the humans in the surroundings interacting with the robot. Notably, people data is synchronized with the robot’s behavioral data. Figure on the right shows in particular two groups of markers: on the left the markers corresponding to the skeleton of the perceived person; on the right, the kinematic structure of the robot. Such synchronized data can be registered as a log and exploited ofline for characterizing the social interplay between the robot, the child and the caregiver, as well as to track down correlations and causality events between them.
2More on the tablet writing games: http://dynamico.ch/
The presented framework is ready to be employed by clinicians of Pitié-Salpêtrière medical
hospital in handwriting training sessions of children with DCD. The developed behaviors as
well as the WoZ interface will assure to caregivers the possibility of setting up efective training
interventions based on the learning by teaching paradigm. On the other hand, the developed
perception system allows the perception and the log of children behavioral data for further,
in-deeper, ofline or online analysis. We will focus, in particular, on engagement metrics that will
be employed as practical measure to evaluate the efectiveness of the system. At the same time,
the caregiver experience will be evaluated through usability tests like SUS or AttrakDif[