=Paper=
{{Paper
|id=Vol-3070/paper01
|storemode=property
|title=Stimulating Deaf Children Through Vibration and Playful Activities for Musical Experience
|pdfUrl=https://ceur-ws.org/Vol-3070/paper01.pdf
|volume=Vol-3070
|authors=Cristiano da Silva Benites,Ismar Frango Silveira
}}
==Stimulating Deaf Children Through Vibration and Playful Activities for Musical Experience==
https://ceur-ws.org/Vol-3070/paper01.pdf
Stimulating deaf children through vibration and playful
activities for musical experience.
Cristiano da Silva Benites1 [0000-0002-1300-7944], Ismar Frango Silveira2 [0000-0001-8029-072X]
Faculty of Computing and Informatics, Mackenzie Presbyterian University, Brazil-SP
benites_silva@hotmail.com 1
ismarfrango@gmail.com 2
Abstract. This paper presents reports of children with deafness who had contact
with music through playful activities combined with robotics and other
applications, here we also demonstrate how it helped them to evaluate visual
elements and allowed interaction with music. It is detailed every procedure that
was performed in the 3 experiments and all the steps to achieve the goal that was
children to have practical contact with music theory with a totally digital mode
without the use of books. This work aimed to present how technology can be a
differential in the practice of music teaching and an aid to generate social
inclusion of deaf children allowing their contact with hearing people in all places
with audible events.
Keywords: deaf children, musical elements with colors, technological
applications in music, robotics in music, playful activities.
1 Introduction
In the present day there is still prejudice towards the deaf and their musicality [1]
conducted a research between which relationships there are between music and the deaf
person learning music, in which it was possible to verify that some of them hate music,
but in return, there are deaf people who love it. There are deaf people who understand
music, but have deaf people who have not the slightest interest in understanding it.
There are deaf people who when they feel the music are moved, and there are others
who can feel it in different ways. There are deaf people with great conditions to delight
in music, but there are already deaf people who will hardly have an experience of
feeling the musical pleasure when seeing a play or other event.
Music provokes in listeners emotions in circumstances difficult to explain, emotions
that can be felt by deaf people, but if you are going to use music as support for deaf
subjects it can achieve even other goals, such as improving speech, something that the
deaf may know, information that the same must know to keep in mind that this not only
include the same, but bring several other benefits. The mechanical form of music used
in therapy sessions, instrumental presentations, sometimes minimizes the interest that
deaf people have to learn music, and at first the desire of the deaf is to know the music
that is a right of all, and the deaf are also included in this amount of people, but it is up
to professionals in the area to show the dynamism of it, so that it can convince them
and even enchant them [2].
Copyright © 2021 for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
2
There are deaf musicians who are already from the artistic environment, or act in
cinemas, soap operas, others are writers, among others, they have a coexistence
between borders, stand between two worlds, the listening community, and the deaf
community. The inclusion of interculturality in the context of music for deaf people
reaches new dimensions not previously seen, overcoming challenges, thus the musical
need among these cultures cited by the author is essential, so that it is possible to
recognize it and, historical situations and events, adding social and educational values
[2].
It would be impossible to speak of the deaf in the middle of music and not to remember
Ludwig Van Beethoven a German who was a musician and composer, who had an
experience as a listener until the year 1796, from then on began to feel the first
symptoms of deafness, although until that time he had already made several attempts
to treat deafness, it only increased. The ninth symphony, one of the most recognized
works in the musical world, Beethoven composed at this time, at the end of his life he
was already completely deaf.
Beethoven with hearing loss in adulthood did not have the ability to capture new
sounds, but until then he had enough knowledge to comwrite new symphonies in his
brain and create his scores, because the fact of not being born deaf allowed Beethoven
enough knowledge and technique to commake in his mind.
2 The deaf and the vibrations
Although they do not listen, deaf people can feel sounds and adapt as an experiment
conducted by [3], where deaf individuals could feel music throughvibrations, their
perceptions of vibrations have real sonic equivalence by being processed in the same
place in the brain; the author even addressed a study that helps in explaining deaf people
being at concerts and musical events.
The author performed functional MAGNETIC RESONANCE scans on the brains of
ten deaf people with a profound degree of loss, they were volunteers, and they had
eleven more people who were hearing. The two groups of people had activities in the
brain region, where it processes vibrations. In addition, all deaf volunteers provided
activities in the auditory cortex, where it is active only during auditory stimulation.
According to the author, these data indicate that there was still much to be understood
in the perception of these deaf people, because the fact of having activated the cortex
the region comes into action, when other people hear some kind of sound.
The author also states that people with deafness can enjoy music in the same way as
people who listen normally, which is the explanation because many of them like music
and still by fine make up. Doctor Shibata reports that the discovery brings us strong
argument for deaf children to be placed on exposure to music when they are still young,
and further concludes that this experience of the deaf is something like that of the
hearing people in relation to music, the perception of musical vibrations by the deaf is
intense when the sound is processed in the same place of the brain , with this statement
related to vibration, it was seen characteristics that sound information, with this, for the
other deaf, one modality can replace the other in the same function in the brain.
As the author proved, the deaf have a locality in the brain active for musical purposes,
the information processed in this region, is organized according to vibrations, which is
3
understood in the musical brain. Understanding better what has been reported by [3],
[4] reinforces that musical activity mobilizes all regions of the brain, and [5] presents a
figure that facilitates the understanding of what Levitin explained.
Fig. 1 - "Musical Brain".
Chapter 3 addresses technological aspects used to assist deaf people in music
in different contexts and some examples of existing devices
3 Combination of colors in music to assist deaf people
For several years, researchers tried to relate musical notes to colors and even in the old
east these relationships were made between these two. We will show below a table
where the relationship made by several researchers during the centuries that sought to
idealize this relationship between music and colors is presented:
Table 1: three centuries of color scales
4
Table 1 shows that relationships do not follow the same pattern between colors and
intervals of musical notes, because there were many personal perceptions among the
researchers, which means that there was only meaning for those who defined them. For
this reason, there is a need for a standardization between music and colors, so that the
learning process can be carried out through audiovisuality and show efficiency.
[7] is also used for a relationship between colors and musical notes with purpose in the
frequencies of sounds and frequency of colors. With this thought of analyzing the low
colors (430x1012) which in this case represents the red with darker tint and making use
of the concept of tuning (There 440 Hertz), the researcher noticed that the tone that has
the closest relationship would be the SUN that is below the A 392 Hertz. The
comparison is perfect, but it is concluded that it is not all frequencies that are equivalent
in tones due to the temperate nature of the tuning in the west.
In Table 2 we show the Law of the Octaves/Periodic Law, which deal with
the underlying ratio of the vibration scale and thus, therefore, is the basis for all theories
that relate to the relationships between colors and musical notes (the repetition of the
notes on the board means the sharp "#") [7].
Table 2: Sound and Color Frequencies
This research becomes interesting because of the musical notes and colors in the music
are constituted of frequencies, but there is a problem that the musical note is always a
single color or vice versa, so the relationship that the musical note has with a color
would have a lot of variation, and every moment would have other colors.
A researcher in the Arts department of PUC-RJ made a relationship between music and
colors, in the case of Celso Braga Wilmer. The same has a degree in visual
communication, mathematics and music, so the researcher developed a project to
facilitate the learning of sheet music reading as cited (Ortega, 2009).
5
Wilmer developed a work focused on the musical pedagogical form, focused on the
understanding of music through sheet music, so he named his work "Rainbow Score".
The aim of this study was to adapt traditional scores to offer a form of ease to music
students with the established musical symbols, where Wilmer saw them as abstract [9].
Based on the levels of abstraction of learning of Jean Piaget (1896-1980), Wilmer's
project sought to insert the score already existing within a presentation of the chromatic
model that he himself adopted, applying the scales and shades to the largest and smallest
in its duration, harmonic and melodic, etc. Thus after the application of Wilmer's project
are the degrees of the musical scales that receive the colors and cease to be the tones,
as a kind of figure on a background [6].
4 Methodology
In this project we present music for 6 deaf children and a seventh child who had normal
hearing that would be a model for the other children to perform the activities, we studied
the fundamentals of music together with an expert and a pedagogue,such asmusical
notes, rhythms and times of the compass, leaving the traditional way that makes use
only of a book, a blackboard or any other medium other than with the use of technology,
in this way will be made the use of the following materials: robot Mbot, USB vibratory
bracelet, Grand piano Roland, Ipad-Pro e Editom for demonstration.
We present in the table below the communication situation and each child, taking into
account whether they could read, knew LIBRAS (Brazilian Sign Language) and how
deaf:
Table 3. Communication situation of each child.
Reading LIBRAS Degree of
Participant and Age ✔
situation Situation Deafness
Child 1 - 9 years - ✔ -
✔
Child 2 - 6 years - -
✔
Child 3 - 6 years - -
✔
Child 4 - 7 years - -
✔ ✔ ✔
Child 5 - 7 years - -
✔ ✔ ✔
Child 6 - 8 years
Child 7 - 9 years
To achieve the goal in the experiments we will work with a robot integrated with
Scratch that is a programming language composed of blocks created by Papert, and
more specifically use it adapted in a version of MAkeblock, so we will have the
possibility to control the robot and its resources through a mobile device, which can be
a mobile phone, tablet or iPad , where children will make combinations in the
application to send commands to the robot, thus performing musical concepts that we
will detail in the following sections.
Before starting any activity with the robot, we will teach the concepts of the project so
that children can understand what each item means during the practice of the activities,
because none of the children has musical knowledge, this in the theoretical and practical
scope. In this way we will teach the combinations between colors and musical notes
proposed by [8] that was explained in section 3 of this project, and then use a practical
6
script where we will use a musical concept based on BONA that is a book intended for
teaching musical concepts, and the same will be combined with the Editon software to
have a musical pentagram with colored notes.
Fig. 2 - Pentagram Editom with colored musical notes.
After the children know the musical notes combined with colors and have a basis of
what each color in the pentagram means we will move on to the second part of the
experiments that concerns rhythmic time, we will insert the concept of musical time in
the activities. The child will know the equivalences of rhythmic time used in music
using the movements of the robot, so we will use periods of initial times in case those
that have a greater slowness , even because they are slow the robot will move to the
sides in a soft way, so as not to generate confusion for children if we use too much
speed in the movements.
After the children fix the initial concepts we will start practicing colors with the robot,
where each musical note that is played on the digital piano on the mobile device will
have a color representing a musical note, which will go to the robot running an LED
combined with a robot movement, so the child will begin to know which color belongs
to which musical note performed.
For the last step we made the use of the robot that was already used from the first stage,
but unlike the other occasions the children would not command the robot via mobile
device, but rather the robot would be an indicator that a musical note would be being
generated from a piano, in case the children pressed a note music on the piano and the
robot presented the color of the note performed on it
Based on the children's understanding of the theoretical concept of everything that was
taught and managed to get through the previous musical activities of basic theory, we
then set out for the practical part with a real piano. For the realization of this final
activity we used a roland grand piano, in the case of the model "Gp607 Pel", this piano
has USB and P10 inputs, in this case the USB input where we integrated the robot to
the piano by the USB input so that it was possible for the two to act as if they were only
one.
After the whole proposal was presented, we started for the initial test where we
connected the bracelet in the robot and validated the operation of the robot to know if
it would have time accuracy in the vibration of the robot and the bracelet. We
7
programmed the robot in advance in order to just arrive at the site and connect it to the
piano and get satisfactory response with the notes pressed on the piano keys, so we
connected the robot's USB input to the piano, connected it, after that we clicked on a C
note that was represented by the red color in the previous tests and so the robot was in
the red color as we predicted , and so too the bracelet vibrated with all the musical notes
pressed
At the time of the experiment of this stage the child should have his left hand on the
piano, because it is the hand with the bracelet, she also felt the vibration of the piano,
all this occurred in a synchronized way and we sought not to have delays, so as not to
generate confusion in the children's heads.
In Figure 3 we have the final schema of the experiment being performed by one of the
children during the third experiment.
Fig. 3. Child playing piano with all integrated features
In this third stage we intended to make the children play the same music used in activity
2 when they used the digital piano on the mobile device, which occurred without
difficulty. Also as was proposed earlier in the other activities children who were not in
contact with the piano should imitate the movements that the robot generated on top of
the piano and raise their hands when the notes were performed, an activity that all
children performed several times.
All experiments were carried out with the same children, we had the first step we called
as experiment 1, the second stage experiment 2 and the third stage of experiment 3.
The activities we developed at all stages explored:
1- how children would deal with the concept of colors combined with musical
notes, using electronic devices.
8
2- evaluate visual and rhythm issues performed by children when imitating the
robot.
All the children performed the same activities in the same period, so it would be
possible to know what was the abstraction in relation to the period of participation of
all children, knowing where each one managed to get in the application and the time it
took.
In table 5 we present the results obtained in the first activity with the participation of
the children, we used the data collected in the evaluations of the monitors. To facilitate
the evaluation and classification for the monitors we used 10 questions with 10 answers,
in this case grades 1 to 10 in each question, thus how much "lower" the grade of the
higher child was their difficulty in the activity.
Below we present the general notes of all the children in their participations in
experiment 1 following the questions, as given by the monitors.
Table 4. Experiment notes 1.
Children 1 Children 2 Children 3 Children 4 Children 5 Children 6 Children 7
10 8 8 8 10 10 10
10 8 8 8 9 8 9
10 8 8 8 10 9 8
10 9 9 8 9 9 9
10 10 8 10 10 10 10
9 9 9 9 9 9 9
10 10 10 10 10 10 10
9 9 9 9 9 9 10
10 8 8 8 8 10 8
9 9 9 9 10 9 9
We can notice that in activity 1, children 2, 3 and 4 had a little difficulty in relation
to other children, this was because they could not read and do not know pounds, which
generated a little difficulty in communication, but after a second attempt they were able
to perform the activity completely.
In the case of activity 2 we made the sequence of participation different in relation to
the first, in the case of activity 1 started the child 1 and then started the children who
could not read and were less than old, in activity 2 we started with child 1 again to be a
model for the other children, and soon after we started by the children who could read
to draw conclusions.
One of the expected specifications would be whether it was possible for the child to
have kept in mind the concept of the first activity that was performed a week before
activity 2, and be able to perform the musical concepts, and then know if the younger
children could perform the activities more quickly by having more examples performed
before their participation in the execution. Below we present the graph of the second
analysis.
9
Table 5. Notes from experiment 2.
Children 1 Children 2 Children 3 Children 4 Children 5 Children 6 Children 7
10 10 10 9 10 10 10
10 9 9 9 9 9 8
10 10 9 9 10 8 9
10 9 10 10 9 9 9
10 10 10 9 10 10 10
10 9 9 9 9 9 9
10 10 10 10 10 10 10
9 9 9 9 9 10 9
10 10 10 10 8 8 10
10 9 9 10 10 9 9
It was noted in table 2 that not necessarily the deaf child knowing how to read it will
prevail in efficiency in the activities in relation to other deaf children who cannot read,
it was noted that the musical concept was absorbed equally among them. When the
younger children could not read they began their contact with the game they performed
the initial activities with the interpreters indicating the drawings, but after a certain
point they began to play alone having ease in understanding the images.
At this stage few children presented difficulty in doing the musical activity, according
to the evaluation of the interpreters, this may have been due to lack of affinity with
technology, but item that over time can change. In activity 1 only children who could
not read and did not know pounds had difficulties to start in the musical concept with
the iPad and the robot, but in activity 2 this changed.
In activity 3 according to the data analyzed the children had very easy to perform the
proposed with the piano, had easy to identify the colors, the imitations of the robot were
assertive and managed to perform the music easily, because all of them reached the
maximum score according to the monitors, according to them the repetitions allowed
the children to feel the will , and thus they dominated the proposed activity as shown
in the chart below.
Table 6. Experiment notes 3.
Children 1 Children 2 Children 3 Children 4 Children 5 Children 6 Children 7
10 10 10 10 10 10 10
10 10 10 10 10 10 10
10 10 10 10 10 10 10
10 9 10 9 10 10 10
10 10 10 10 10 10 10
10 10 10 10 10 10 10
10 10 10 10 10 10 10
10 10 10 10 10 10 10
10 10 10 10 10 10 10
10 19 9 10 10 10 10
10
Below is the chart showing in general how was the performance in children's
notes in all activities, so it is better to have a view with the general notes.
Fig. 4. Performance overview in experiments.
According to the results presented in the experiments so far, it can be analyzed that the
child was able to learn the musical concepts due to repetitions of theory and practice,
and another relevant factor was the child to see the other children practicing when on
their opportunity with the devices, and also with the help of the robot to generate the
movements and present the colors as an aider was a differential to the result obtained,
so know that something happened when performing a musical note on the iPad.
5 Final Considerations
Seeking to see beyond sound, involving different methods to help children in their
interpretation of music as use of colors, we also sought to present through the literature
review that there is no neurological obstacle that can compromise an individual to have
a musical experience, such that the child aims to be able to make use of images and
vibrations of devices , so these items gave us a basis to be able to plan how to achieve
the goal better, thus allowing to insert applications and hardware in experiments with
hearing impaired children.
As soon as we carried out the musical experiments with the hearing impaired children,
it was possible to analyze that the architecture used with the application and the robot
allowed the children to perform the musical activity, thus assisting in the perception of
the sounds that happened and that were performed with the compass, vibration and
rhythms through the robot. After the children realized the sound through the color and
movement of the robot, they were able to reproduce and move according to the intensity
of it, when the robot generated movement and colors.
11
With this it was possible to reflect on what benefits music played in children through
their attitudes in the activities, and how they gave us a positive return with their
participation in the activities and also with the feedback we had of them at the end of
the activities, the comments of the advisors at the time of the activities, because they
were doing the evaluation all the time of the children while they were performing the
activities , thus presenting the practical concept that it is possible and pleasurable to
teach music to deaf children.
Based on the evaluations and opinions of the advisors who were all children's educators,
it was generally possible to realize that the use of scratch application with robotics
comes to be a great way to be used in the musical activities of deaf children, even for
reasons of values that are accessible and easy to handle by people even if they are lay
people in the subject , the Mbot robot used in the experiment is easily accessible
financially, with extensive documentation on the WEB and the solution used here can
be easily replicated for people who wanted to perform the same technological context
with the documentation available.
The activities carried out in this research indicate that the combination of the Scratch
application together with a hardware, in the case of this work a robot, can help provide
music learning for hearing impaired children. It presented potential for musical
activities, and can be adapted to other situations of different visions, thus being able to
teach other points in music theory and children learn to have fun
12
References
1. Sousa, A. P. Surdos listen to music. 2006. Available in: . Access In: 28 September (2021)
2. Strobel, K. Deaf: cultural traces not recorded in history. Dissertation (Master) - Federal
University of Santa Catarina, Local, (2008)
3. Shibata, D. Deaf brains adapt to feel the music. In: scientific assembly and annual meeting
of the radiology society of north america, 87., 2001, Anais. Available from:
. Accessed 25
September (2018)
4. Mossesgeld, R. Concept paste allows the Deaf to groove. 2010. Available at:
. Accessed: 28
September (2018)
5. Muszka, M. Music, neuroscience and human development. 2014. Available at:
. Access 25 September 2018.
6. Ortega, I. The Colors of Sound. Graduation in Music Therapy). Music therapy, Faculdade
Paulista de Artes, São Paulo.(2009)
7. Leinig, C. E. Music and Science meet: an integrated study between music, science and music
therapy. Curitiba: Juruá, (2008)
8. Wilmer, C. Pontifical catholic university - RJ. PUC Science, 1989.
9. Ortega, I.; Wilmer, C.; Gattino, G. S. The use of colored score in the evaluation process in
music therapy. Brazilian Symposium on Music Therapy. XIV, A. D. Olinda: Association of
Music Therapy of the Northeast (2012)