=Paper=
{{Paper
|id=Vol-3099/paper3
|storemode=property
|title=Use of Music to Enable the Social Inclusion of Deaf Children Using Robotics and Recreational Activities
|pdfUrl=https://ceur-ws.org/Vol-3099/paper3.pdf
|volume=Vol-3099
|authors=Cristiano da Silva Benites,Ismar Frango Silveira
}}
==Use of Music to Enable the Social Inclusion of Deaf Children Using Robotics and Recreational Activities==
https://ceur-ws.org/Vol-3099/paper3.pdf
Use of Music to Enable the Social Inclusion of Deaf
Children Using Robotics and Recreational Activities
Cristiano da Silva Benites1 [0000-0002-1300-7944], Ismar Frango Silveira1 [0000-0001-8029-072X]
1
Faculty of Computing and Informatics, Mackenzie Presbyterian University, Brazil-SP
benites_silva@hotmail.com, ismarfrango@gmail.com
Abstract. This project demonstrates challenges encountered in the inclusion of
children with hearing impairment in music learning and presents how society
sees these children, seeks to know how deaf people see music, presents reports
of deaf people who have had contact with music and how it has helped them in
society and seeks to evaluate visual elements combined with robotics in the
practical teaching performed with deaf children. It is envisioned with this work
to present how technology can be a differential in the practice of music teaching
and an aid to generate social inclusion of deaf children, allowing their contact
with hearing people in all places with audible events.
Keywords: hearing impaired children, deaf community and culture, colorful
musical elements, deaf aid technology applications, robotics in music.
1 Introduction
At different moments in history and school policy in Brazil, music education was
present in the context, always marked by difficulties, the process of building music
teaching was marked by document elaborations and actions that were the result of
struggles to insert it in schools. Thus, higher education institutions created undergrad-
uate courses in Artistic Education by legal determination, thus working the teaching
of visual, performing, and musical arts. On the other hand, the use of these different
languages as consequence generated the weakening of music education in schools,
and over time was replaced by only fine arts, which contributed to the disappearance
of music as cited [1].
In view of this [2] believes that music should not only be worked as a discipline, but
be worked in a thorough way by professionals with solid knowledge of it to pass a
solid knowledge to those who study it, she says this because it analyzes that teachers
without musical training summarize in practical musical foundations only activities
such as singing or musical listening with basic information. Thus it is difficult to im-
agine that teachers with the ability to teach early beings would have conception of
music teaching equally to a teacher prepared and licensed for the teaching of music,
because they obtained in their curriculum, the theoretical, practical way, the expres-
sive form of music teaching, and finally one of the most relevant items, acquired
techniques to be able to teach the mastery of musical instruments to children as men-
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).
1
2
tioned [3], the self still defends how important music is being taught in schools for
better cognitive development of children.
The author [3] also proposed some questions to be analyzed in this world of music
education in schools, one of them is to analyze the fact that a teacher not licensed in
music guiding children, the error inserted in this medium also, of teachers not gradu-
ated in music, teach this specific subject in schools. Another factor that is also rele-
vant is the consideration of equality between teachers with musical training or not
according to law 11,769 in 2008. Thus, it is intended to seek new alternatives that can
help meet these needs in schools in the musical sphere, for this it is necessary to ex-
plore the technological means to be as an aid of music learning in schools, in addition
there are cases of children with very specific needs, such as hearing impairment,
which would be another case study factor for music teaching.
Currently in Brazil there are different types of deficiencies, and it can be auditory that
is the theme, or visual, motor, with its degrees of severity. In the case of auditory,
according to census data, it reached the third position, lagging the visual, and motor
impairment, which occupies the first general position [3].
This research sought to delimit the theme, comprising a search for bibliographic ref-
erences in relation to the theme, thus we were aware of several articles and books
among which we highlight an article of [4] with the name "Deafness and Music: is
this a paradox?" where this same author also has a doctoral thesis with the theme
"Teaching Music to deaf students: perspectives for inclusive pedagogical action",
among the various books analyzed we find the book "The musicality of the deaf: rep-
resentation and stigma" of the researcher Nadir Haguiara-Cervellini, we had other
articles and books used as a theoretical basis, but all the others were musical currents
and others pointed to libras learning.
The formation of deaf individuals has broken many paradigms that until a few years
ago represented unique models, and has allowed deaf people to have new opportuni-
ties, such as going to a cinema, to a park, to school and being able to interact simply,
and being able to learn music and feel it like other people.
2 Use of colors in music to aid deaf people
For several years, researchers tried to relate musical notes with colors and even in the
old east these relationships were made between these two [5]. We will show below a
table where the relationship made by several researchers during the centuries that
sought to idealize this relationship between music and colors is presented in Fig. 1:
Fig 1. Three centuries of color scales
2
3
It is possible to analyze in this table that the relationships do not follow the same pat-
tern between colors and intervals of musical notes, because there were many personal
perceptions among the researchers, which means that there was only meaning for
those who defined them. For this reason, there is a need for a standardization between
music and colors, so that the learning process can be carried out through audiovisuali-
ty and show efficiency.
[6] is also used for a relationship between colors and musical notes with purpose in
the frequencies of sounds and frequency of colors. With this thought of analyzing the
low colors (430x1012) which in this case represents the red with darker tint and mak-
ing use of the concept of tuning (There 440 Hertz), the researcher noticed that the
tone that has the closest relationship would be the SUN that is below the A 392 Hertz.
The comparison is perfect, but it is concluded that it is not all frequencies that are
equivalent in tones due to the temperate nature of the tuning in the west.
In Fig. 2 we show the Law of the Octaves/Periodic Law, which deal with the under-
lying ratio of the vibration scale and thus, therefore, is the basis for all theories that
relate to the relationships between colors and musical notes (the repetition of the notes
on the board means the sharp "#") [6].
Fig. 2. Sound and Color Frequencies
This research becomes interesting because of the musical notes and colors in the mu-
sic are constituted of frequencies, but there is a problem that the musical note is al-
ways a single color or vice versa, so the relationship that the musical note has with a
color would have a lot of variation, and every moment would have other colors.
A researcher in the Arts department of PUC-RJ made a relationship between music
and colors, in the case of Celso Braga Wilmer (1948). The same has training in visual
communication, mathematics, and music, of which were the researcher developed a
project to facilitate the learning of reading sheet music as quoted [6].
Wilmer developed a work focused on the musical pedagogical form, focused on the
understanding of music through sheet music, so he named his work "Rainbow Score".
The aim of this study was to adapt traditional cores to provide a form of ease to music
students with established musical symbols, where Wilmer saw themes [7]. abstracts.
Based on the levels of abstraction of learning of Jean Piaget (1896-1980), the project
sought to insert the score already existing within a presentation of the chromatic mod-
3
4
el that he himself adopted, applying the scales and shades the largest and smallest in
its duration, harmonic and melodic, etc. Thus, after the application of Wilmer's pro-
ject are the degrees of the musical scales that receive the colors and cease to be the
tones, as a kind of figure on a background [6]. You can see below how this relation-
ship works in the Fig. 3.
Fig. 3. Harmonic functions with colors.
Still dealing with harmonic functions, the researcher Wilmer made a relationship
between musical intervals and colors very different from previous researchers, and so
many others of today, the work proposed by the same is somewhat effective, and this
can be seen as a great principle for analyzing music using colors, is considered rela-
tional character what notes and colors can have in a given music.
Another project that deserves to be mentioned is the "CO. Diesis" which was con-
ceived by the Balearic Islands University in Spain, where it was developed thinking
of blind people and works the relationship of colors with sound, where color becomes
sound[7]. This work was a pathway for the software "CromoTMusic" that also stands
out in this concept.
The principle of work is to transform sounds into colors for blind people, already in
the background one has the idea of working on the transformation of musical notes
into colors to help deaf people in contact with music. At the point of view of audio-
visuality the circumstance cited is very important because we speak of two deficien-
cies (hearing and vision) that are treated in audiovisuality one of them deafness can be
compensated through colors, in the case of contact with music.
3 Methodology
In this project we present to the child's deaf fundamentals of music, such as musical
notes, rhythms and tempo of the compass, coming out of the traditional way that
makes use only of a book, a blackboard or any other medium other than with the use
of technology, in this way will be made the use of the following materials:
• Mbot robot adapted with a single LED that supports 7 colors / bluetooth.
4
5
• USB vibratory bracelet could not find a bluetooth bracelet.
• Grand piano Roland,
• iPad-Pro,
• Python language to configure the robot.
• Editom for demonstration (SOL clave).
To achieve the goal in the experiments we will work with a robot integrated with
Scratch that is a programming language composed of blocks created by Papert in
2008, and more specifically use it adapted in a version of Makeblock, so we will have
the possibility to control the robot and its resources through a mobile device, this can
be a mobile phone, tablet or iPad, where children will make combinations in the ap-
plication.
Before starting any activity with the robot, we will teach the concepts of the project so
that children can understand what each item means during the practice of recreational
activities, because none of the children has musical knowledge, this in the theoretical
and practical scope. In this way we will teach the combinations between colors and
musical notes proposed by Wilmer, and musical concept based on BONA which is a
book intended for teaching musical concepts.
After the children know the musical notes combined with colors and have a basis of
what each color means in the musical pentagram we will move on to the second part
of the experiments that concerns rhythmic time, we will insert the concept of musical
time in the activities. The child will know the rhythmic time equivalences used in
music using the robot's movements.
A group of 7 children aged between 5 and 9 years was selected to do the musical ex-
periment using robotics and playful activities with applications. Among these children
6 have hearing problems and need interpreters to help them in communication and 1
of them listens normally and will be a model for the other children when performing
the activities. Throughout the process there was consent of their guardians and we
were assisted by 4 monitors who were psychologists and pedagogues who accompa-
nied the children during the moments we met and participated in the experience as
evaluators.
Table 1 shows the communication situation of each of the 7 children who participated
in the experiments, where the symbol "X" indicates positive, this information was
proven during the experiments.
Table 1. Communication situation of each child.
Participant Reading LIBRAS Degree of Deafness
and Age situation Situation Moderate Severe Deep
Child 1 - 9 years X
Child 2 - 6 years X
Child 3 - 6 years X
Child 4 - 7 years X
Child 5 - 7 years X
Child 6 - 8 years X X X
Child 7 - 9 years X X X
5
6
The first contact with the children lasted about 4 hours, for them to become familiar
with the proposed scenario and feel at ease it was necessary to do an activity to gener-
ate entertainment, so we did an activity with the use of balloons representing the col-
ors of the musical notes that we would use in the application next. So, after the activi-
ties with balloons we set up the robot together with the kids as a mode of fun.
In the first stage of the experiments the child listener began to make the first contact
with the robot and the other children followed what was done because then the other
children would perform the same activity. Thus, the child listener practiced musical
concepts.
In the second contact that was a following Sunday it was expected that it would be
necessary to remind the children of the colors of the notes and the times that were
made the movements of the robot because it had been 7 days, but it was not necessary
to do this activity again because the children still remember easily what was proposed
in the first activity, this was verified with a simple activity similar to that of the bal-
loons that was performed previously, and through tests with the robot before the ex-
periment of that day.
We started this time by the youngest child to verify that she still had the theoretical
concept in mind, even because there was little time spent on the day of the first exper-
iment and so could perform any musical activity similar to another one with little
difficulty, this because she imitates the robot during the preparation of the same be-
fore the activity that day , which was proven in the participation of the child, she per-
formed the activities without problems.
After the youngest child finished his participation it was the turn of the hearing child
to take over the post and do the activities proposed on that day and again be an exam-
ple for the other children, because soon after all of them they would do the same ac-
tivity, as we performed in the first experiment.
In order to generate entertainment and the children could have a sense of the experi-
ments for this stage were defined new stages of activities using the application, at that
moment we made the use of the song called "Slaves of Job", where each stage of the
game in the application in this experiment taught an excerpt of the music and a
movement with the musical notes with colors , the steps would end at the time it was
played entirely by each child. Just as in the previous activity the other children mim-
icked the robot's movements and raised their hands for each note presented by the
robot.
For the third stage we made the use of the robot that was already used from the first
stage, but unlike the other occasions the children would not command the robot via
mobile device, but rather the robot would be an indicator that a musical note would be
generated from a piano, in case the children pressed anote musical on the piano and
the robot presented the color of the note executed on it.
As in the first two experiments the children had no contact with a physical piano, this
could generate difficulty for them, so we inserted in this stage of the project a USB
bracelet integrated into the robot, so when the children played a musical note on the
piano it would vibrate by the fact that it was acoustic, the robot would light the LED
indicating the note that was played on the piano and the bracelet would be integrated
into the robot generating a vibration in the children's wrist.
6
7
We programmed the robot in advance in order to just get to the place and connect it to
the piano and get satisfactory response with the notes pressed on the piano keys, so
we connected the robot's USB input to the piano. At the time of the experiment of this
stage the child should have his left hand on the piano, because it is the hand with the
bracelet, she also felt the vibration of the piano, all this occurred in a synchronized
way and we sought not to have delays, so as not to generate confusion in the chil-
dren's heads.
Fig. 4 shows the final scheme of the experiment being performed by one of the chil-
dren during the third experiment.
Fig 4. Child playing piano with all the built-in features.
In this third stage we intended to make the children play the same music used in activ-
ity 2 when they used the digital piano on the mobile device, which occurred without
difficulty. Also as was proposed earlier in the other activities children who were not
in contact with the piano should imitate the movements that the robot generated on
top of the piano and raise their hands when the notes were performed, an activity that
all children performed several times.
All experiments were carried out with the same children, we had the first step we
called as experiment 1, the second stage experiment 2 and the third stage of experi-
ment 3.
The activities we developed at all stages explored how children would deal with the
concept of colors combined with musical notes, contact with electronic devices and
musical rhythm, and evaluate visual and rhythmic issues performed by children when
imitating the robot.
All the children performed the same activities in the same period, so it would be pos-
sible to know what was the abstraction in relation to the period of participation of all
children, knowing where each one managed to get in the application and the time it
took.
7
8
At all stages the children were evaluated by monitors present at the site, one of the
expected specifications would be to know if it was possible for the child to have kept
in mind the concept of the activities that were performed previously, and be able to
perform the musical concepts, and then know if the younger children could perform
the activities more quickly because they had more examples performed before their
participation in the execution.
Below is the chart showing Fig. 5 in general how was the performance in children's
notes in all activities, so it is better to have a view with the general notes.
Fig. 5. Performance overview in experiments.
According to the results presented in the experiments so far, it can be analyzed that the child
was able to learn the musical concepts due to the repetitions of theory and practice, and another
relevant factor was the child to see the other children practicing when on their opportunity with
the devices, and also with the help of the robot to generate the movements and present the col-
ors as an aider was a differential for the result obtained , so know that something happened
while performing a musical note on the iPad.
4 Final considerations
Seeking to see beyond sound, involving different methods to help children in their interpreta-
tion of music as use of colors, we also sought to present through the literature review that there
is no neurological obstacle that can compromise an individual to have a musical experience,
such that the child aims to be able to make use of images and vibrations of devices , thus these
items gave us a basis to be able to make a planning of how to achieve the goal allowing to
insert applications and hardware in experiments with hearing impaired children.
8
9
As soon as we carried out the musical experiments with the hearing impaired children, it was
possible to analyze that the architecture used allowed the children to perform the musical activi-
ty, thus assisting in the perception of the sounds that happened and that were performed with
the compass, vibration and rhythms through the robot.
With this it was possible to reflect on what benefits music played in children through their
attitudes in the activities, and how they gave us a positive return with their participation in the
activities and also with the feedback we had of them at the end of the activities, the comments
of the advisors at the time of the activities, because they were doing the evaluation all the time
of the children while they were performing the activities , thus presenting the practical concept
that it is possible and pleasurable to teach music to deaf children.
Based on the evaluations and opinions of the advisors who were all pedagogues, it was possible
to realize in general that the use of architecture comes to be a great way to be used in the musi-
cal activities of deaf children, even for reasons of values that are accessible, and also easy to
handle by people even if they are lay people in the subject, with extensive documentation on
the WEB the solution used here can be replicated easily for people who wanted to accomplish
the same technological context with the documentation available.
The activities performed in this research indicate that the junction that can help provide music
learning for hearing impaired children. It presented potential for musical activities and can be
adapted to other situations of different visions, thus being able to teach other points in music
theory and children learn to have fun.
Thus, it is essential to make use of adaptive technological means in the education of deaf chil-
dren, verify the effectiveness of the application and hardware, study ways and know what the
real impact caused in those who participate in learning.
References
1. Alvarenga, C. H.; Mazzotti, Tarso B. Musical education and legislation: reflections on the
veto to specific training in Law. Opus, Porto Alegre, (2011) 51-72
2. Antunes, S. F. Free robotics as a didactic alternative to music learning. 2016. Dissertation
(Master's degree in Education) - University of Passo Fundo, Passo Fundo, (2016)
3. Finck, R. Teaching music to deaf students: perspectives for inclusive pedagogical action.
Thesis (Doctorate in Education), Federal University of Rio Grande do Sul, Porto Alegre:
(2009)
4. Gobbi, V. A esthetic education through musical appreciation: an experience. 1999. Dis-
sertation (master’s in education) - University of Passo Fundo, Passo Fundo, (1999)
5. Leinig, C. E. Music and Science meet: an integrated study between music, science, and
music therapy. Curitiba: Juruá, (2008)
6. Ortega, I. The Colors of Sound. (Graduation in Music Therapy). Music therapy, Fac-
uldade Paulista de Artes, São Paulo. (2009) 67- 68
7. Ortega, I.; Wilmer, C.; Gattino, G. S. The use of colored score in the evaluation process
in music therapy. Brazilian Symposium on Music Therapy. XIV, A. D. Olinda: Associa-
tion of Music Therapy of the Northeast (2012)
8. Wilmer, C. Pontifical University Catolic - RJ. PUC Science, (1989)
9