Stimulating deaf children through vibration and playful activities for musical experience. Cristiano da Silva Benites1 [0000-0002-1300-7944], Ismar Frango Silveira2 [0000-0001-8029-072X] Faculty of Computing and Informatics, Mackenzie Presbyterian University, Brazil-SP benites_silva@hotmail.com 1 ismarfrango@gmail.com 2 Abstract. This paper presents reports of children with deafness who had contact with music through playful activities combined with robotics and other applications, here we also demonstrate how it helped them to evaluate visual elements and allowed interaction with music. It is detailed every procedure that was performed in the 3 experiments and all the steps to achieve the goal that was children to have practical contact with music theory with a totally digital mode without the use of books. This work aimed to present how technology can be a differential in the practice of music teaching and an aid to generate social inclusion of deaf children allowing their contact with hearing people in all places with audible events. Keywords: deaf children, musical elements with colors, technological applications in music, robotics in music, playful activities. 1 Introduction In the present day there is still prejudice towards the deaf and their musicality [1] conducted a research between which relationships there are between music and the deaf person learning music, in which it was possible to verify that some of them hate music, but in return, there are deaf people who love it. There are deaf people who understand music, but have deaf people who have not the slightest interest in understanding it. There are deaf people who when they feel the music are moved, and there are others who can feel it in different ways. There are deaf people with great conditions to delight in music, but there are already deaf people who will hardly have an experience of feeling the musical pleasure when seeing a play or other event. Music provokes in listeners emotions in circumstances difficult to explain, emotions that can be felt by deaf people, but if you are going to use music as support for deaf subjects it can achieve even other goals, such as improving speech, something that the deaf may know, information that the same must know to keep in mind that this not only include the same, but bring several other benefits. The mechanical form of music used in therapy sessions, instrumental presentations, sometimes minimizes the interest that deaf people have to learn music, and at first the desire of the deaf is to know the music that is a right of all, and the deaf are also included in this amount of people, but it is up to professionals in the area to show the dynamism of it, so that it can convince them and even enchant them [2]. Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2 There are deaf musicians who are already from the artistic environment, or act in cinemas, soap operas, others are writers, among others, they have a coexistence between borders, stand between two worlds, the listening community, and the deaf community. The inclusion of interculturality in the context of music for deaf people reaches new dimensions not previously seen, overcoming challenges, thus the musical need among these cultures cited by the author is essential, so that it is possible to recognize it and, historical situations and events, adding social and educational values [2]. It would be impossible to speak of the deaf in the middle of music and not to remember Ludwig Van Beethoven a German who was a musician and composer, who had an experience as a listener until the year 1796, from then on began to feel the first symptoms of deafness, although until that time he had already made several attempts to treat deafness, it only increased. The ninth symphony, one of the most recognized works in the musical world, Beethoven composed at this time, at the end of his life he was already completely deaf. Beethoven with hearing loss in adulthood did not have the ability to capture new sounds, but until then he had enough knowledge to comwrite new symphonies in his brain and create his scores, because the fact of not being born deaf allowed Beethoven enough knowledge and technique to commake in his mind. 2 The deaf and the vibrations Although they do not listen, deaf people can feel sounds and adapt as an experiment conducted by [3], where deaf individuals could feel music throughvibrations, their perceptions of vibrations have real sonic equivalence by being processed in the same place in the brain; the author even addressed a study that helps in explaining deaf people being at concerts and musical events. The author performed functional MAGNETIC RESONANCE scans on the brains of ten deaf people with a profound degree of loss, they were volunteers, and they had eleven more people who were hearing. The two groups of people had activities in the brain region, where it processes vibrations. In addition, all deaf volunteers provided activities in the auditory cortex, where it is active only during auditory stimulation. According to the author, these data indicate that there was still much to be understood in the perception of these deaf people, because the fact of having activated the cortex the region comes into action, when other people hear some kind of sound. The author also states that people with deafness can enjoy music in the same way as people who listen normally, which is the explanation because many of them like music and still by fine make up. Doctor Shibata reports that the discovery brings us strong argument for deaf children to be placed on exposure to music when they are still young, and further concludes that this experience of the deaf is something like that of the hearing people in relation to music, the perception of musical vibrations by the deaf is intense when the sound is processed in the same place of the brain , with this statement related to vibration, it was seen characteristics that sound information, with this, for the other deaf, one modality can replace the other in the same function in the brain. As the author proved, the deaf have a locality in the brain active for musical purposes, the information processed in this region, is organized according to vibrations, which is 3 understood in the musical brain. Understanding better what has been reported by [3], [4] reinforces that musical activity mobilizes all regions of the brain, and [5] presents a figure that facilitates the understanding of what Levitin explained. Fig. 1 - "Musical Brain". Chapter 3 addresses technological aspects used to assist deaf people in music in different contexts and some examples of existing devices 3 Combination of colors in music to assist deaf people For several years, researchers tried to relate musical notes to colors and even in the old east these relationships were made between these two. We will show below a table where the relationship made by several researchers during the centuries that sought to idealize this relationship between music and colors is presented: Table 1: three centuries of color scales 4 Table 1 shows that relationships do not follow the same pattern between colors and intervals of musical notes, because there were many personal perceptions among the researchers, which means that there was only meaning for those who defined them. For this reason, there is a need for a standardization between music and colors, so that the learning process can be carried out through audiovisuality and show efficiency. [7] is also used for a relationship between colors and musical notes with purpose in the frequencies of sounds and frequency of colors. With this thought of analyzing the low colors (430x1012) which in this case represents the red with darker tint and making use of the concept of tuning (There 440 Hertz), the researcher noticed that the tone that has the closest relationship would be the SUN that is below the A 392 Hertz. The comparison is perfect, but it is concluded that it is not all frequencies that are equivalent in tones due to the temperate nature of the tuning in the west. In Table 2 we show the Law of the Octaves/Periodic Law, which deal with the underlying ratio of the vibration scale and thus, therefore, is the basis for all theories that relate to the relationships between colors and musical notes (the repetition of the notes on the board means the sharp "#") [7]. Table 2: Sound and Color Frequencies This research becomes interesting because of the musical notes and colors in the music are constituted of frequencies, but there is a problem that the musical note is always a single color or vice versa, so the relationship that the musical note has with a color would have a lot of variation, and every moment would have other colors. A researcher in the Arts department of PUC-RJ made a relationship between music and colors, in the case of Celso Braga Wilmer. The same has a degree in visual communication, mathematics and music, so the researcher developed a project to facilitate the learning of sheet music reading as cited (Ortega, 2009). 5 Wilmer developed a work focused on the musical pedagogical form, focused on the understanding of music through sheet music, so he named his work "Rainbow Score". The aim of this study was to adapt traditional scores to offer a form of ease to music students with the established musical symbols, where Wilmer saw them as abstract [9]. Based on the levels of abstraction of learning of Jean Piaget (1896-1980), Wilmer's project sought to insert the score already existing within a presentation of the chromatic model that he himself adopted, applying the scales and shades to the largest and smallest in its duration, harmonic and melodic, etc. Thus after the application of Wilmer's project are the degrees of the musical scales that receive the colors and cease to be the tones, as a kind of figure on a background [6]. 4 Methodology In this project we present music for 6 deaf children and a seventh child who had normal hearing that would be a model for the other children to perform the activities, we studied the fundamentals of music together with an expert and a pedagogue,such asmusical notes, rhythms and times of the compass, leaving the traditional way that makes use only of a book, a blackboard or any other medium other than with the use of technology, in this way will be made the use of the following materials: robot Mbot, USB vibratory bracelet, Grand piano Roland, Ipad-Pro e Editom for demonstration. We present in the table below the communication situation and each child, taking into account whether they could read, knew LIBRAS (Brazilian Sign Language) and how deaf: Table 3. Communication situation of each child. Reading LIBRAS Degree of Participant and Age ✔ situation Situation Deafness Child 1 - 9 years - ✔ - ✔ Child 2 - 6 years - - ✔ Child 3 - 6 years - - ✔ Child 4 - 7 years - - ✔ ✔ ✔ Child 5 - 7 years - - ✔ ✔ ✔ Child 6 - 8 years Child 7 - 9 years To achieve the goal in the experiments we will work with a robot integrated with Scratch that is a programming language composed of blocks created by Papert, and more specifically use it adapted in a version of MAkeblock, so we will have the possibility to control the robot and its resources through a mobile device, which can be a mobile phone, tablet or iPad , where children will make combinations in the application to send commands to the robot, thus performing musical concepts that we will detail in the following sections. Before starting any activity with the robot, we will teach the concepts of the project so that children can understand what each item means during the practice of the activities, because none of the children has musical knowledge, this in the theoretical and practical scope. In this way we will teach the combinations between colors and musical notes proposed by [8] that was explained in section 3 of this project, and then use a practical 6 script where we will use a musical concept based on BONA that is a book intended for teaching musical concepts, and the same will be combined with the Editon software to have a musical pentagram with colored notes. Fig. 2 - Pentagram Editom with colored musical notes. After the children know the musical notes combined with colors and have a basis of what each color in the pentagram means we will move on to the second part of the experiments that concerns rhythmic time, we will insert the concept of musical time in the activities. The child will know the equivalences of rhythmic time used in music using the movements of the robot, so we will use periods of initial times in case those that have a greater slowness , even because they are slow the robot will move to the sides in a soft way, so as not to generate confusion for children if we use too much speed in the movements. After the children fix the initial concepts we will start practicing colors with the robot, where each musical note that is played on the digital piano on the mobile device will have a color representing a musical note, which will go to the robot running an LED combined with a robot movement, so the child will begin to know which color belongs to which musical note performed. For the last step we made the use of the robot that was already used from the first stage, but unlike the other occasions the children would not command the robot via mobile device, but rather the robot would be an indicator that a musical note would be being generated from a piano, in case the children pressed a note music on the piano and the robot presented the color of the note performed on it Based on the children's understanding of the theoretical concept of everything that was taught and managed to get through the previous musical activities of basic theory, we then set out for the practical part with a real piano. For the realization of this final activity we used a roland grand piano, in the case of the model "Gp607 Pel", this piano has USB and P10 inputs, in this case the USB input where we integrated the robot to the piano by the USB input so that it was possible for the two to act as if they were only one. After the whole proposal was presented, we started for the initial test where we connected the bracelet in the robot and validated the operation of the robot to know if it would have time accuracy in the vibration of the robot and the bracelet. We 7 programmed the robot in advance in order to just arrive at the site and connect it to the piano and get satisfactory response with the notes pressed on the piano keys, so we connected the robot's USB input to the piano, connected it, after that we clicked on a C note that was represented by the red color in the previous tests and so the robot was in the red color as we predicted , and so too the bracelet vibrated with all the musical notes pressed At the time of the experiment of this stage the child should have his left hand on the piano, because it is the hand with the bracelet, she also felt the vibration of the piano, all this occurred in a synchronized way and we sought not to have delays, so as not to generate confusion in the children's heads. In Figure 3 we have the final schema of the experiment being performed by one of the children during the third experiment. Fig. 3. Child playing piano with all integrated features In this third stage we intended to make the children play the same music used in activity 2 when they used the digital piano on the mobile device, which occurred without difficulty. Also as was proposed earlier in the other activities children who were not in contact with the piano should imitate the movements that the robot generated on top of the piano and raise their hands when the notes were performed, an activity that all children performed several times. All experiments were carried out with the same children, we had the first step we called as experiment 1, the second stage experiment 2 and the third stage of experiment 3. The activities we developed at all stages explored: 1- how children would deal with the concept of colors combined with musical notes, using electronic devices. 8 2- evaluate visual and rhythm issues performed by children when imitating the robot. All the children performed the same activities in the same period, so it would be possible to know what was the abstraction in relation to the period of participation of all children, knowing where each one managed to get in the application and the time it took. In table 5 we present the results obtained in the first activity with the participation of the children, we used the data collected in the evaluations of the monitors. To facilitate the evaluation and classification for the monitors we used 10 questions with 10 answers, in this case grades 1 to 10 in each question, thus how much "lower" the grade of the higher child was their difficulty in the activity. Below we present the general notes of all the children in their participations in experiment 1 following the questions, as given by the monitors. Table 4. Experiment notes 1. Children 1 Children 2 Children 3 Children 4 Children 5 Children 6 Children 7 10 8 8 8 10 10 10 10 8 8 8 9 8 9 10 8 8 8 10 9 8 10 9 9 8 9 9 9 10 10 8 10 10 10 10 9 9 9 9 9 9 9 10 10 10 10 10 10 10 9 9 9 9 9 9 10 10 8 8 8 8 10 8 9 9 9 9 10 9 9 We can notice that in activity 1, children 2, 3 and 4 had a little difficulty in relation to other children, this was because they could not read and do not know pounds, which generated a little difficulty in communication, but after a second attempt they were able to perform the activity completely. In the case of activity 2 we made the sequence of participation different in relation to the first, in the case of activity 1 started the child 1 and then started the children who could not read and were less than old, in activity 2 we started with child 1 again to be a model for the other children, and soon after we started by the children who could read to draw conclusions. One of the expected specifications would be whether it was possible for the child to have kept in mind the concept of the first activity that was performed a week before activity 2, and be able to perform the musical concepts, and then know if the younger children could perform the activities more quickly by having more examples performed before their participation in the execution. Below we present the graph of the second analysis. 9 Table 5. Notes from experiment 2. Children 1 Children 2 Children 3 Children 4 Children 5 Children 6 Children 7 10 10 10 9 10 10 10 10 9 9 9 9 9 8 10 10 9 9 10 8 9 10 9 10 10 9 9 9 10 10 10 9 10 10 10 10 9 9 9 9 9 9 10 10 10 10 10 10 10 9 9 9 9 9 10 9 10 10 10 10 8 8 10 10 9 9 10 10 9 9 It was noted in table 2 that not necessarily the deaf child knowing how to read it will prevail in efficiency in the activities in relation to other deaf children who cannot read, it was noted that the musical concept was absorbed equally among them. When the younger children could not read they began their contact with the game they performed the initial activities with the interpreters indicating the drawings, but after a certain point they began to play alone having ease in understanding the images. At this stage few children presented difficulty in doing the musical activity, according to the evaluation of the interpreters, this may have been due to lack of affinity with technology, but item that over time can change. In activity 1 only children who could not read and did not know pounds had difficulties to start in the musical concept with the iPad and the robot, but in activity 2 this changed. In activity 3 according to the data analyzed the children had very easy to perform the proposed with the piano, had easy to identify the colors, the imitations of the robot were assertive and managed to perform the music easily, because all of them reached the maximum score according to the monitors, according to them the repetitions allowed the children to feel the will , and thus they dominated the proposed activity as shown in the chart below. Table 6. Experiment notes 3. Children 1 Children 2 Children 3 Children 4 Children 5 Children 6 Children 7 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 9 10 9 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 19 9 10 10 10 10 10 Below is the chart showing in general how was the performance in children's notes in all activities, so it is better to have a view with the general notes. Fig. 4. Performance overview in experiments. According to the results presented in the experiments so far, it can be analyzed that the child was able to learn the musical concepts due to repetitions of theory and practice, and another relevant factor was the child to see the other children practicing when on their opportunity with the devices, and also with the help of the robot to generate the movements and present the colors as an aider was a differential to the result obtained, so know that something happened when performing a musical note on the iPad. 5 Final Considerations Seeking to see beyond sound, involving different methods to help children in their interpretation of music as use of colors, we also sought to present through the literature review that there is no neurological obstacle that can compromise an individual to have a musical experience, such that the child aims to be able to make use of images and vibrations of devices , so these items gave us a basis to be able to plan how to achieve the goal better, thus allowing to insert applications and hardware in experiments with hearing impaired children. As soon as we carried out the musical experiments with the hearing impaired children, it was possible to analyze that the architecture used with the application and the robot allowed the children to perform the musical activity, thus assisting in the perception of the sounds that happened and that were performed with the compass, vibration and rhythms through the robot. After the children realized the sound through the color and movement of the robot, they were able to reproduce and move according to the intensity of it, when the robot generated movement and colors. 11 With this it was possible to reflect on what benefits music played in children through their attitudes in the activities, and how they gave us a positive return with their participation in the activities and also with the feedback we had of them at the end of the activities, the comments of the advisors at the time of the activities, because they were doing the evaluation all the time of the children while they were performing the activities , thus presenting the practical concept that it is possible and pleasurable to teach music to deaf children. Based on the evaluations and opinions of the advisors who were all children's educators, it was generally possible to realize that the use of scratch application with robotics comes to be a great way to be used in the musical activities of deaf children, even for reasons of values that are accessible and easy to handle by people even if they are lay people in the subject , the Mbot robot used in the experiment is easily accessible financially, with extensive documentation on the WEB and the solution used here can be easily replicated for people who wanted to perform the same technological context with the documentation available. The activities carried out in this research indicate that the combination of the Scratch application together with a hardware, in the case of this work a robot, can help provide music learning for hearing impaired children. It presented potential for musical activities, and can be adapted to other situations of different visions, thus being able to teach other points in music theory and children learn to have fun 12 References 1. Sousa, A. P. Surdos listen to music. 2006. Available in: . Access In: 28 September (2021) 2. Strobel, K. Deaf: cultural traces not recorded in history. Dissertation (Master) - Federal University of Santa Catarina, Local, (2008) 3. Shibata, D. Deaf brains adapt to feel the music. In: scientific assembly and annual meeting of the radiology society of north america, 87., 2001, Anais. Available from: . Accessed 25 September (2018) 4. Mossesgeld, R. Concept paste allows the Deaf to groove. 2010. Available at: . Accessed: 28 September (2018) 5. Muszka, M. Music, neuroscience and human development. 2014. Available at: . Access 25 September 2018. 6. Ortega, I. The Colors of Sound. Graduation in Music Therapy). Music therapy, Faculdade Paulista de Artes, São Paulo.(2009) 7. Leinig, C. E. Music and Science meet: an integrated study between music, science and music therapy. Curitiba: Juruá, (2008) 8. Wilmer, C. Pontifical catholic university - RJ. PUC Science, 1989. 9. Ortega, I.; Wilmer, C.; Gattino, G. S. The use of colored score in the evaluation process in music therapy. Brazilian Symposium on Music Therapy. XIV, A. D. Olinda: Association of Music Therapy of the Northeast (2012)