Soundscapes of the soul: armonia vitale Federica Perez1 , Claudia Rabaioli2 1 Accademia di Belle Arti Santa Giulia di Brescia, Via Tommaseo, 49, Brescia, Italy 2 University of Milano-Bicocca, viale Sarca, 336, 20126 Milano, Italy Abstract This preliminary study investigates unexplored immersive approaches of music as a therapeutic tool, focusing on its potential to promote wellness and support the healing process, engaging human psycho- physiological responses. The project’s core is an interdisciplinary approach that combines music therapy theories with art and immersive technology, aiming to amplify music’s therapeutic effects through interactive sound environments. To investigate these effects, physiological parameters such as heart rate and electrodermal activity, collected in a previous study, were used to inspire and generate art. The integration of artistic and scientific techniques, through the ”Soundscapes of the soul” project, has allowed us to translate these physiological responses into visual representations, offering a multisensorial experience. These initial researches open new perspectives on the application of music therapy and immersive technologies in improving psychological and physical well-being. Keywords music therapy, psycho-physiological responses, immersive technologies, generative art 1. Background The idea behind this research stems from the growing interest in music therapy and psychology about the transformative power of music and its therapeutic potential. In an age marked by rising psychological disorders and an intensified focus on mental well-being, music may be considered not only as an art form or source of entertainment but also as a potential tool for enhancing mental and physical health. In recent decades, music therapy has gained increasing importance in psychology and medicine, with numerous studies confirming its effectiveness in treating various psychological and physiological disorders. However, the link between music, emotions, and physiological responses remains an open field of study, with many areas still to be explored. In literature, there is a broad consensus that music could be employed to influence emotions and moods. Music therapy theories, such as those developed by Nordoff-Robbins [1], highlight how music can be used in therapeutic contexts to facilitate emotional expression and promote psychological recovery, particularly in patients with post-traumatic disorders, anxiety, or depression[2]. At the same time, neuroscientific studies have shown how music can stimulate different areas of the brain[3], promoting neuronal plasticity and contributing to emotional regulation. However, the effectiveness of music on specific physiological parameters such as heart rate or Italian Workshop on Artificial Intelligence for Human Machine Interaction (AIxHMI 2024), November 26, 2024, Bolzano, Italy Envelope-Open federica.perez02@gmail.com (F. Perez); claudia.rabaioli@unimib.it (C. Rabaioli) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings electrodermal activity has only been partially explored, and further research is needed to fully understand the underlying mechanisms. This research aims to investigate more deeply the interaction between music and emotions by examining the measurable physiological effects elicited through music listening, utilizing an artistic project that translates physiological data into generative art. This study states the following hypothesis: (1) music has the capacity to evoke measurable psycho-physiological responses, influencing parameters such as heart rate and electrodermal activity, whose behaviour could be visually translated into dynamic art forms; (2) familiar melodies, compared to unfamiliar ones, are expected to resonate more profoundly, fostering positive emotions and states of comfort that manifest in art reflecting more relaxed patterns; (3) immersive technologies can intensify the emotional and physiological experience of music, creating a layered, multi-sensory engagement that draws the body and mind into deeper resonance, captured vividly in the generative artwork produced from participants’ physiological data. 2. The project The concept of this project is to create a bridge between art and science, using art as a way to promote well-being. Exploring how different types of melodies impact physiological responses highlights the potential for art to intersect with scientific research, influencing both body and mind. This study involves physiological parameters such as heart rate and skin conductance, with future plans to include brain activity. This approach utilizes technology not only to deepen the understanding of music’s effects but also to pave new pathways for therapeutic applications rooted in art. The aim of this work is twofold: on the one hand, to collect empirical data that will con- tribute to the scientific literature on music therapy, exploring how music could be an effective therapeutic tool; on the other hand, to artistically translate these physiological responses into visual works that reflect the emotional and physical impact of music on the subjects involved. The project addresses three main objectives: scientific, artistic, and educational. From a scientific perspective, this research aims to contribute to the exploration of music therapy, based on the hypothesis that music can be a powerful tool for enhancing psychophysiological well-being. Currently, the study provides a conceptual framework and preliminary insights into how music may influence human physiological and emotional states. On the artistic front, the project seeks to create a bridge between science and art, using scientific data as inspiration for visual works. Lastly, from an educational perspective, the goal is to foster greater awareness of the therapeutic potential of music among the general public and within the scientific community. The project is divided into three main phases. The first phase focuses on data collection from previous studies, where physiological data such as heart rate and skin conductance are gathered to observe the psychophysiological responses to different types of music. In the second phase, the collected data is used to create generative art through coding, translating the physiological responses into visual representations. The final phase involves 3D modeling of the exhibition space, where the generative art is integrated into the environment, providing an immersive, spatial experience that allows visitors to engage with the effects of music on the body and mind. Figure 1: Sensors used to collect physiological data. 2.1. Physiological data A collaboration with the MultiMedia Signal Processing (MMSP) Laboratory of the Department of Informatics, Systems and Communication at the University of Milano-Bicocca, was fundamental to obtain physiological data to be employed in the artistic project. Forty participants were involved in a previous experiment. During the experiment, the subjects listened to six songs, two of which were specially chosen by them to evoke their memory and emotions. During the whole experiment, the heartbeat of each participant was collected through Photopletysmography (PPG) using the Shimmer3 GSR+ Unit [4] with a sampling frequency of 128 Hz. In addition to PPG, the Electrodermal Activity (EDA) of each subject was acquired using the same sensor. The Shimmer3 GSR+ Units are non-invasive and completely painless sensors that could be easily worn by the participants, as shown in Figure 1. In this preliminary work, two participants’ physiological responses to the audio stimuli have been selected. The choice is due to the presence of good-quality signals and a clear correlation between the participants’ emotional responses and their chosen songs. The participants were intentionally chosen to represent distinct age groups—one younger and one older—to facilitate the examination of potential age-related differences in their responses. 2.2. Generative art code The second phase of the project focuses on the development of a generative art code, where the collected data is transformed into dynamic visual art, creating a direct representation of the physiological responses to music. Generative art is a form of digital art that leverages algorithms and mathematical models to create evolving, dynamic artworks. Unlike traditional static pieces, generative art unfolds in real-time, allowing it to exist as a continuous process that evolves in space and time[5]. This approach was selected because it reflects the fluid nature of human emotions and Figure 2: Official screenshot of the generative art code. physiological responses, making it an ideal medium for visualizing the impact of music on inner states. By employing generative art, we established a bridge between science and art, transforming real-time physiological responses to music into visually dynamic experiences. This approach allowed the creation of art that evolves based on participants’ unique psycho- physiological data, such as heart rate and skin conductance. This data-driven generative system provided a scientific basis for the artwork, enabling a personalized, immersive experience that reflects the emotional and physical impact of music on each individual. In this way, the project showcases how artistic expression can be informed by empirical data, offering a novel perspective on the interplay between art, music, and human physiology. Figure 2 is a frame of the final opera, where the speed of particles was mapped directly to the subjects’ heartbeat, while skin conductance data modulated the saturation of the visuals. Whenever there was a significant fluctuation in heart rate, it would control the speed of the particles, making them stop and accelerate, creating a pulsing effect to evoke the sensation of a heartbeat. Similarly, for skin conductance, the higher the value, the more saturated the colors became, visually reflecting the intensity of the participant’s emotional arousal. This algorithm was designed to produce evolving, unique works of art that not only embodied the physiological responses to music but also visually captured the personal emotional journey of each participant. The result was an artistic expression of raw biological data, blending science with art in an interactive, immersive way. However, the static image alone here reported cannot fully convey the dynamic nature and profound impact of this artistic translation. The subtle interplay between the participants’ emotional states and the music becomes evident through the real-time motion of the particles.In those preliminary representations, familiar melodies seem to evoke a sense of calm, causing the particles to move in a smooth, harmonious flow. In contrast, unfamiliar music show more pronounced fluctuations in heart rate, resulting in dramatic particle movements that alternated between complete stillness and sudden, rapid bursts of motion, frame by frame. This remarkable visual contrast highlighted the unique Figure 3: Exhibition’s 3D model: view from the top. Figure 4: Exhibition’s 3D model: isometric view. Figure 5: Exhibition’s 3D model: the sphere indicates where the particles will be projected. ways individuals respond to different auditory stimuli, suggesting deep connection between music, emotion, and physiology. Further studies could deepen these aspects to validate them as common patterns, increasing the subjects involved. 2.3. Exhibition space To further bridge the gap between science and the public, in the third phase of the project an interactive exhibition space is designed to make scientific concepts more accessible through the medium of art 4 3. This exhibition would not only invite visitors to engage with the potential Figure 6: Exhibition’s 3D model: the isolated cabins that allow an individual experience. Figure 7: Exhibition’s 3D model: the draping indicates the entry into the reflection zone. Figure 8: Exhibition’s 3D model: the area where people can reflect on the experience and leave feedback. benefits of music but also demonstrate how art and science can be intricately intertwined, offer- ing a deeper understanding of both disciplines. This immersive environment also would allow us to monitor participants’ physiological responses, such as heart rate and skin conductance while listening to different melodies. Visually representing contextually those data let the audience experience the music’s impact on the body, promoting greater awareness of its therapeutic potential and creating a bridge between science, art, and personal experience. Technology plays a central role in the project, helping to create an immersive and interactive experience for visitors. Sound-isolating head- phones 6 allow one to enjoy the melodies optimally, improving concentration and promoting a deep perception of sound stimuli. Physiological parameters such as heart rate, body temperature, and brain activity will be measured by specially placed external sensors to which participants connect before the start of the experience. Advanced holographic technologies 5 would project the generative artworks, creating a dynamic three-dimensional space in which visitors can move freely, surrounded by floating particles responding to their physiological data. Up to four people can experience this simultaneously, with their projections fused together and responding to the same musical stimuli. In addition, isolated cabins are available, as shown in 6, offering a controlled environment for an even deeper and more personalized immersion, allowing participants to explore the interaction between music and body intensively and individually. Programmable LEDs, placed around the room, change colour and intensity in harmony with music and generative artwork, creating an immersive atmosphere that maintains low brightness to enhance the holographic projections. The visitor’s experience is conceived as a narrative path that begins with an introduction to the communicative power of music 7, through informative panels and explanatory videos, and culminates in the interactive experience itself in the following space. At the end of the exhibition, visitors will find a dedicated reflection area designed to enhance their post-experience contemplation 8. This space features ergonomic seating and sound-absorbing walls, creating a quiet environment for introspection. Visitors are encouraged to reflect on their experience and share their thoughts by providing feedback through tablets placed in this zone. Soundscapes of the soul is not just an exhibition space, but represents a comprehensive experience that explores the deep connection between music and emotions, and in this case, how both of them can be linked to generative art. This project offers visitors a unique journey into the world of interactive generative art, combining technology, science, and creativity to raise awareness about the importance of music for our psycho-physical well-being, highlighting their relationship through the visual representation of our body reaction to music listening. 3. Discussion This research contributes to the hypothesis that music may impact significantly on the human physiological responses and suggests that familiarity with a musical piece significantly modu- lates these responses. Visually interpreting the participants’ physiological responses to music allowed us to translate those scientific data into more comprehensible language for a wider audience, using art as a medium. This artistic representation of the data provided a unique lens through which to observe the physiological impacts of music, offering insights into the calming or stimulating effects different tunes can elicit. The reduction in heart rate and skin conductance while listening to familiar tunes suggests that musical familiarity could activate mechanisms of comfort and safety, which are reflected in stress reduction. This effect may be particularly useful in clinical settings, where music could be used as a tool to reduce anxiety and promote relaxation, especially in patients with stress-related disorders or chronic anxiety. The ability of music to modulate the physiological parameters of patients offers interesting prospects for music therapy as a non-invasive and accessible intervention. In addition, based on the interpretation of the artwork, it is hypothesized that listening to unfamiliar melodies may elicit greater emotional and physiological activation than familiar ones, suggesting a more intense cognitive response to new stimuli. This assumption opens new avenues for exploring the potential role of music in cognitive rehabilitation, though further empirical investigation is required to substantiate these ideas. It may be of interest to explore whether exposure to unfamiliar melodies can promote cognitive recovery in patients with brain injuries or cognitive deficits, such as those resulting from trauma or neurodegenerative disorders. The hypothesis is that music can not only facilitate relaxation but also stimulate more complex cognitive processes, promoting neuronal plasticity and contributing to the functional reorganisation of the brain. Research suggests that immersive technologies could be effective tools for amplifying the effects of music, as the integration of visual and sound stimuli may further enhance emotional and physiological responses. The use of immersive environments, combined with music therapy, could offer new therapeutic solutions for patients with neuropsychological disorders, helping them to recover cognitive and motor functions through a multisensory experience. The inte- gration of virtual reality and musical biofeedback would help with the treatment of conditions such as depression, anxiety disorders and post-traumatic stress disorders (PTSD)[6]. Another promising application area is physical rehabilitation. The relaxing and activating effects of music, combined with immersive technologies, could facilitate the rehabilitation of patients with motor difficulties or paralysis, stimulating muscle activation and reducing emotional stress associated with physical rehabilitation. In the future, it would be interesting to include the impact of music on brain responses in the project, using techniques such as functional magnetic resonance imaging or positron emission tomography. These tools could provide a more detailed view of the areas of the brain involved in emotional regulation during musical listening, contributing to a deeper understanding of the underlying neuropsychological mechanisms, allowing us to more efficiently use immersive environments as a therapeutic tool. 4. Conclusion This study aims to raise awareness of the innovative use of technology in exploring the influence of music on human physiological responses, addressing an ongoing issue in a novel way. By examining the potential impact of familiar and unfamiliar melodies on emotional and physiological activation, the project emphasizes how music can be leveraged not only as an art form but also as a powerful tool for understanding psycho-physical well-being. These insights suggest that familiar melodies may foster relaxation, while unfamiliar ones could induce heightened emotional responses, opening new opportunities for research into the therapeutic potential of music. The exploration of immersive technologies offers a new frontier for enhancing the effective- ness of music therapy. The study highlights the potential for multisensory environments that combine music with interactive visual stimuli to amplify therapeutic effects, particularly in patients with neuropsychological or motor disorders. Although the study does not offer conclusive evidence, it sets the stage for further investiga- tion, proposing innovative pathways to improve the application of music therapy and expand its use in medical and rehabilitation contexts. Acknowledgments I would like to extend my deepest gratitude to professor Simone Lombardi from Accademia of Fine Arts Santa Giulia of Brescia, whose dedication and expertise have been essential guiding this journey. His patience and encouragement pushed to explore new frontiers and enriched the development of this project. We are also sincerely thankful to Professor Francesca Gasparini and Aurora Saibene from MMSPlab (https://mmsp.unimib.it/) of Milano-Bicocca University, for their continuous scientific support, significant contributions, and invaluable advice. Lastly, also Milano-Bicocca University itself should be surely thanked for allowing us to use their laboratory facilities, which made this study possible. References [1] K. Aigen, Being in Music: Foundations of Nordoff-Robbins Music Therapy, Barcelona Publishers, 2005. [2] G. Manarolo, Manuale di musicoterapia: Teoria, metodi e strumenti per la formazione, Cosmopolis, 2020. [3] C. Toader, C. P. Tataru, I.-A. Florian, R.-A. Covache-Busuioc, B.-G. Bratu, L. A. Glavan, A. Bordeianu, D.-I. Dumitrascu, A. V. Ciurea, Cognitive crescendo: How music shapes the brain’s structure and function, Brain Sciences 13 (2023). URL: https://www.mdpi.com/ 2076-3425/13/10/1390. doi:10.3390/brainsci13101390 . [4] A. Burns, E. P. Doheny, B. R. Greene, T. Foran, D. Leahy, K. O’Donovan, M. J. McGrath, Shimmer™: an extensible platform for physiological signal capture, in: 2010 annual international conference of the IEEE engineering in medicine and biology, IEEE, 2010, pp. 3759–3762. [5] M. Digitale, Generative art: orizzonti e limiti delle ai “creative”, Neosperience (2022). URL: https://www.neosperience.com/blog/generative-art-orizzonti-e-limiti-delle-ai-creative/. [6] J. L. Maples-Keller, B. E. Bunnell, S.-J. Kim, B. O. Rothbaum, The use of virtual reality technology in the treatment of anxiety and other psychiatric disorders, Harvard Review of Psychiatry 25 (2017) 103–113. doi:https://doi.org/10.1097/hrp.0000000000000138 .