Modern artificial intelligence (AI) technologies are becoming an integral part of many spheres of human activity, including creative industries [1,2]. One such area where AI demonstrates significant potential is sound design. This discipline combines art and technology to create sound compositions used in cinema, video games, advertising, music and other media. The use of AI in sound design opens up new possibilities for process automation, sound generation and interactive sound accompaniment, changing traditional approaches to working with sound.
Despite significant interest in the use of artificial intelligence in creative industries, the topic of AI application in sound design is not yet fully explored in modern scientific literature. Most studies focus on specific aspects such as sound synthesis, audio signal processing or adaptive sound systems for interactive environments. However, a holistic analysis of the theoretical foundations, practical applications, and the impact of these technologies on the industry as a whole remains fragmented.
Some works highlight the technical aspects, describing the algorithms and methods used to generate or process sound. Others focus on applied cases, such as the integration of AI in the production of music or sound effects for cinema and games. Meanwhile, a comprehensive approach that would take into account both creative and technical challenges, ethical aspects and development prospects is still lacking.
This indicates the need for deeper research that would create a general concept of using AI in sound design. This article attempts to fill this gap by analysing not only existing technologies, but also their impact on the process of sound creation, as well as outlining future prospects for this field.
In the classical sense, sound design is the process of obtaining (generating), editing, and implementing sound elements (samples) in a multimedia composition [3]. It covers a wide range of applications, including cinema, theatre, video games, advertising, the music industry and even architectural design of sound environments. The main principles of classical sound design include the following aspects [4]:
• Realism and authenticity. The main principle underlying the classical approach is the creation of realistic sounds that correspond to the visual or dramatic context. • Technical skill. Sound designers rely on traditional methods of recording sound using microphones, field recorders, analogue and digital processing tools. • Foley art. A special place in classical sound design is occupied by the art of creating sound effects manually, using real objects and materials to imitate various sounds. Real sounds are recorded and processed by appropriate means of artistic sound processing, such as reverberation, echo, chorus, etc., to achieve the desired result.
• Composition and editing. The sound designer combines sounds into a sound composition, using editing to achieve the desired rhythm, harmony and dramatic impact. That is, a synergistic combination of sound and dynamic change of images is performed.
The classical approach laid the fundamental principles of sound design, which still remain relevant. However, changes in the digital landscape pose new challenges. Classical methods of sound design have their limitations (for example, lack of adaptability, instrumental and technical limitations, time and resource costs).
The gradual development of digital technologies has also changed the principles and means of sound design. With the advent of VST (Virtual Studio Technology) and AU (Audio Units), sound designers gained access to thousands of digital instruments, simulators of analogue and digital synthesisers, classical musical instruments and effects [5,6]. This significantly reduced equipment costs and expanded the possibilities for experimentation. The development of the gaming industry stimulated the emergence of adaptive audio systems, where sound changes depending on the player's actions or environment. Wwise (figure 1) and FMOD technologies have become the standard in the field of interactive sound design [7].
Digital sound libraries gradually began to appear -sets of pre-recorded samples that can be quickly applied to one's own projects. The use of ready-made sounds was not a new practice in itself -the 1950s-60s saw the creation of the first commercial libraries storing sounds of gunshots, natural phenomena, transport, etc. [8]. They were recorded on analogue media (e.g. magnetic tape) and used in cinema and television.
Modern sound design has reached a level where technologies allow the creation of high-quality sound for various types of media -from cinema and video games to advertising and virtual reality. Audio processing tools have become more powerful, and access to large sound libraries, virtual instruments and modern technical means for recording and processing sound have greatly simplified the process of creating sound content. But the demand for constant improvement remains unchanged. That is why the question arises: how to adapt and use the possibilities of artificial intelligence, which continues to develop comprehensively, in the sound design industry. And how expedient is such use?
The use of artificial intelligence for sound generation has become one of the most promising areas in the field of sound design. At a basic level, this process is based on the ability of algorithms to learn from large volumes of audio data, analyse them and create new sound textures that can be used in music, cinema, video games and virtual reality [9].
In the early stages of development, artificial intelligence algorithms worked primarily with existing sounds. They could restore audio, remove noise or imitate the sound character of specific instruments. However, with the development of machine learning [10] and neural networks [11], AI has become capable of creating completely new soundscapes that did not exist before. For example, generative adversarial networks (GANs) allow systems to synthesise sounds that have a natural timbre, and recurrent neural networks (RNNs) learn to predict the next sound segments, creating a continuous audio stream.
One of the most striking examples of AI applications is the creation of sounds for music. Algorithms analyse thousands of music tracks, extracting patterns and harmonies, and then generate melodies or rhythms. Programs like AIVA (Artificial Intelligence Virtual Artist) are capable of creating entire compositions in various genres, providing composers with a foundation for further work [12]. In the field of electronic music, AI is often used to create unique samples or synthetic textures that can be integrated into compositions.
At the beginning of 2024, the Suno AI network gained a high level of popularity [13]. Suno AI is an innovative platform that uses artificial intelligence to generate music based on text prompts. The user enters a description of the desired song, specifying the style, genre or theme, and the system creates a corresponding composition. This process takes about two minutes, after which the user receives two versions of the track: one with vocals, the other instrumental.
Suno AI technology is based on artificial intelligence models such as Bark and Chirp, which are capable of generating not only instrumental music but also adding vocal parts to songs. The algorithm analyses the entered text, determines its rhythmic and semantic features, and then synthesises a melody and harmony that match the given description. The vocals are synthesised taking into account the rhythm and intonation of the text, giving the song a natural sound.
The system uses an approach similar to large language models, such as ChatGPT [14]: it splits the text into individual segments (tokens), studies millions of usage variants, styles and structures, and then reconstructs them on request. However, creating audio, especially music, is a more complex task, as it requires taking into account many parameters such as melody, harmony, rhythm and timbre.
Of course, the tracks generated by both Suno AI and other platforms and models have noticeable subjective flaws, which most often manifest themselves in distortions in vocal parts, sharp changes in volume or banal misunderstanding or neglect of prompts. Artificial intelligence can work much more accurately with small sounds. Classic elements of sound design are various sound transitions, for example, a gradually increasing sound (rise), a sound of impact (hit), a sound of cutting air (whoosh), etc.
Artificial intelligence is able to generate and process such short sound effects with high accuracy due to its ability to analyse thousands of samples and extract key sound characteristics. These effects have a clear structure and predictable dynamics, making them ideal material for algorithm work. AI models can create variations of hits, rises or noises based on text descriptions or user settings, providing precise adjustment of the duration, frequency spectrum and amplitude of each sound.
In addition, thanks to machine learning technologies, artificial intelligence can automatically select sounds for different scenes, creating smooth transitions and adapting them to the visual content. For example, AI can generate a whoosh sound of varying intensity depending on the speed of an object in the frame or synchronise impact effects with moments of climax.
There are already services that provide the ability to create cinematic sounds with a text prompt. But such sounds are only a small part of sound design. For us, sound design is primarily a complex sound landscape, an immersive global environment. Is artificial intelligence capable of forming something like this?
Immersive environment sound design requires not only layering sounds on top of each other, but also fine-tuning spatial acoustics, dynamics and the emotional content of each layer. In the real world, sounds interact with each other in unpredictable ways -echoes in space, gradual fading or swelling, the influence of textures of materials and objects that create or reflect sound. It is difficult for algorithms to reproduce this chaos and versatility of the sound environment in the same way as human hearing and perception.
Currently, artificial intelligence does an excellent job of reconstructing real environments through recordings and spatial analysis, but creating completely fictional sound landscapes that have no analogues in reality requires creative intuition. A human sound designer works not only with sounds as such, but with a concept -they create a story through sound, using audio as a tool to evoke emotions and build atmosphere.
However, there are also positive aspects. Artificial intelligence algorithms are becoming increasingly effective in creating procedural sound landscapes. They are able to analyse visual sequences or text descriptions and generate corresponding sound environments, automatically adding necessary elements: the sound of wind, raindrops, city bustle or any other simple ambient.
Artificial intelligence cannot fully construct a multi-layered sound environment. But it can be used as a tool that provides a certain foundation to work with. For example, it is possible to create simple patterns of classical instruments in a given key and rhythm, perform their gradual processing using classical means in any sound editing environment, mix the tracks, supplement them with various generated sounds, and further integrate the created composition into complex multimedia environments.
Another equally important aspect is the integration of artificial intelligence technologies into various plugins for working with sound. Often these are a wide variety of tools for different tasks, which have appeared quite a lot recently. AI assistants are used to perform general mastering (such as the built-in assistant in Izotope Ozone 10/11), for individual tasks such as compression, limiting, saturation, equalisation, etc. AI plugins are trained on large volumes of audio data. Developers train the neural network based on various recordings manually processed by professional sound engineers. The model analyses how classical tools for saturation, compression or limiting work, and learns patterns of effect application depending on the type of sound, genre or processing style.
When the user loads the plugin, the algorithm performs a multivariate analysis of the audio signalanalyses the frequency spectrum, dynamics, harmonics and noise level. AI models are able to detect problem areas or potentially weak zones and suggest processing parameters. Practical experience shows that these parameters are often not optimal, but can be used as a basis for further work with sound.
Much more interesting from the point of view of sound design are plugins such as Synplant 2. The Genopatch technology (figure 2) built into the plugin allows generating a variety of new sounds based on a single loaded sample [15]. The unique capabilities and interface of Synplant 2 promote experiments in the field of intuitive sound design, allowing to explore how non-standard methods of interaction with technology can influence the creative process.
Considering all of the above, we can note that artificial intelligence is already transforming the field of sound design today, opening up new possibilities for creativity and automation. Despite significant achievements, AI technologies in sound design face certain significant challenges: limited emotional depth of generated sounds, complexity of creating complex immersive environments, and various technical defects that can distort the perception of the overall picture. However, these limitations stimulate the development of the industry and create space for improving algorithms, integrating new approaches and synergy with human creative ideas. AI does not replace sound engineers, but becomes a powerful tool that helps accelerate the workflow and expand creative horizons. Now, let's demonstrate the possibilities and ways of applying artificial intelligence technologies in sound design through practical experience.
In this part, as an example of the possibilities of using AI in sound design, we will form a simple sound design for several scenes that are planned to be used in a visual novel. Visual novels (VN) are a genre of interactive games where the main emphasis is on the plot and characters, and sound plays an important role in creating an emotional response. The scenes themselves were created using Dalle-E 3 and refined in Adobe Photoshop (figure 3). To begin with, we break down the scenes into components and determine the overall mood and which sounds we need (figure 4).
Wind noise; Melancholic classical instruments. Now let's determine the necessary artificial intelligence tools. For forming the general landscape, the aforementioned Suno AI from the previous section will work. Using the following prompt, we generate two compositions for download: Violin and piano, melancholic style, slow tempo, dark ambient. We transfer the downloaded result to the FL Studio environment and perform the following sequential processing: slowing down, equalisation, adding reverb and echo effects using the Crystalize granular generator from the developer SoundToys (figure 5).
It is worth noting that the recording generated in Suno AI without further processing would not fully correspond to the general concept of sound design for this project. As already mentioned above, Suno AI often does not interpret prompts very accurately, which leads to problems with generating music in less well-known genres. However, artistic processing tools allow to significantly change and improve the nature of the input sound and adapt it to the needs of the project.
Unlike Suno AI, the AudioGen AI instrument handled the prompt more accurately, generating distant humming of electric wires and cracking of branches with a rather short request (figure 6). To create various variations of the sample generated using AudioGen, we will use Synplant2 and its Genopatch technology. We load the sample into the plugin environment, after which Synplant2 automatically generates new sound samples based on the provided one (figure 7).
After combining all the generated sounds, we have as a result a simple but subjectively quite highquality sample of sound design for scenes from a visual novel. Thus, based on practical experience, 319-328 the feasibility of using artificial intelligence technologies as a tool for quickly obtaining the necessary sound samples for their further processing and combining into a coherent composition was confirmed.
The article provides a thorough analysis of the theoretical and practical aspects of using artificial intelligence technologies in the field of sound design. The current state of the industry is highlighted, considering its technical capabilities and creative challenges. A study of classical and modern tools for forming sound environments is conducted.
It has been proven that artificial intelligence significantly changes the traditional approach to sound creation, providing process automation, time savings, and expanded possibilities for experimentation. For the first time, a detailed analysis of the main trends and directions of the impact of artificial intelligence on sound design is presented. Special attention is paid to tools such as Suno AI, AudioGen, Synplant2, which demonstrate significant potential for generating sound textures and integration into creative projects.
The practical aspect of the research is based on the example of creating a sound accompaniment for visual novels, where artificial intelligence was used to generate musical compositions and sound effects. These materials, after further processing, can become the basis for high-quality full-fledged sound design. It is important to emphasise that although AI tools provide speed and adaptability in working with sound, their results often require refinement to match creative ideas.
In this article, for the first time, an integrated approach to the use of various artificial intelligence tools for creating sound design is proposed. This approach takes into account both technical capabilities and creative needs. The study outlines the advantages of modern AI algorithms, such as efficiency in creating short sound effects, as well as their limitations, including difficulties in forming complex immersive sound landscapes. Also, for the first time, the article presents a methodology for selecting artificial intelligence tools for specific tasks in the context of sound design for multimedia projects. For example, it is determined that tools such as Suno AI are appropriate for creating music and musical effects, AudioGen for generating sounds of certain environments, and Synplant2 for editing sounds. This methodology is formed on the basis of practical work with these tools and subjective evaluation of the generation results.
The overall results and prospects for further development of this problem can be defined as follows:
• A study of the main directions of using artificial intelligence in sound creation has been conducted, including the generation of musical compositions, short sound effects, and procedural sound landscapes. It is shown that tools like Suno AI, AudioGen, and Synplant2 are able to effectively perform sound generation and processing tasks, which greatly simplifies the complex process of creating sound design; • The article presents an example of creating sound design for visual novels, which illustrates the capabilities of AI for quickly obtaining basic sound textures. It is shown that artificial intelligence can be used to automate sound creation with further processing and refinement, which allows achieving high-quality final results; • An integration approach is proposed, which consists in using various artificial intelligence tools for different tasks that may include short musical compositions, simple sound landscapes, and short sounds. Subjective evaluation of the quality of the created samples shows that they are quite suitable for use in various multimedia projects.
The practical significance of the obtained results lies in increasing the efficiency of sound design creation processes through the integration of artificial intelligence technologies. In particular, the proposed methods allow automating routine tasks, such as generating basic sound textures and creating simple sound or musical effects. This reduces the time and resources required for work and allows designers to focus on the creative aspects of projects.
Prospects for further research are primarily related to improving algorithms for creating immersive sound environments, deepening the synergy of AI and human creativity, and more active integration of generated sounds into multimedia projects. These prospects demonstrate the potential for further transformation of the sound design industry, expanding the capabilities of creative professionals and stimulating the development of innovations in the use of artificial intelligence.
The study confirmed the practical value of artificial intelligence in transforming sound design, expanding the toolkit for creating sound compositions and opening up new horizons in creative industries.
Declaration on Generative AI: The authors have not employed any Generative AI tools.