This article presents an innovative approach to creating virtual imagery through text and speech interfaces for immersive environments (VR/AR/MR). The research focuses on developing and integrating a system that transforms textual descriptions and voice commands into visual images that can be dynamically embedded in virtual and augmented realities. The developed system employs advanced natural language processing and speech recognition methods combined with generative models to create contextually relevant visual elements. Experimental studies have shown significant improvements in the speed and accuracy of virtual object creation compared to traditional modeling methods. The proposed method demonstrates substantial enhancement of user experience and expands interaction capabilities in immersive environments. The research results open new perspectives for developing intuitive interfaces in virtual and augmented reality, while also contributing to the democratization of content creation for immersive technologies.
In the modern era of digital transformation in education, immersive technologies are becoming a
crucial tool for knowledge transfer. According to research [
In this context, the development of intuitive interfaces for generating virtual content becomes particularly relevant. Using natural human communication methods – speech and text – appears to be a promising solution to this problem. Cognitive psychology research [5] confirms that verbal descriptions can create clear mental images that can be transformed into visual representations.
The scientific novelty of the proposed approach lies in: 1. Development of a hybrid method for transforming natural language descriptions into threedimensional visual images for immersive environments. 2. Creation of an algorithm for contextual adaptation of generated content considering
VR/AR/MR platform specifics. 3. Formation of a methodology for evaluating the effectiveness of text-speech interfaces in the educational process.
Unlike existing solutions [6, 7], the proposed system has the following innovative characteristics: Multimodal input processing (text + speech) using advanced Natural Language Processing algorithms. Adaptive image generation considering technical limitations of the target immersive environment. Real-time integration with popular VR/AR/MR platforms through a universal API.
The technical implementation of the system is based on modern achievements in machine learning and computer vision. It uses a combination of transformer models for natural language processing and generative adversarial networks for creating visual content [8, 9]. The system is implemented using TensorFlow with optimized models for real-time operation.
The main objective of the research is to develop and validate a system that transforms textual descriptions and voice commands into visual images for immersive environments [10, 11]. The practical significance of the work is confirmed by the results of pilot implementation in the educational process, demonstrating an 82% reduction in content creation time and a 64% increase in student engagement.
The current state of research in integrating text-speech interfaces with immersive technologies is characterized by active development in several interconnected directions. Analysis of existing literature reveals the following key research areas [12, 13].
The development of speech interfaces in immersive environments has undergone significant evolution. Early work by Smithson and Wang (2019) focused on basic voice command recognition in VR environments, achieving accuracy of around 85% under laboratory conditions. Subsequent research by Martinez et al. (2021) substantially improved this indicator to 97% through the application of transformer architectures and contextual command processing. Particularly significant was the study by Kumar and colleagues (2022), which presented the first comprehensive system of continuous speech interaction in AR environments capable of adapting to ambient noise and user accent [14, 15, 16].
In the field of text interfaces for virtual content generation, a significant breakthrough is associated with the work of Chen and Li (2023), who introduced the concept of "semantic anchors" – special text markers allowing precise description of spatial relationships between generated objects. Their approach was successfully expanded in Thompson et al.'s (2024) research, adding support for temporal descriptions to create animated scenes [17].
Special attention should be paid to the direction of multimodal integration, where text and speech interfaces are combined with gesture control systems. The fundamental work of Rodriguez and Park (2023) proposed a universal architecture for synchronizing various input modalities, which significantly enhanced the naturalness of user interaction with the virtual environment [18, 19]. Their method has been successfully applied in several commercial VR applications, demonstrating the practical value of theoretical developments.
Performance and optimization issues are also widely covered in the literature. Liu et al.'s (2024) research presented efficient methods for caching and pre-generating content, which reduced system response latency to acceptable VR values (less than 20 ms). Simultaneously, work by a European research group led by Anderson (2024) demonstrated the possibility of significantly reducing the computational complexity of generative models while maintaining high-quality results [20].
An important aspect is also the study of user experience and interaction ergonomics. Yang and colleagues' (2024) large-scale study, covering over 1,000 users, identified key factors affecting user satisfaction when working with text-speech interfaces in VR/AR environments. Their findings formed the basis of modern recommendations for designing user interfaces for immersive technologies.
Despite significant progress, literature analysis reveals several unresolved problems and promising directions for future research. In particular, issues of semantic consistency in generating complex scenes, optimization of computational complexity for mobile devices, and development of more intuitive methods for describing spatial relationships in text prompts remain relevant.
This literature review demonstrates the active development of the field and forms the basis for further research in the direction of integrating text-speech interfaces with immersive technologies [21].
This section presents a comprehensive research methodology aimed at creating an effective virtual content generation system based on text and speech inputs. The methodology includes several interconnected stages, each directed at solving specific research tasks.
System Architecture The proposed system is based on a modular architecture consisting of four main components:
Natural Language Processing Module (NLP Module). This component is responsible for primary processing of text and speech inputs. For speech recognition, a modified Wav2Vec 2.0 architecture is used, optimized for operation in immersive environments. Text inputs are processed using a pre-trained BERT language model adapted for working with spatial descriptions:
QK T A (Q , K , V )=softmax ( )V √ dk where Q – query matrix, K – key matrix, V – value matrix, dk– key dimension 2. Semantic Parser. This module transforms processed inputs into structured semantic graphs describing spatial and temporal relationships between objects. An original graph construction algorithm is used, taking into account the specifics of three-dimensional scene description. The transformation of textual descriptions into vector representations is carried out as follows.
Let = {1,2,…,} - be a text description consisting of n tokens. The vector representation of the text is calculated as:
I t=G ( E (T ) , I {t−1}, Δt )
S (T , I )= λ1 S{sem}(T , I )+ λ2 S{geom}(I )+ λ3 S{text }(I ) where It – current frame, I{−1} - previous frame, Δt- time interval, G – generation function. Generation quality is evaluated through an integral conformity metric: where S{sem} - semantic correspondence, S{geom} - geometric consistency, S{text} - texture quality λi – weight coefficients.
where wi defined as:
E ( )=∑ wi∗v (ti) wi exp ( s (ti)) ∑ exp ( ) s (t j) (1) (2) (3) (4) (5) (6)
Generative Subsystem. The component responsible for creating three-dimensional models and textures based on semantic graphs. Implemented using a modified Neural Radiance Fields (NeRF) architecture optimized for real-time operation. Virtual image generation is based on transforming the semantic vector into a three-dimensional representation:
D ( )= F ( E (T ) , p)=σ (∑ α k ϕk (E(T ), p)) where F – generator neural network, ϕk – basis functions, αk – mixing coefficients, σ – activation function.
4. Integration Module. Ensures the embedding of generated content into the existing immersive environment, considering physical constraints and scene context.
To ensure temporal consistency, a recurrent formula is used: Data Collection Process Three datasets were collected for system training and validation: Text Corpus 10,000 natural language descriptions of three-dimensional scenes.
Annotations of spatial relationships.
Metadata about description complexity and context 1. 2. Speech Dataset
1,000 hours of voice command recordings. Various accents and recording conditions. Timestamps and transcription markup.
4.1. Title information Within the research, comprehensive experiments were conducted to evaluate the effectiveness of the developed virtual image generation system based on text and speech inputs. Testing was carried out in several stages focusing on various aspects of system operation. Speech Interface The first stage of experiments was aimed at evaluating the effectiveness of the speech interface as a means of inputting commands for virtual image generation. Testing was conducted using Google Web Speech API and included the following aspects:
Integration into VR/AR/MR Environments The third stage of experiments focused on evaluating system integration into various immersive environments:
The research conducted on integrating text-speech interfaces with VR/AR/MR technologies revealed several important aspects and opened new directions for further development in this field.
The speech interface effectiveness showed promising results, achieving recognition accuracy of 95.2% under optimal conditions. This significantly exceeds previous research indicators, where accuracy rarely exceeded 85%. However, it's important to note that system performance significantly depends on environmental conditions, as confirmed by accuracy reduction to 88.7% with background noise.
Virtual image generation based on text and speech descriptions demonstrated high efficiency for simple objects (96.5% correspondence) but showed certain limitations when working with complex scenes (85.7% correspondence). This indicates the need for further optimization of generational algorithms for more complex use cases.
Integration with immersive environments revealed the critical importance of minimizing system response latency. The achieved indicator of 11.2 ms is a significant improvement compared to existing solutions, where latency often exceeds 50 ms. However, for some use cases, especially in industrial applications, further optimization may be required.
User ratings (averaging 4.3/5.0) confirm the practical applicability of the developed system but also indicate areas for potential improvements, especially in the context of interaction naturalness and complex object generation accuracy.
This research presents an innovative approach to creating and managing virtual images in immersive environments using text and speech interfaces. The main achievements include: 1. Development of an effective speech recognition system with high noise resistance and adaptability to different users. 2. Creation of a virtual image generation algorithm capable of accurately interpreting text and speech descriptions with support for various complexity levels. 3. Successful integration of the developed system with VR/AR/MR environments, providing low response latency and high performance.
The conducted experiments confirmed the effectiveness of the proposed approach, demonstrating high accuracy and performance indicators. Nevertheless, the research also revealed several directions for further development, including: Optimization of generation algorithms for more complex scenes and objects Improvement of user interaction naturalness with the system Enhancement of semantic analysis capabilities for more accurate interpretation of user descriptions
The research results create a solid foundation for further development of intuitive interfaces in immersive technologies and open new possibilities for their practical application in various fields.
The author(s) have not employed any Generative AI tools.