The COVID-19 pandemic has accelerated the shift of many meetings to online platforms. In crucial multi-person meetings, real-time speech separation can significantly enhance subsequent applications. Additionally, real-time speech separation in various speech-related products, such as hearing aids, can improve backend services. However, achieving real-time speech separation poses substantial challenges. It requires segmenting speech over time, and as the information within each segment decreases, the dificulty of source separation increases, along with the necessary computational time. Most previous research has focused on non-real-time speech source separation, and real-time systems often rely on additional information or resources, primarily utilizing English corpora as the dataset. This research aims to develop a real-time speech source separation system. We use the architecture proposed in [1] as our main framework and modify it to handle segmented speech processing. The system is designed to perform real-time computations using a single-core CPU. The scale-invariant signal-to-noise ratio (SI-SNR) is employed as the evaluation metric for objective assessment. We validate the performance of our method with a small dataset in a real-time system and simulate the speech segmentation of the real-time system ofline using a large dataset.
Sound is vital for human perception and communication, carrying useful information such as scene, speaker, and language. It has many applications like hearing aids, speaker recognition, and speech recognition. Therefore, computer processing of speech signals is a popular research topic.
Source separation is a field that many researchers have delved into with abundant resources,
but the signal processing methods used vary. The traditional techniques mainly include
beamforming [
Combining the research from both fields, more and more researchers hope to use
timedomain signals for source separation. The paper [
The dual-path recurrent neural network (DPRNN) method in [
To achieve real-time source separation, methods like multi-input-multi-output (MIMO) require
spatial information, increasing costs and resources. Real-time tracking using upper-body
humanoid robots and distributed processing, as seen in [
Building on [
For the evaluation task, we utilized the Taiwan Mandarin Chinese version of the Hearing in
Noise Test (TMHINT) [
A total of 19 participants, comprising 12 males and 7 females aged between 20 and 28 years, took part in the audio recording for this experiment. None of the participants exhibited noticeable pronunciation dificulties or issues with oral expression, thus representing typical speech conditions.
During the recording process, the equipment, including the laptop array microphone and a standard microphone, was first calibrated for frequency response and to assess the impact of environmental noise on the recording results. The recording took place in a semi-reverberant room measuring 2m x 2m x 2m. The participants sat facing a computer, with the standard microphone positioned at mouth height to minimize distance variation due to diferences in participant height. We used Audacity software to control the laptop array microphone and ACQUA software for the standard microphone, both operating at a 48kHz sampling rate. Each participant read the 320 phrases from the TMHINT text sequentially. Each phrase lasted three to four seconds, with a one-second pause between phrases. After reading all 320 phrases, the recordings were stopped in both Audacity and ACQUA, and the files were saved and checked for errors to complete the session.
In the data processing stage, we addressed issues of uneven speech volume and varying speech duration. To ensure consistency, the entire corpus was recorded in one session to avoid variations due to diferent recording times or changes in speaker conditions. We inserted a blank interval of approximately 1.5 seconds between each sentence to facilitate audio segmentation. We then extracted 320 individual audio files based on these intervals. Voice activity detection (VAD) was performed on the data to remove silent parts, preventing excessive silence in the clips that could negatively impact model training. We manually verified the efectiveness of the VAD method in eliminating silent portions.
In this paper, we use the DPRNN as our main reference. Figure 1 illustrates the DPRNN process. Initially, an audio clip is input and converted into features that are easily learned by the model through an encoder. The audio then enters the separation model, which is divided into three parts: segmentation processing, DPRNN block processing, and overlap-add.
In segmentation processing, the encoded audio features are broken into block-like segments and connected to form a three-dimensional tensor. Here, represents the time length, is the feature-length, is the input signal, and is the length of one block. A 50% overlap rate was used in this study. represents the number of blocks generated during this process, resulting in a 2 * * three-dimensional tensor.
After obtaining the three-dimensional tensor, DPRNN block processing is performed, which is divided into Intra-chunk RNN and Inter-chunk RNN. The Intra-chunk RNN processes local information within each block in parallel, while the Inter-chunk RNN processes global information across all blocks. Data in the Intra-chunk RNN undergoes a Bi-LSTM, a fully connected layer, and a normalization layer. The Inter-chunk RNN follows the same procedure. These two components combine to form a complete DPRNN block, which can be stacked to increase model depth. In our experiments, we used six stacking layers.
The purpose of overlap-add is to convert the data processed by the DPRNN block back to the input format. As shown in Fig 1, the three-dimensional output of the last DPRNN block is transformed back to a sequence output through overlap-add.
For generating training data, we randomly selected ten males and six females from the previously recorded data, totaling sixteen participants. The training data had a sampling rate of 8kHz. The data from these participants were mixed and divided into three combinations: male-male, male-female, and female-female, with 13,500, 10,000, and 6,500 sentences, respectively. The generation of each combination followed these steps: (1) Two diferent people were selected from the sixteen, each with 170 sentences of speech signal. (2) One sentence was randomly selected from each person’s 170 sentences, resulting in two speech signals. (3) A signal-to-signal ratio between − 5 to 5 was randomly selected, and the two speech signals were mixed to obtain a mixed speech signal. (4) Steps (1) to (3) were repeated to obtain a training corpus of approximately 30 hours, serving as a small training database according to the quantity of each combination.
The test set included four trained speakers and four untrained speakers, each with two males and two females. The test data generation followed these steps: (1) Three gender combinations were used for mixing: male-male, male-female, and female-female. (2) Two speakers were selected within each gender combination, each with 150 speech signals, diferent from the training set. (3) One sentence was selected from each speaker’s 150 speech signals, resulting in two speech signals for each gender combination. (4) The two selected speech signals were mixed at a signal-to-signal energy ratio of 0.5 to obtain a mixed speech signal. (5) Steps (2) to (4) were repeated until each gender combination contained 2000 sentences, resulting in a small test set of 6000 sentences. The number of training and testing data is listed in Table Table 1.
F-F 6500 2000
Total 30000 6000
In this paper, we initially used a mobile phone to play the audio, which we then recorded directly using the computer’s microphone. However, this method resulted in the system capturing audio with doubled environmental noise and potentially uneven volume due to human factors during the recording process. To address these issues, we decided to use the mobile phone as the input device for the real-time system, connecting it to the computer via an audio cable. This setup allowed us to avoid the aforementioned problems. In recording mode, the system saves the entire separated audio file to the computer’s hard drive at the end of the execution. In playback mode, after separating a segment of the audio, the system plays it in real time through the speakers.
As shown in Fig 2, the experimental steps are as follows: (1) Data collection and preprocessing of audio files. (2) Simulation and computer validation of source separation algorithms. (3) Optimization of algorithm performance. (4) Development of time-domain source separation techniques. (5) Development of real-time system simulation and objective evaluation standards. (6) Real-time system performance optimization.
To efectively evaluate the performance of the system, we further expanded it with two modes: recording mode and playback mode. In the recording mode, the system executes the complete process and generates audio files, which are then saved to the computer’s hard drive. In the playback mode, the system is capable of playing the separated audio in real-time, allowing users to assess its performance immediately. The application of these two modes enables a more comprehensive evaluation of the system’s performance while providing users with a better experience.
To meet the requirements of real-time computation, we cannot input the complete audio file into the model for processing and output the results. Therefore, the audio files need to be segmented, resulting in a reduction in the amount of data input to the model. This reduction in data volume significantly afects the separation performance of the model.
Therefore, we made some improvements to the DPRNN method. We studied the model 3 ) z H k ( y 2 c n e u q e r 1 F 0 3 ) z H k ( y 2 c n e u q e r 1 F 0 0.5 1 1.5 2 0.5 1 1.5 2
Time (s) Time (s) (a) Original model speaker1.(b) Improved speaker1.
0.5 1 1.5 2 0.5 1 1.5 2
Time (s) Time (s) model (c) Original model speaker2.(d) Improved speaker2.
model architecture of DPRNN, where the original DPRNN model first segments an input audio file into several chunks and then stacks them sequentially. The x-axis represents the length of each segmented audio file, and the y-axis represents the number of segmented audio files. The stacked segmented audio files are then fed into the RNN block of the DPRNN, which consists of two parts: intra-chunk operations based on the temporal sequence of the inputs and interchunk operations across the segmented audio files. In the context of real-time computation, the audio is segmented before each model inference, resulting in short segmented audio files being processed each time. As a result, the efectiveness of the inter-chunk RNN (Inter chunk) that operates across time on the segmented audio files is significantly reduced. In situations where inter-chunk RNN does not provide significant benefits, we choose to omit it. This saves half of the computational workload and reduces the impact of audio file segmentation on the system. Finally, considering model performance and computational time trade-ofs, we settled on a model architecture with 4 layers of intra-chunk RNN.
In addition to the 4 layers of intra-chunk RNN, we believe that even though the inter-chunk RNN may not provide substantial help when the audio files are segmented and stacked, having a single layer of inter-chunk RNN can still aid the model in feature learning. Therefore, based on the architecture of 4 layers of intra-chunk RNN, while maintaining its computational workload, we replace the last layer of intra-chunk RNN with inter-chunk RNN. The final architecture consists of 3 layers of intra-chunk RNN plus 1 layer of inter-chunk RNN.
The original full model, consisting of 6 layers, had a size of 31MB. We adjusted the model architecture to 3 layers of intra-chunk plus 1 layer of inter-chunk, achieving real-time computational requirements, and reducing the model size to approximately 10MB.
Fig 3 shows the spectrograms of the results before and after model adjustment. It can be observed that there is little impact on the results before and after model adjustment, and the spectrograms of both still exhibit a high degree of similarity. 3.3. Computer hardware and software connections in the system In the system, it is necessary to establish connections between the model and external audio sources. To achieve this purpose, we utilize a tool called "pyaudio". Pyaudio is an open-source Python tool developed by a third party, providing capabilities for recording, reading audio files, and playing audio files within Python. The use of this tool enables us to easily apply audio sources and transmit audio data to our system for further processing.
One of the advantages of pyaudio is its cross-platform compatibility. It supports various operating systems, including Linux, Microsoft Windows, and Apple macOS, making our system theoretically capable of functioning across diferent operating system environments.
With pyaudio, we can efortlessly set up recording or playback functions for audio sources and connect audio data to our model through appropriate configurations. This facilitates an efective linkage between audio source input and model separation, equipping the system with essential tools and functionalities for smooth operation.
Overall, the utilization of pyaudio allows us to seamlessly handle audio within the Python environment and connect it to our model. This empowers our system with enhanced capabilities and flexibility, enabling us to achieve the goals of eficient and accurate audio separation and real-time playback.
Furthermore, while using Pyaudio, it’s important to note that the precision of audio data input is in 16-bit format, whereas the model’s computation requires data in Float32 format. Hence, before feeding data into the model for computation, a conversion from 16-bit to Float32 format is necessary to ensure proper data processing by the model. Once the model’s computation is complete, the results must be converted back to a 16-bit format for audio data storage using pyaudio.
4.1. SI-SNR
The Scale-Invariant Signal-to-Noise Ratio (SI-SNR)[
:= ˆ − ⎪⎪⎩ − := 10 ‖‖2 ‖‖2 (1)
Since calculating SI-SNR requires aligning every point of the two audio signals, there may be human errors in real-time systems, such as from the start of the system to playing the audio ifle, or from the end of the audio file to the end of the system, which can lead to imprecise separation of the audio files. Therefore, we propose a dual-channel engineering method to address this issue. Fig 4 shows the core of this method. That is the dual-channel approach that simultaneously mixes the target audio file with the mixed audio file and changes the system’s input to the processed dual-channel audio file. This ensures that the blank spaces in the output audio file and the target audio file are of the same length, thereby achieving alignment.
The reason for doing this is to align the timing of the mixed speech and the target speech. At the same time, we want the input and output of the system in recording mode to be a dual-channel audio file. Therefore, we first merge the mixed speech and target speech A into a dual-channel audio file, where the first channel is the mixed speech and the second channel is the target speech A. This ensures that both enter the system simultaneously and are output simultaneously, with aligned timing. However, once inside the system, only the first channel, which is the mixed speech, enters the model for separation. After going through the model, we obtain a separate speech A, while the target speech A does not enter the model. Before outputting, we mix these two speech segments back into a dual-channel audio file.
In Table 2, we conducted experiments using the audio files mentioned earlier, analyzing malemale, male-female, and female-female combinations. The DPRNN model achieved an average SI-SNR=16.7dB, with male-female pairs performing the best at SI-SNR=17.9 dB. We speculate that this is due to the greater diference in frequency range between male and female voices, making it easier for the model to separate the two.
Additionally, we added two types of noise, computer fan noise, and pink noise, to the training and testing data used in a clean background environment. The noise was mixed into the data at SNRs of 0, 5, and 10. We randomly selected one type of noise and one SNR and mixed it into 30,000 sentences of training data and 6,000 sentences of testing data. We trained the model with the noisy data, changing the output from two speakers to three, including two speakers and the noise. The purpose of this was to treat the noise as a third speaker for separation.
We tested the model on audio files in both noisy and clean background environments, with an average SI-SNR of 14.9 in the noisy environment. Male-male, male-female, and female-female pairs achieved SI-SNRs of 15.5, 15.8, and 13.2, respectively.
Real-time System Performance in a Small Dataset: We selected five sentences from the clean test data to conduct small data testing on a single-core CPU. The selection criterion was based on an average SNR of 16.7. We chose two sentences above the average SNR, two sentences below the average, and one sentence approximately equal to the average, resulting in a total of five sentences. As shown in Table 3, we tested the impact of diferent input sizes on the system’s performance. The input sizes were 800 and 2000 points per input. It can be observed that the smaller the input size, the poorer the system’s performance. This is because our main method is based on LSTM, which is highly influenced by the length of the input sequence. If the information available for each model processing is reduced, its performance will be adversely afected.
Due to the limitation of obtaining only one audio segment at a time in the real-time system, evaluating the performance with a large amount of data would be time-consuming. Therefore, we conducted ofline simulations by manually segmenting the audio files to mimic the segmentation approach in the real-time system. These segmented audio files were then processed through the model in separate batches, and the resulting segmented audio segments were concatenated to form a complete sentence. We compared five diferent models: a 6-layer full model, a 4-layer full model, a 2-layer full model, a 4-layer intra-chunk model, and a 3-layer intra-chunk with a 1-layer inter-chunk model, running on a single-core CPU. We tested the input lengths of 800 samples and 2000 samples. Table 4 shows that the adjusted models outperformed the original DPRNN model significantly. Compared to the 6-layer full model, the 4-layer intra-chunk model achieved a 0.74 dB improvement in SI-SNR for the input length of 2000 samples, and a decrease of 0.15 dB in SI-SNR for the input length of 800 samples. It also showed improvements compared to other full models. In the 3-layer intra-chunk with the 1-layer inter-chunk model, there was a 2.28 dB improvement in SI-SNR for the input length of 2000 samples compared to the 4-layer intra-chunk model, and improvements compared to other models for the input length of 800 samples. Therefore, we concluded that when the audio is segmented in time and the input length is shorter, adding a significant number of inter-chunk layers would lead to a decrease in model performance. However, not adding any inter-chunk layer would also not achieve the best performance. Thus, the 3-layer intra-chunk with 1-layer inter-chunk model exhibited the best performance in both input length scenarios.
This study focuses on developing a real-time speech source separation system in a single-core CPU computing environment. First, we validate the performance of the complete DPRNN model on Chinese speech data and in the presence of noise backgrounds, demonstrating its feasibility. Next, we modify the DPRNN model architecture to meet the requirements of a realtime system. In a real-time system, the input audio needs to be segmented over time. Therefore, we change the original model’s structure to a 3-layer intra-chunk plus 1-layer inter-chunk architecture, which satisfies the computational time constraints of a real-time system using a single-core CPU. The objective evaluation criterion used is SI-SNR. The 3-layer intra-chunk plus 1-layer inter-chunk model architecture performs best even with an input length of 2000 samples. Additionally, we develop a dual-channel engineering approach to evaluate the performance of the real-time system under a small dataset scenario. Again, the 3-layer intra-chunk plus 1-layer inter-chunk model architecture achieves the best performance evaluation results with an input length of 2000 samples. Furthermore, the 3-layer intra-chunk plus 1-layer inter-chunk model architecture meets the real-time computational time requirements in both single-core CPU and GPU computations.
However, the current performance is maintained when the input length is 2000 samples, which corresponds to a minimum of 250 milliseconds (ms) processing time. In applications that require audiovisual synchronization, this delay is noticeable to users. Therefore, future research can focus on achieving a balance between performance and reducing the input length per model inference, aiming to minimize latency.
The authors would like to thank the financial support from the Ministry of Justice (113-1301-0504-01) and the National Science and Technology Council (NSTC112-2221-E-155-017-MY2 and NSTC112-2222-E-155-002).