Voice disorders constitute a significant health concern, with an annual prevalence of approximately 7% among the adult population, adversely afecting patients' quality of life, encompassing both social and occupational functioning. Also, the majority of diagnostic methodologies continue to depend on invasive techniques, whereas non-invasive automated diagnostic approaches have not been extensively investigated yet. This study introduces a transformer-based method for detecting voice disorders aimed at enhancing detection eficacy through a multimodal fusion strategy. Specifically addressing two distinct types of voice recordings - extracted from sentences reading and vowels emissions -- we devised and assessed five multimodal fusion strategies across three stages: early, mid, and late. Our experimental findings indicate that the cross-attention mid-fusion method harnesses the benefits of both data types, and it achieves a detection accuracy of 0.885 and a macro F1 score of 0.843 on an internal dataset. These results represent an improvement of +.03 to +.06 in accuracy and +.02 to +.05 in macro F1 score when compared to unimodal models (trained on sentence or vowel data only). This study represents an advancement for an efective non-invasive detection of voice disorders and provides insights for clinical practice.
Voice disorders have a significant impact on people’s lives,
with 7% of adults sufering from them each year. They can
lead to communication dificulties, reduced work
productivity (7.4 work days lost per year on average), and even career
changes, with 4% of patients reporting a career change due
to voice problems [
Although traditional diagnostic techniques such as
laryngoscopy and speech assessment are widely used clinically,
they have significant limitations [
The development of artificial intelligence technology [
The rest of the paper is organized as follows. In Section 2 we first review the relevant research on voice disorder detection and analyze the main challenges of existing methods in application. Section 3 describes the proposed Transformerbased method in detail, focusing on diferent multimodal CEUR
ceur-ws.org fusion strategies. In Section 4 we introduce the experimental setup and evaluation methods, while in Section 5 we provide a comprehensive analysis and interpretation of the experimental results. Finally, in Section 6 we summarize the significance of our research, explore potential limitations, and provide suggestions for future research directions.
This section reviews the relevant research on voice disorder detection and provides a theoretical basis for the tools and methods used in subsequent sections. The discussion focuses on the evolution of voice feature analysis techniques, the application of classifiers in detecting voice disorders, and the latest progress in data augmentation and fusion models.
Traditional voice disorder detection methods rely on
artiifcial feature engineering, that is, extracting acoustic
features such as Mel-frequency cepstral coeficients (MFCC),
pitch jitter, and amplitude shimmer from speech signals
[
The advent of deep learning has transformed voice
disorder detection, as it can automatically extract features from
raw speech signals. Unlike traditional methods that rely on
handcrafted features, deep learning models such as
convolutional neural networks (CNNs) and recurrent neural
networks (RNNs) can learn more abstract and comprehensive
feature representations directly from data [
Recently, transformer-based architectures have made
breakthroughs in automatic speech recognition and related
tasks [
In voice analysis, multi-modality refers to input data
extracted from diferent data sources or forms of information
[
In multimodal fusion, there are three main strategies:
early fusion, mid-level fusion, and late fusion [
The research by Koudounas et al. [
Building on this, our study introduces a systematic study
of multimodal fusion strategies in voice pathology
detection. We focus on early, mid, and late fusion methods,
especially mid and late fusion, because these two methods
have greater flexibility and can capture deeper correlations
between modalities. Compared with the method of [
This section outlines our contributions to multimodal fusion strategies, emphasizing the mathematical formulation of the problem and the model architecture. Specifically, we introduce early, mid-level, and late fusion strategies in transformer architecture that integrate multiple modalities for robust prediction.
In this study, we used two speech-based modalities to
solve the voice pathology detection task, each capturing
voice characteristics. The first modality 1 represents the
original features extracted from the sentence reading
recording, while 2 represents the features extracted from the
second modality, the sustained vowel pronunciation recording.
Given a multi-modal architecture , we input the raw audio
waveforms into the Wav2Vec2 model [
ℎ2 = 2( 2)
= [ℎ1; ℎ2] =̂ softmax (( )) spectively.
modalities. • ℎ1 and ℎ2: high-dimensional embeddings extracted from modalities 1 and 2 using CNN extractor, re• : Concatenated feature embeddings from both • represents the transformer encoder layers. • : a 1-second silence padding between the two
The early fusion strategy connects the raw features of the two modalities into a unified input representation. In this stage, we first truncate all audio samples to a uniform length to standardize the input length and eliminate the bias caused by the diference in sample length. Then, we directly concatenate the raw audio features from the two modalities. Specifically, the raw audio features from the two modalities are concatenated in a fixed order: modality 1 ifrst, then modality 2. To further distinguish the two of them, a 1-second silence ( ) padding is inserted between the two, providing a clear boundary for the model (Eq.2). After concatenation, the generated unified features are fed as input into the pre-trained Wav2Vec2 model for prediction. Fig.1 visually depicts this process.
= [ 1; ; 2] =̂ softmax ( ( )) (1) (2)
In the mid-level fusion strategy, feature fusion is performed after CNN encoding but before the features are fed into the transformer encoder. This approach combines modalityspecific features in a shared representation space, allowing the model to leverage interactions between modalities for more robust predictions. We will analyze two diferent fusion strategies: concatenated embedding and crossattention.
In the concatenated embedding strategy, features are first extracted from each modality using a separate CNN layer and mapped to the same vector space (Eq.3). We thus decompose the network into the composition of two modules = ∘
, where is the CNN-based feature extractor, while
represents the transformer encoder layers. After feature
(3)
(4)
(5)
(6)
(7)
(8)
The cross-attention mechanism [
First, given input feature matrices ℎ1 and ℎ2 of the two modalities, we generate , , and through linear transformation,
= ℎ 1 , = ℎ 2 , = ℎ 2 Here, , , and are learnable weight matrices for the query, key, and value, respectively. Next, we calculate the attention matrix between the Query ( ) and the Key ( ) by measuring their similarity, then normalized using softmax. The attention weight is used to perform a weighted sum of the Value to generate output features : = softmax ( =
√ ⊤ ) where: • is the general attention matrix.
tion factor used for scaling.
• is the dimension of the key, √ is the normalizaAs illustrated in Figure 3, cross-attention is computed in both directions to efectively capture interactions between the two modalities.
1. We use ℎ1 as the query and ℎ2 as the key and value to compute the attention (Eq.9). 2. We reverse the roles of the modalities and use ℎ2 as the query and ℎ1 as the key and value (Eq.10).
Finally, the outputs of the cross-attentions from both directions, 1→2 and 2→1, are concatenated to form a unified representation, fused, as shown in Eq.11.
This concatenation helps to merge the information from both modalities in a unified feature space. The fused features are then processed through a shared fusion layer before passing them to a transformer encoder for deeper feature extraction and ultimately classification.
1→2 = 1→2 2 = CrossAttention(ℎ1, ℎ2) 2→1 = 2→1 1 = CrossAttention(ℎ2, ℎ1) fused = [ 1→2; 2→1] =̂ softmax (( ) (9) (10) (11) (12)
While mid-level fusion captures fine-grained interactions between modality-specific embeddings, late fusion is performed at the decision level, allowing each modality to be optimized independently and then integrated into a unified prediction. This approach allows each model to focus on its specific modality before being integrated, although it doubles the size of the final model. Two late fusion techniques are employed in our study.
Simple average In this approach, the outputs of the two models, 1̂ and 2̂, are combined by taking their simple average, as illustrated in the top part of Figure 4. This strategy assumes that both models contribute equally to the ifnal prediction. The combined output l̂ate1 is computed as follows:
l̂ate1 = 21 ( 1̂ + 2̂) (13) where 1̂ and 2̂ are the probability distributions produced by the two individual models.
This fusion method is simple and it is computationally eficient as it avoids any extra parameters.
Mixture of Expert As a second late fusion strategy, we employ a shallow mixture of experts (MoE) to combine the outputs of two independent models and improve the overall performance of the system. Unlike the simple averaging method, this approach assigns weights to each model’s predictions based on how relevant they are to the final output.
As shown in Figure 4, we use a simple multi-layer perceptron (MLP) configured with a single hidden layer to predict weights to combine the outputs of each model. The input layer of the MLP is a probabilistic concatenation of the two modalities ( 1̂, 2̂), and the output layer applies a softmax function to ensure that the sum of all model weights is 1 (Eq.14). During inference, the final prediction is computed using the weights to combine the contributions of both models (Eq. 15). This approach improves the system’s performance on unseen data while maintaining a simple architecture.
[ 1, 2] = softmax (MLP([ 1̂ ; 2̂])) l̂ate2 = 1 ⋅ 1̂,test + 2 ⋅ 2̂,test (14) (15) Here: • [ 1, 2]: Weights learned from the concatenated outputs 1̂ and 2̂ on the validation set. • 1̂,test, 2̂,test: Predicted probabilities from the two models on the test set.
This section provides an overview of the datasets and preprocessing methods used in our experiments, followed by a detailed description of the training setup to ensure reproducibility.
All experiments were conducted in a cloud-based environment equipped with a Tesla P100-PCIE-16GB GPU1.
Details of the software environment can be found in the project repository2.
IPV The Italian Pathological Voice (IPV) dataset is a novel
and diverse resource designed specifically for voice
pathology research, currently unpublished and introduced in [
The dataset comprises two modalities: sustained
phonation of the vowel /a/ (SV) and reading of five phonetically
balanced sentences (CS) adapted from the Italian version
of CAPE-V [
Audio Preprocessing To ensure the consistency of audio duration and facilitate comparison, we cropped the samples in the datasets to fixed lengths: CS samples were cropped to 19 seconds, and SV samples were cropped to 18 seconds. These lengths are designed to cover approximately 90% of the samples in each modality, ensuring that most voice 1We gratefully acknowledge the computational resources provided by Kaggle (https://www.kaggle.com/) for this research. We also appreciate the early-stage support from HPC@Polito (http://www.hpc.polito.it). 2Github repository: github.com/multimodal_pathologies_prediction information is preserved for efective model training while reducing the impact of outlier samples that are too long. For samples shorter than the fixed lengths, zero-padding was applied to extend them to the required duration.
Then, the audio data was standardized using the predeifned processor provided by the Wav2Vec2 framework. The processor first resamples the audio data to 16kHz to ensure compatibility with the framework, and reduce computational overhead. Then the converted feature representation can not only efectively capture the key information in the speech signal, but also provide consistent and eficient input features for the model to support subsequent training tasks.
In order to avoid issues with the imbalance of pathological voice data (healthy samples are less than pathological samples), a stratified sampling method was used in the data division process to ensure proportional representation of healthy and pathological samples across all splits. We divided the data into training, validation, and test sets in a ratio of 8:1:1 to ensure fair and reproducible evaluations. The test set was first separated using a fixed random seed. Subsequently, the training and validation sets were further split using three diferent random seeds to create multiple splits. The final results are calculated by averaging the performance metrics over these splits to ensure the robustness and reliability of the evaluation.
To verify the efectiveness of our proposed method and
provide a comparison, we designed a series of traditional
baseline models, including the classic multi-layer
perceptron (MLP) and a lightweight convolutional neural network
(MobileNetV2 [
In the feature extraction process of the baseline model, the audio data is uniformly sampled to 16kHz and truncated to a fixed maximum duration to ensure sample consistency. We extract 40-dimensional MFCC features through librosa, transpose them into a time-step sequence form, and uniformly zero-fill the feature sequence. At the same time, a padding mask is generated to distinguish between real data and padding parts. The following is the specific design of the two baseline models.
MLP is designed with two fully connected layers
containing 50 hidden units, using the ReLU activation function to
extract high-dimensional features, aggregating the time
dimension information through the global average pooling
layer, and finally performing binary classification through
the Softmax output layer. The training process uses the
Adam optimizer with a learning rate of 0.01, a batch size of
16, and an early stopping strategy to prevent overfitting.
2D-CNN The audio features are converted to 2D images
by repeating a single channel to RGB three channels to fit
the input requirements of the pre-trained model. We load
the pre-trained weights (ImageNet [
Our method is based on a pre-trained Wav2Vec2.0 model (trained on the LibriSpeech 960-hour dataset) and evaluates three fusion strategies on the IPV dataset: early fusion, midlevel fusion, and late fusion.
Early fusion is accomplished by directly concatenating the original audio of CS and SV, and adding 1 second of silence (38 seconds) after the total length of the audio to avoid feature loss. The concatenation is performed on the same individual. The concatenated audio signals are uniformly processed in a Wav2Vec2.0 processor to ensure consistency in feature extraction. Mid-level fusion is based on 2 finetuned Wav2Vec2.0 models, and global feature modeling is achieved through a shared Transformer encoder (initialized with sentence reading, SV to a single modality with vowel pronunciation. Values spanning both columns refers to modality fusion methods. Bold values indicate the best performance for a given metric.
Modality Single Multi
Method MLP Wav2Vec2 Early Fusion 2D-CNN (Train all layers) 2D-CNN (Fine-tune classify head) Mid (Concatenated Embeddings) Mid (Cross Attention) Late (Simple Average) Late (MoE)
Accuracy
Macro F1
CS .801±.011 .667±.011 .789±.019 .859±.029
SV .750±.057 .673±.000 .782±.048 .827±.000
CS .767±.022 .400±.004 .765±.021 .837±.038
SV 10 hidden nodes) that determines modality weighting based on the probabilities from the training and validation sets.
All experiments above were completed within 50 training rounds (epochs), and using fixed random seed to ensure the reproducibility of the results. The AdamW optimizer (weight decay = 0.01) was used for all experiments. A linear learning rate scheduler is used to optimize the learning rate adjustment. The scheduler reduced the learning rate linearly over the total number of training steps, with no warm-up steps. Initial learning rates are optimized by manual adjustment, using 1e-5 for single modality and concatenated fusion and 6e-6 for cross-attention fusion. To address class imbalance, a weighted cross-entropy loss function was applied, with class weights computed based on the training dataset’s label distribution. The batch size was set to 8, and an early stopping strategy with the patience of 10 epochs was used to terminate training when the validation performance plateaued. More experimental details and hyperparameter configurations can be found in the GitHub repository of the article.
To evaluate the performance of the model in the voice disorder detection task, we used two key metrics:
Accuracy measures the proportion of correctly predicted samples to the total number of samples, providing an overall assessment of classification performance: Accuracy =
Number of Correct Predictions
Total Number of Samples (16) While accuracy is a useful general metric, it can be less informative in imbalanced datasets.
To better evaluate performance across imbalanced classes, we adopted the macro-average F1 score, which calculates the F1 score for each class and then averages them: 1 = 2 ×
Precision ⋅ Recall Precision + Recall
In this section, we analyze and interpret the experimental results by focusing on two key aspects: comparing baseline models to assess their efectiveness as reference, and evaluating diferent fusion models to explore their ability to integrate multimodal information and improve generalization to unseen data. By systematically studying these factors, we aim to highlight the strengths and limitations of the proposed approach and provide insights for future improvements.
of the performance of unimodal baseline models for voice disorder detection on the IPV dataset.
As expected, Wav2Vec2 achieved the best results among the four baseline models, with accuracy of .859 and .827 in CS and SV modes, respectively, and .837 and .793 for F1 macro, respectively. The superior performance of Wav2Vec2 underscores the benefits of self-supervised pre-training on large-scale audio data. This means that the model does not need to be trained from scratch, but through pre-training and transfer learning capabilities, it can have audio features with good generalization capabilities, even with a small amount of labeled data. Moreover, it benefits of the attention mechanism which better extract relevant features from long sequence of data.
The MLP model performs well in CS mode with an accuracy of .801 and F1 Macro of .767, but drops to .750 and .686 in SV mode, highlighting its limitations in capturing complex audio features with limited contextual information. Compared to MLP, our method improves F1 Macro by +.07 in CS mode and +.10-.11 in SV mode, with corresponding accuracy improvements of +.05-.06 and +.07-.08.
For the 2D-CNN, fully fine-tuning all layers leads to poor performance (.667 and .673 accuracy in CS and SV modes, respectively; .400 and .402 F1 Macro), likely due to the disruption of pre-trained features. Fine-tuning only the classification head improves the performance to .789 and .782 accuracy in CS and SV modes, and .765 and .723 F1 Macro, respectively. However, our method performs better than the fine-tuned 2D-CNN, with +.07-.08 improvement in F1 Macro, +.07-.08 and +.04-.05 improvement in accuracy in CS and SV modes.
The above results show that fine-tuning the pre-trained Wav2Vec2 model is an efective solution for small dataset tasks, it highlights the necessity of carefully designed optimization methods.
fusion method shows an advantage over the single modality by efectively combining the complementary information of CS and SV inputs. In particular, as shown in Table 2, the proposed mid-level fusion pipeline shows significant improvements over single modality models. Concatenated Embeddings improves accuracy by +.02 and macro F1 by +.001 on the CS model, and by +.05 and +.04 on the SV model, respectively. Cross Attention performs even better, with accuracy and F1 gains of +.03 and +.006 on the CS model, and +.06 and +.05 on the SV model for accuracy and macro F1, respectively. These results highlight the benefits of leveraging complementary information from multiple modalities.
When compared to other fusion strategies, instead, midlevel fusion consistently outperforms both early and late fusion methods. The cross-attention method achieves the best results with .885 accuracy and .843 macro F1, which is +.02-.03 in accuracy and +.01-.02 in macro F1 compared with early fusion. Similarly, it achieves +.01-.04 improvement in accuracy and +.01-.03 improvement in macro F1 compared to late fusion strategies such as Mixture of Experts (MoE). These results demonstrate the efectiveness of dynamically capturing inter-modality dependencies during feature integration.
Compared with early fusion that concatenates raw features, the proposed mid-level fusion method can model complex inter-dependencies, leading to robust feature representation. In contrast, late fusion methods, while simpler to implement, operate at the decision level and cannot fully exploit the interactions between modalities.
In summary, our proposed mid-level fusion strategy, especially the cross-attention strategy, achieves the best performance among all methods. The results show that it is able to dynamically integrate complementary modality information, leading to significant improvements in accuracy and macro F1 performance.
This study investigates the efectiveness of various models, and fusion methods for speech impairment detection using unimodal and multimodal approaches. We leverage endto-end pre-trained models Wav2Vec2, which is once again proven to be an efective model for solving audio tasks, even with a limited dataset size. This not only reduces the steps of manual feature extraction but also enables robust features to be extracted from audio data through self-supervised pre-training, showing good generalization ability.
Among multimodal methods, our experiments show that mid-level fusion strategies, especially the cross-attention mechanism, outperform early and late fusion techniques. The cross-attention mechanism dynamically captures finegrained inter-modal dependencies, leading to the highest performance. In contrast, early fusion methods, while beneifcial for capturing joint features from the beginning, may lack flexibility in handling complex interactions between modalities. This often leads to inferior performance compared to mid-level fusion. Late fusion methods are easier to implement but have limited capabilities in modeling complex feature interactions and a higher number of parameters.
These findings provide valuable insights into the design of voice disorder detection systems, especially with regard to their potential applications in clinical diagnosis and health monitoring.
Future Work Although this study provides valuable insights, there are still some limitations and directions for improvement.
First, the experiments are limited to a specific dataset, IPV, which contains two homogeneous audio modalities and cannot cover a wider range of scenarios. Future work can explore larger and more diverse datasets, including datasets collected in realistic noisy environments, or cross-lingual datasets to evaluate the reliability of the model in the real world. In addition, future work can integrate other medical modalities (e.g. laryngoscope images + audio samples), to expand audio beyond the audio domain for more comprehensive voice disorder detection. Second, the current study only focuses on voice disorder detection tasks. In the future, it can be expanded to multi-classification tasks to more comprehensively evaluate the efectiveness of the model in practical applications, especially in the classification of diferent types of pathologies that are at the root of the voice disorder.
Third, we only used the wav2vec2 model for feature extraction and multi-modal fusion, and did not compare it on other advanced Tansformer models (e.g. Hubert, WavLM, etc). Future work can explore and evaluate their efectiveness in the medical voice pathology analysis of these models.
Data augmentation techniques can also be combined to enhance the generalization ability of the model, so as to maintain excellent performance in more diverse application scenarios.
By addressing these limitations, future research can build on this study to develop more powerful, eficient, and scalable voice disorder detection solutions, thereby bringing greater social and technological impact for practical applications.