Speech models often struggle with performance inconsistencies across diferent subgroups, leading to degraded accuracy for certain speaker demographics, accents, or recording conditions. These discrepancies may originate from multiple reasons, such as imbalanced training data, suboptimal representation learning, and limitations in model generalization. Addressing these issues allows for improving model robustness and reliability in real-world applications. We propose to mitigate performance disparities of subgroups that underperform, i.e., exhibit daivergence, relative to overall model performance. We tackle the performance disparities both via in-processing solutions, i.e., implementing mitigation measures during model development, and a post-processing one, refining already trained models. As in-processing solutions, we propose three approaches: divergence-aware regularization, targeted data augmentation, and contrastive learning (CLUES). Each method improves model learning in diferent ways: divergenceaware regularization adjusts training to focus on low-performing subgroups, targeted data augmentation generates synthetic variations to enhance model robustness, while CLUES refines latent representations. The post-processing strategy introduces a divergence-aware data acquisition method to prioritize acquiring real-world samples from underperforming subgroups.
CEUR ISSN1613-0073
Speech models are widely used in modern applications, including virtual assistants, transcription
services, and accessibility tools1,[
Various methods have been proposed to address these challenges. Many approaches rely on
manually identifying specific speech characteristics that might cause performance issues1[
Recent research has explored automated subgroup identification, using clustering techniques to detect data patterns where models underperform12[]. While these data-driven approaches help identify performance gaps, they often lack interpretability and do not clearly describe the underlying problems. Consequently, they do not provide insights into the specific sources of performance inconsistencies nor guidance for data acquisition for model improvement.
Our paper presents a framework addressing these limitations through four complementary
methods. We propose to mitigate the performance disparities within data subgroups that deviate
significantly, i.e., exhibit a divergence, from the overall model performance. We propose both
post-processing and in-processing approaches. For post-processing, i.e., improving already
trained models 2[
To evaluate these methods, we conduct experiments on two spoken language understanding
datasets: Fluent Speech Commands (FSC) in English2[
Speech models often exhibit inconsistent performance across diferent speaker groups. To
address this issue, it is necessary first to identify and analyze these subgroups systematically. A
challenge in subgroup identification is ensuring that the subgroups are interpretable, meaning
they provide clear insights into why performance disparities occur. For instance, “young men
in noisy scenarios” is an interpretable subgroup, allowing both understanding and intervention.
To achieve this identification, we leverage the techniques of2[
Interpretable Metadata. Speech datasets typically include a variety of metadata attributes that can influence model performance. Demographic attributes such as gender, age, and native language are among the most common factors afecting recognition accuracy. Beyond demographics, speech characteristics such as speaking rate and silence duration also impact recognition performance3[0]. Faster speech or heavily accented pronunciation may introduce additional challenges, especially if the training data lacks suficient diversity. In addition to speaker characteristics, recording conditions also contribute to subgroup disparities. Factors such as background noise, microphone type, and reverberation levels can create variations in audio quality, afecting model predictions. A model trained primarily on clean audio data may struggle when encountering noisy environments, leading to disparate performance outcomes for speakers who record in less controlled conditions. Task-specific metadata, such as intent categories in spoken language understanding, also play a role in subgroup performance. Certain intents or command structures may be more frequently represented in training data, resulting in better recognition accuracy compared to less frequent or more complex intent formulations.
Automatic subgroup identification . To systematically extract underperforming subgroups,
we adopt DivExplorer3[
Specifically, let denote the dataset and the set of metadata attributes. Anitem is defined as an attribute-value pair. For exampleg,ender=female or speaking rate=high are items. A subgroup corresponds to the subset of data instances that satisfy one or more such items, represented as anitemset . Given a statistic (e.g., accuracy or error rate), the divergenΔce( ) of a subgroup identified by the itemset is defined as: Δ ( ) = ( ) − () . It indicates that the subgroup underperforms significantly relative to the dataset overall. A high negative divergence value indicates that the subgroup is significantly underperforming compared to the dataset as a whole. To ensure statistical reliability, subgroup discovery is constrained by a minimum support threshold, which filters out small subgroups where performance estimates may be unreliable. The subgroups are extracted by augmenting frequent pattern mining techniques, such as FP-Growth or Apriori, over the defined interpretable metadata, to also compute the divergence during the extraction process. By identifying subgroups with significant divergence, this method provides a structured way to analyze and mitigate performance inconsistencies. The identified subgroups inform post-processing targeted data acquisition (3§.1) and in-processing techniques via regularization 3(§.2), data augmentation (§3.3), and contrastive learning (3§.4).
Bias in speech models arises when performance varies significantly across diferent subgroups, often due to imbalanced representation in training data. To mitigate these disparities, various techniques have been proposed, broadly categorized into post-processing methods, which refine a trained model, and in-processing methods, which modify the training process itself. Postprocessing methods are useful when fairness issues emerge after deployment, as they adjust model predictions or incorporate new data without requiring full retraining. In-processing methods, on the other hand, introduce fairness-aware mechanisms directly into the learning process to ensure balanced performance from the outset.
This study covers 4 bias mitigation techniques: one is a post-processing method, 3 are inprocessing. For post-processing, we use targeted data acquisition, which enhances fairness by collecting additional subgroup-specific data. In-processing approaches include divergenceaware regularization, which modifies the loss function to prioritize underperforming subgroups; targeted data augmentation, which increases subgroup diversity through synthetic transformations; and contrastive learning (CLUES) to refine latent representations for improved fairness.
Targeted data acquisition is a post-processing approach that improves subgroup performance by supplementing the training set with additional real-world examples from underperforming subgroups. This method identifies performance disparities after model deployment and retrains the model with newly collected data.
The process begins by evaluating the trained model to identify subgroups with significantly lower accuracy compared to the overall dataset. These subgroups are interpretable, ensuring that their characteristics are clearly defined. By guaranteeing interpretability, we can perform targeted data acquisition to acquire new speech samples that better represent them. These additional samples are then integrated into the dataset, and the model undergoes additional ifne-tuning to improve its ability to generalize across all subgroups.
One of the key advantages of targeted data acquisition is its reliance on real-world speech variations rather than artificial data. This ensures that the model learns from natural speech patterns, accents, and recording conditions that were previously underrepresented. However, this method requires significant resources for data collection, annotation, and model retraining. Despite these challenges, targeted data acquisition is particularly valuable in deployed systems, where performance problems across groups only become apparent after real-world use.
Traditional model learning functions optimize for overall performance, often overlooking subgroup disparities. Divergence-aware regularization is an in-processing technique that directly modifies the training process to subgroup learning. This approach dynamically adjusts the loss function to focus on underperforming subgroups, ensuring they receive increased attention during training. In this method, the model training continuously monitors performance across diferent subgroups. If a subgroup exhibits significantly lower accuracy compared to the overall dataset, its samples are assigned higher loss weights during training. By amplifying the contribution of these samples, the model is encouraged to learn representations that better capture subgroup-specific variations.
Divergence-aware regularization is an efective solution for bias mitigation without requiring additional data. Since it operates directly on the training loss, it improves subgroup performance without altering the dataset size or introducing synthetic transformations.
Targeted data augmentation is another in-processing method that improves subgroup performance by artificially incrementing the training data for underperforming subgroups. Instead of collecting new samples, this approach applies synthetic transformations to the existing data to increase subgroup representation. Several augmentation techniques are commonly used in speech processing, including time stretching, which alters the speed of speech, pitch shifting, which changes the speaker’s tone, and noise injection, which simulates diferent recording environments. These transformations create diverse variations of the same speech sample, allowing the model to become more robust to variations in speaking style, accent, or background noise. In our context, once the underperforming subgroups are identified, we apply targeted data augmentation techniques to increase their presence in the training set.
One key advantage of this approach is its eficiency, as augmentation can be applied easily to existing samples. However, this method does not introduce truly new linguistic or demographic diversity—it only manipulates existing samples. Despite this limitation, it serves as a costefective way to improve model robustness for underperforming subgroups.
Contrastive learning has gained attention as an efective technique for refining the latent space representations of deep learning models. The CLUES (Contrastive Learning framework for Underperforming Subgroups) method applies contrastive loss to guide the model in learning more structured and subgroup-aware representations. Unlike regularization or data augmentation, which focuses on altering training behavior, contrastive learning reshapes the model’s internal feature space to better distinguish between subgroups.
CLUES operates at three levels of contrastive learning. At the task level, it ensures that samples belonging to the same class are grouped closely together while separating samples from diferent classes. At the subgroup level, it clusters samples from the same subgroup while pushing apart those from diferent subgroups. Finally, at the error level, it groups correctly classified samples separately from misclassified ones within each subgroup. By optimizing these three objectives, CLUES improves how the model encodes subgroup-specific information, leading to improved subgroup performance.
A key advantage of CLUES is that it improves model representations at the subgroup level without requiring additional data by restructuring the way data is represented. By explicitly shaping the latent space, CLUES reduces overlap between subgroup distributions, preventing the model from learning biased or entangled representations. However, contrastive learning introduces additional computational complexity. Despite this, experimental results show that CLUES provides the most efective technique for mitigating bias and improving performance.
Summary. Post-processing and in-processing methods ofer distinct strategies for addressing bias in speech models. Targeted data acquisition, as a post-processing method, enhances subgroup performance by incorporating real-world samples into model fine-tuning. In contrast, in-processing methods adjust the training process to achieve improvement at the subgroup level without external data collection. The selection of an appropriate bias mitigation method depends on the specific requirements of the application, including available data and computational constraints objectives. In the next section, we outline the experimental setup used to evaluate these methods and analyze their efect on subgroup and overall model performance.
This section presents the results of applying the four bias mitigation methods, analyzing their impact on overall model performance, subgroup fairness, and latent space representations.
Dataset and models. We conduct experiments on two spoken language understanding datasets:
Fluent Speech Commands (FSC) [
Metrics. We evaluate accuracy and macro F1 score to measure overall performance. For subgroup performance, we evaluate the maximum subgroup divergencΔe ( ), average divergence for the top-10 (Δ -10) underperforming subgroups, and the average divergence in absolute terms (|Δ - |). We also performed a latent space analysis using the Silhouette Score to assess how well the model distinguishes between subgroups when adopting CLUES.
Baselines. We compare our mitigation methods when using our automatic identification
approach against a set of alternative baselines that aim to identify challenging samples for model
improvement. Therandom baseline selects samples randomly, serving as a control to highlight
the efectiveness of subgroup-based selection. Theclustering baseline follows1[
w/ clustering ours - w/DivExplorer original - no held out w/ random w/ KNN w/ clustering w/ error-driven ours - w/DivExplorer original - all w/ random w/ KNN w/ clustering ours - w/DivExplorer w/ random w/ KNN w/ clustering ours - w/DivExplorer
w/ clustering ours - w/DivExplorer acquisition acquisition acquisition acquisition acquisition target data++ target data++ target data++ target data++ regularization regularization regularization regularization
CLUES
CLUES acquisition acquisition acquisition acquisition acquisition target data++ target data++ target data++ target data++ regularization regularization regularization regularization
CLUES CLUES
Accuracy
considered challenging if its nearest neighbors in the validation set are frequently misclassified,
with K optimized per dataset. Finally, therror-driven baseline, close to3[
Overall Performance. We report the results in Tabl1e. The four proposed bias mitigation methods lead to varying degrees of improvement in model accuracy and fairness. Divergenceaware regularization and contrastive learning (CLUES) achieve highest overall accuracy while allowing the highest reductions in subgroup disparities. Coupling any strategy with our identification methodology generally always achieves the best results (light yellow).
On the FSC dataset, when using the full training seto(riginal-all), the baseline wav2vec 2.0 model achieves an accuracy of 93.42% and an F1 macro score of 93.11%, but exhibits high divergence across subgroups. After applying mitigation strategies, CLUES improves overall accuracy to 98.79% and reduces subgroup divergence significantly. Divergence-aware regularization similarly enhances subgroup performance while maintaining a competitive accuracy of 98.5%, while targeted data augmentation yields more moderate improvements, particularly benefiting subgroups with lower representation. On ITALIC, the baseline XLS-R model achieves 73.22% F1 Macro (original - all). Divergence-aware regularization and contrastive learning improve overall performance to 74.85% for the former and to 76.10% and 76.72% when using CLUES coupled with clustering or our identification approach based oDnivExplorer.
Subgroup Performance. We assess how well the methods reduce subgroup disparities. Before mitigation, the baseline FSC model on overall data haΔs a of 53.18% (i.e., the least accurate subgroup performs significantly worse than the global accuracy of 93.42%). CLUES reduces the most this divergence, down to 17.58%. Divergence-aware regularization also substantially reducesΔ to 24.49%, confirming its efectiveness in addressing subgroup imbalances. For ITALIC, the baseline XLS-R model on overall data starts witΔh a of 47.54%. Contrastive learning and divergence-aware regularization reduce this gap to 40.15% and 30.10%.
Latent Space Analysis. We use the Silhouette Score to investigate the impact of bias mitigation on the latent space representations. A higher Silhouette Score indicates that the model better separates subgroups, reflecting improved internal representations of speech variations.
The baseline FSC model achieves a Silhouette Score of 0.737. CLUES improves this to 0.894, demonstrating that targeting subgroup representation learning significantly enhances the model’s ability to distinguish between subgroups. A similar pattern is observed in the ITALIC dataset, where contrastive learning improves Silhouette Scores from 0.319 to 0.539. This suggests that models trained with contrastive objectives learn more structured and subgroup-aware representations, contributing to improvements in subgroup performance. A complete analysis of model representations can be found in24[].
This paper outlined a framework for improving speech model performance by identifying and mitigating subgroup disparities, leveraging interpretable metadata to systematically detect underperforming, i.e.d, ivergent, subgroups. We explored four mitigation techniques: the post-processing targeted data acquisition and thein-processing divergence-aware regularization, targeted data augmentation, and contrastive learning (CLUES). Each method addressed performance inconsistencies diferently, either by enhancing model training, refining latent representations, or incorporating subgroup-specific data. The experimental results show that CLUES and the divergence-aware regularization are the most efective in reducing subgroup disparities. Moreover, CLUES enhances latent space representations. The findings highlight the value of adopting divergence-aware subgroup identification in speech model development.
This work is partially supported by the FAIR - Future Artificial Intelligence Research and received funding from the European Union NextGenerationEU (PIANO NAZIONALE DI RIPRESA E RESILIENZA (PNRR) – MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1.3 – D.D. 1555 11/10/2022, PE00000013) and the spoke “FutureHPC & BigData” of the ICSC - Centro Nazionale di Ricerca in High-Performance Computing, Big Data and Quantum Computing funded by the European Union - NextGenerationEU. This manuscript reflects only the authors’ views and opinions, neither the European Union nor the European Commission can be considered responsible for them.
During the preparation of this work, the author(s) used, Grammarly, ChatGPT to check grammar and spelling, paraphrase and reword. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. volume 33, 2020, pp. 12449–12460. URL:https://proceedings.neurips.cc/paper/2020/file/ 92d1e1eb1cd6f9fba3227870bb6d7f07-Paper.pdf. [35] A. Babu, et al., XLS-R: Self-supervised Cross-lingual Speech Representation Learning at
Scale, in: Proc. Interspeech 2022, 2022. do1i:0.21437/Interspeech.2022-143. [36] R. Magar, A. B. Farimani, Learning from mistakes: Sampling strategies to eficiently train machine learning models for material property prediction, Computational Materials Science 224 (2023).