Spoken language understanding (SLU) models are commonly evaluated based on overall performance or predefined subgroups, often overlooking the potential insights gained from more comprehensive subgroup analyses. Conducting a more thorough analysis at the subgroup level can reveal valuable insights into the variations in speech system performance across diferent subgroups. Yet, identifying interpretable subgroups in raw speech data poses inherent challenges.
CEUR ceur-ws.org
Intelligent systems with speech recognition, transcription, and comprehension capabilities
are increasingly common across various domains, including virtual assista1n,t2s],[customer
service [
Recent literature has highlighted issues of model bias and unequal treatment across data
subgroups [
In this work, we propose an automated method for identifying critical subgroups to address
these limitations. Unlike existing clustering-based speaker embedding techniqu1e5s, 1[
Our approach. We introduce a novel task-, model-, and dataset-agnostic methodology for automating the characterization and comparison of data subgroups induced by metadata attributes. We identify all “frequent subgroups,” i.e., those exceeding a certain support threshold (e.g., at least0.1% of the dataset), that exhibit maximal disparities in intra- and cross-model performance. We provide end-users with interpretable representations of such critical subgroups within a given speech task and model and further use this information to mitigate model inner biases.
The primary contributions of this work are: (i) a novel framework for analyzing SLU models by identifying subgroups exhibiting large performance gaps; (ii) insights into the efects of model size at the subgroup level; and (iii) a subgroup-guided targeted data acquisition approach to enhance overall and across subgroups model performance.
We conduct comprehensive experiments across three speech tasks (Automatic Speech
Recognition (ASR), Intent Classification (IC), Emotion Recognition (ER)), three datasets
(LibriSpeech [
Our approach examines model performance at the subgroup level, whesruebgaroup is defined as a subset of the data characterized by specific metadata values, and denoted as itemset. This metadata covers mixed factors, including speaker traits (e.g., gender, age), speech features (e.g., speaking rate, number of pauses), and task-specific attributes (e.g., intents, labels). For instance, the subgroup{gender=male, age ∈ [41-65]} signifies utterances from male speakers aged 41 to 65.
Our analysis of subgroup behavior leverages two key concepts: intra-model divergence and cross-model performance gap. The former indicates the disparity in model performance between a subgroup and the entire dataset, revealing subgroups associated with performance variations, be it below-average, above-average, or equivalent. We will also leverage this aspect to guide the data acquisition strategy. Conversely, the latter quantifies the performance diferences between two models on the same subgroup, facilitating comparative assessments at the subgroup level.
We analyze speech model behavior by slicing data into interpretable subgroups. We define interpretable metadata as attributes understandable by humans, e.g., speaker age or gender or utterance noise level. For instanc“eo,ld men in noisy scenarios” is an interpretable subgroup. Metadata Description. Identifying interpretable subgroups in raw speech data poses intrinsic challenges. To overcome this issue, we enrich speech data with interpretable metadata from various domains, providing a human-understandable description of utterances. They can be inherent to the dataset or derived from utterances/transcriptions. Examples of such metadata attributes include: (i)speaker demographics like gender or age, (iit)ask-specific features , like intent or emotion associated with an utterance, (iirie)cording conditions, such as environment type and noise level, and (ivs)peech features, such as speaking rate and duration of silences. Items and Itemsets. Let represent our dataset an d denote its metadata attribute set. An item represents an attribute equality= , where is an attribute in , and is its value. We only focus on discretized attributes, thus continuous-valued attributes are discretized before applying our techniques. Examples of items includgeender = male and age ∈ [41 − 65], if gender and age are attributes. Asubgroup corresponding to an item denotes the dataset portion satisfying it. We ensure that subgroups form a dataset partition for each attribute. For example, the age ranges must not overlap within thaege attribute, and collectively, they must cover all potential age ranges.
Items facilitate the selection of data subsets based on single attributes, whitielmesets allow
slicing across multiple attributes. An item sectomprises zero or more items, each including
a diferent attribute. For instance, an itemset like{gender = female, age ∈ [
We aim to identify subgroups exhibiting performance disparities compared to the overall dataset.
We rely onDivExplorer [
We employ the concept of subgroup divergence (i.e., intra-model performance gap) as
introduced in [
Assessing performance discrepancies at the subgroup level is also crucial for model comparison. We introduce the concept of cross-model performance gap, which measures the performance diference between two models on a specific subgroup. This gap could be used to compare diferent models, characterized by diferent size, architecture, or pre-training objective. Specifically, given two models 1 and 2, the performance gap from mode l 1 to model 2 for the itemset is defined as the change in performance on obtained by replacin g 1 with 2: gap ( , 1, 2) = ( , 2) − ( , 1) (2) The definitions of intra- and cross-model gaps apply to generic SLU models for any task, enabling assessment of subgroup performance for a given dataset annotated via metadata. This methodology thus remains task-, model-, and dataset-agnostic. To evaluate the statistical significance, we employ Welch’s t-test to test the hypothesis that the means of the statist ic are equal for (i) the subgrou pand the entire populatio n , and (ii) the two model s 1 and 2.
After identifying itemsets exhibiting significant divergence or gap, we seek to understand the contribution of each item to these metrics. We employ game theory concepts to provide local insights into subgroup behavior.
The local contribution quantifies the role of each item within an itemset in influencing its
divergence or gap, using Shapley values. In this framework, items within an itemset are akin to
team members, and the divergence or gap metric represents the team’s total score. Specifically,
for an item within itemset and a metric of interest( ) , i.e., divergence or gap, the Shapley
value (, ) measures how much contributes to( ) , with ∑∈ (, ) = ( ) . More details on
this local as well as the global contribution can be found17in,2[
After evaluating the performance of a given speech model, our objective is to improve it both overall and across diferent subpopulations. We identify the critical subgroups (i.e., itemsets) characterized by negative divergence, representing challenging scenarios for the model. We implement a pruning procedure to eliminate redundancy among such subgroups, follow2i2n]g. [ Specifically, when encountering two subgroups , and , where includes along with an additional metadata condition, we retain only the more general subgr o u, pi,f the absolute diference in their divergences is below a predefined threshold. This approach is based on the rationale that adequately captures the divergence exhibited b y, as the extra metadata in only marginally afects the divergence. Pruning the critical subgroups yields a more concise representation, forcing the data acquisition process to focus on the most pertinent attributes.
We prioritize data acquisition eforts on the top - critical subgroups with the highest negative divergence in accuracy and retrain the model with additional data belonging to these subgroups. The parameter allows us to control the data acquisition process and observe its impact on model performance overall and within subgroups. Further details can be foun3d0]in. [
We assess the efectiveness of our methodology by (i) analyzing its ability to identify sources of
errors, (ii) examining the influence of factors such as model size, architecture, and pre-training
(a) w2v2-b. Δacc = -31.22%
(b) w2v2-b. Δacc = -0.65%
objective on subgroup-level performance, and (iii) evaluating the efect of using subgroup-level
information to guide a data acquisition strategy in enhancing model performance and mitigating
biases. Please refer to1[
Metadata. We enrich the datasets with various metadata categories. We first incorporate demographic attributes of speakers where available, including gender, age, and country. We also consider unique metadata pertinent to each task if available, i.e., intent FfoSrC, and emotion and arousal labels foIrEMOCAP. We finally extract from the raw signal utterance/transcription attributes such as silence duration (total and trimmed), word count, speaking rate (words per second), signal-to-noise ratio, and spectral flatness. The trimmed duration excludes initial and ifnal pauses, while the total silence duration includes the entire utterance without any pauses. As the frequency and duration of intermediate pauses had little efect on model performance across all datasets, except foLribriSpeech, we chose to retain them for this dataset only.
Continuous attributes like speaking rate or utterance duration require discretization into
ifxed ranges. Using frequency-based discretization, we thus discretize this metadata into three
ranges labeled as “low,” “medium,” and “high.”
RQ1: Model understanding at the subgroup level. We focus on the performance of the
wav2vec 2.0 base model 2[
(c) IEMOCAP. Performance improvement for 11.21% of subgroups, decrease for 85.85% of them.
(d) LibriSpeech. Performance improvement for 99.25% of subgroups, decrease for 0.75% of them.
For instance, forFSC, the wav2vec 2.0 base model exhibits its poorest performance for the subgroup characterized by speakers aged 22-40, male gender, no specified location, high speaking rate, and high total silence (Tabl1e,first block), with a divergence ofΔ = −31.2%. Analyzing sensitive attributes like gender is crucial, as evidenced by the significant impact observed. Specifically, female speakers achieve higher accuracy within the identified subgroup than males when all other metadata values remain constant. This trend is further confirmed by the Shapley values illustrated in Figu1r(ea)-(b), where the male gender is associated with lower accuracy. In contrast, the female gender exhibits a positive impact.
Conversely, the analysis also reveals subgroups with above-average performance. For example, the model correctly predicts all utterances associated with the subgroup of speakers aged 22-40 with a low speaking rate, long duration, and “washroom” as the target location.
Similar assessments can be made for other datasets. FLoirbriSpeech, we study the Word Error Rate (WER); a positive WER divergence (i.e., higher than overall) signifies lower performance. RQ2: Model comparison at the subgroup level. We compare diferent model performances at the overall and subgroup levels, detecting which subpopulations benefit the most from model changes. We analyze here how increasing the size of such models afects their performance at both levels. For changes in architecture and pre-training objective, please ref2e9r]t.o [
Larger models tend to be more accurate overall, a3n2d] [claims that larger models are also fairer. However, performance for specific subgroups is complex and depends on the dataset/task. We specifically examine how scaling up the wav2vec 2.0 model influences performance across datasets, with Table2 summarizing the performance gap in terms of the highest performance improvement and decrease, and Figur2eillustrating the distribution of this gap across subgroups.
While a larger model size enhances both overall and subgroup WER inLitbhreiSpeech dataset, it diminishes performance at both levels fIoErMOCAP. We further reveal varying subgroup impacts onFSC, indicating that certain groups benefit more from a larger model size than others. Nonetheless, more than 30% of the explored subgroups decrease performance when scaling up the size. These findings emphasize the importance of analyzing subgroup-specific outcomes when evaluating the efectiveness of larger models.
RQ3: Subgroup-guided data acquisition. We use the identified critical subgroups to guide a targeted data acquisition to improve model performance and mitigate its biases. We discuss the results forFSC. Further outcomes onITALIC [33], an IC dataset in Italian, can be found in 3[0].
We partition our dataset into training, held-out, validation, and test sets, employing an 80-20 split for training and held-out data, respectively, while retaining the original validation and test sets. We first identify critical subgroups using the validation set, then acquire data samples from the held-out set, and retrain the model with these samples. Evaluation on the test set (T3a)ble reveals consistently superior performance across overall and subgroup-level metrics, compared to baseline methods such as indiscriminate random and clustering-guided acquisiti1o5n], [ where samples are selected from the acoustic embedding clusters with subpar performance.
Selecting only the top 2 critical subgroups leads to significant performance improvements at both overall and subgroup levels. Specifically, it achieves the best F1 score and accuracy performance, as well as the lowest maximum divergenΔce− ( ) and the lowest average divergence for the top-10Δ(−−10 ), 20 (Δ−−20 ), and 50 (Δ−−50 ) subgroups with the highest negative divergence. While performance slightly lowers when increasing the nu mobfer critical subgroups, it remains significantly better than the original model performance and the one obtained when adding all available data. The lowest average absolute divergence is found with = 5 critical subgroups, indicating reduced performance disparities across subgroups.
Overall, these results underscore the efectiveness of targeted data acquisition in mitigating performance disparities and improving model robustness across diverse subgroups.
This study presents a novel methodology for evaluating spoken language understanding (SLU) system performance by analyzing model bias at the subgroup level. We enrich raw speech data by extracting metadata that include speaker demographics, task- and signal-related features to allow the definition of human-interpretable subgroups. By automating the detection of performance disparities within subgroups, our approach enhances error analysis, facilitates model comparison, and mitigates biases, thus improving overall performance. This versatile methodology demonstrates efectiveness across various speech tasks, datasets, and model sizes, ofering insights into which subgroups benefit most from system enhancements and contributing to the development of more inclusive and efective speech technologies.
Endow. 14 (2021) 2835–2838. doi:10.14778/3476311.3476357. [29] A. Koudounas, E. Pastor, G. Attanasio, V. Mazzia, M. Giollo, T. Gueudre, E. Reale, L. Cagliero, S. Cumani, L. de Alfaro, E. Baralis, D. Amberti, Towards comprehensive subgroup performance analysis in speech models, IEEE/ACM Transactions on Audio, Speech, and Language Processing 32 (2024) 1468–1480. doi1:0.1109/TASLP.2024.3363447. [30] A. Koudounas, E. Pastor, G. Attanasio, L. de Alfaro, E. Baralis, Prioritizing data acquisition for end-to-end speech model improvement, in: ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 7000–7004. doi:10.1109/ICASSP48485.2024.10446326. [31] A. F. Siegel, Chapter 10 - hypothesis testing: Deciding between reality and coincidence, in: A. F. Siegel (Ed.), Practical Business Statistics (Sixth Edition), sixth edition ed., Springer Science & Business Media, 2012, pp. 249–287. doi1:0.1016/B978-0-12-385208-3.00010-9. [32] Y. Sheng, J. Yang, Y. Wu, K. Mao, Y. Shi, J. Hu, W. Jiang, L. Yang, The larger the fairer? small neural networks can achieve fairness for edge devices, arXiv preprint arXiv:2202.11317 (2022). [33] A. Koudounas, M. La Quatra, L. Vaiani, L. Colomba, G. Attanasio, E. Pastor, L. Cagliero, E. Baralis, ITALIC: An Italian Intent Classification Dataset, in: Proc. INTERSPEECH 2023, 2023, pp. 2153–2157. doi:10.21437/Interspeech.2023-1980.