Analysis of audio signals is widely used and very effective technique in several domains like healthcare, transportation, and agriculture. In a general process the output of the feature extraction method results in huge number of relevant features which may be difficult to process. The number of features heavily correlates with the complexity of the following machine learning method. Dimensionality reduction methods have been used successfully in recent times in machine learning to reduce complexity and memory usage and improve speed of following ML algorithms. This paper attempts to compare the state of the art dimensionality reduction techniques as a building block of the general process and analyze the usability of these methods in visualizing large audio datasets.
With recent advances in machine learning, audio, speech, and music processing has evolved greatly, with many applications like searching, clustering, classification, and tagging becoming more efficient and robust through machine learning techniques. Well performing dimensionality reduction methods have been successfully applied in many areas of computer science and interdisciplinary fields.
A very important phenomenon that justifies the use of such methods is the curse of dimensionality. The basic idea is that the increase of dimensionality increases the volume of space, causing the data to become sparse. Therefore, the amount of data required for a model to achieve high accuracy and efficiency increases greatly.
Considering audio signals, meaningful features need to be extracted before applying any dimensionality reduction method. Using methods for feature extraction like Short Time Fourier Transformation (STFT), Mel-frequency cepstral coefficient (MFCC), or high-level descriptors like zero-crossing-rate, representative data can be extracted from the audio data. After extraction these features can be projected into two-dimensional space for cluster analysis and examination of class separation.
The dataset used is the UrbanSound 8k dataset,
containing 8732 labeled sound excerpts ( 4 seconds) of urban
sounds from 10 classes, from which 5 have been used in
this work: car horn, dog bark, engine idling, gun shot, and
street music [
Related work is presented in Section 2, basic mathematical notation used is described in Section 3, while the different methods of the pipeline are briefly presented in Section 4. Section 5 contains data about the evaluation methodology, Section 6 presents the results and conclusions are formulated in Section 7.
The goal of this paper is to find a combination of feature extraction and dimensionality reduction methods which can be most efficiently applied to audio data visualization in 2D and preserve inter-class relations the most. 2
A large number of research addresses the problem of mapping a collection of sounds into a 2-dimensional map and examining the results both visually and analytically.
Roma et al. [
Fedden [
Hantrakul et al. [
Dupont et al. [
Charles Lo’s work [
Given a dataset of discrete audio signals, we denote the i-th recording as x(i) and the dataset as X = fx(1) ;x(2) ; :::;x(N) g. The corresponding groundtruth labels are stored in Y = fy(1); y(2); :::; y(N)g. A single audio element of the dataset is comprised of a number of samples: x(i) = (x1(i); x2(i); :::xM(i)). 4
The building blocks of the process used in this work are presented in the following sections and are also shown on figure 1. As seen on the figure, a set of discrete audio signals are transformed trough consecutive methods to result in a 2D map of datapoints. Applying filtering of some kind before feature extraction could be justifiable to remove frequency intervals that usually do not have discriminative power. The simplest such solution is applying a band-pass filter. A band-pass filter passes frequencies between a lower limit fL and a higher limit fH , while rejecting frequencies outside this interval. Band-pass filters are usually implemented using Butterworth filters, as these have maximally flat frequency response in the passband.
The following low-cutoff and high-cutoff frequency pairs have been used as band-pass filter specifications: In order to extract meaningful data, that can be interpreted by machine learning models, the step of feature extraction is required. This step also acts as a dimensionality reduction process, because it transforms a sound segment represented by hundreds of thousands of samples to a data structure containing only a few thousand values or less. This is a crucial step as it enables to apply dimensionality reduction methods later that will ultimately project the sample into the lowest dimensionality of just 2 components. STFT The Short-time Fourier transform (STFT) captures both temporal and spectral information from a signal. As a result, the STFT transforms the signal from time-amplitude representation to time-frequency representation. Formally, the STFT is calculated by applying the discrete Fourier transform with a sliding window to overlapping segments of the signal. It can be formulated in the following way:
L 1 ST FT fx(i) g := S(i)[k; n] = å x(i)[n + m]w[m]e jk 2Lp m m=0 (1) where w[m] is the window function, L is the window length, n denotes the frame number and k denotes the frequency in question.
The STFT returns a complex-valued matrix, where jS[n; k]j is the magnitude of the frequency bin k at timeframe n.
Mel-scaled STFT As human perception of pitch (frequency) is logarithmic, the Mel-scale is often applied to the frequency axis of the STFT output. Specifically, the Mel-scale is linearly spaced below 1000 Hz and turns logarithmic afterward.
The Mel-scaled frequency value can be computed from a frequency( f ) in Hertz using the following equation: Mel( f ) = 2595 log10(1 + (2) 1. Take the Short-time Fourier transform of the signal. 2. Apply the Mel filterbank to the power spectra of each frame, summing the energies in each frequency band. The Mel filterbank is a set of overlapping triangular filters. 3. Take the logarithm of the calculated energies. 4. Take the Discrete Cosine Transform (DCT) of the log filterbank energies, resulting in a so-called cepstrum. Intuitively, the cepstrum captures information of the rate of change in frequency bands. The DCT is related to the DFT (Discrete Fourier Transform). While DFT uses both sine and cosine functions to express a function as a sum of these sinusoids, the DCT uses only cosine functions. Formally (using the DCT-II):
N 1 C[k] = å x[n] cos n=0 p N
1 (n + )k ; 2 1 k
N (3) 5. The MFCC coefficients are the resulting DCT coefficients.
High-level features High level descriptors that capture temporal or spectral information about the signal are often used in similar experiments. The following six features have been used for frame number n:
vRMS(n) = s åLm=10 x[n + m]2
L where L is the window length.
Zero crossing rate: vZC(n) = 1 L 1
å jsgn(x[n + m]) sgn(x[n + m 2L m=0 (5) where L is the window length. The zero crossing rate is the number of changes of sign in consecutive audio samples.
vSC(n) = åkK=01 k jS[k; n]j2
åkK=01 jS[k; n]j2 The spectral centroid indicates where the center of mass of the spectrum is located.
vSR(n) = i
åik=0 jS[k;n]j=k åkK=01 jS[k;n]j Spectral rolloff is the frequency below which a specified percentage of the total spectral energy lies, with k denoting the percentage score. (4) (6) (7)
vSS(n) = s åkK=01(k åkK=01 jS[k; n]j2 vSC(n))2 jS[k; n]j2 The spectral bandwidth is a measure of the average spread of the spectrum in relation to its centroid.
vSF (n) = q K ÕkK=01 jS[k; n]j
K1 åkK=01 jS[k; n]j The spectral flatness is the ratio of geometric mean and arithmetic mean of the magnitude spectrum. (8) (9) 4.3
After the feature extraction step, the feature set takes the
(N; k; n) form, where N is the size of the dataset, k denotes
the number of frequency bins (or features) and n is the
number of time frames. To be able to perform
dimensionality reduction on the whole dataset, the feature set needs
to be transformed to a (N; d) form, where d is the size of
a feature vector after the reshaping transformation. Three
such methods have been found in associated papers:
Flattening Stacking consecutive rows horizontally to
form one, long 1-dimensional vector. Using this method, a
feature set of shape (k; n) will take the (1; k n) form after
the reshaping operation. Used by [
2S11 6S21 4 ..
.
S12 S22
S13 S23 . . .
S22 : : :3 : : :7 =)
5 S23 : : : S11
S12
S13 : : : S1N
S21
calculating some statistical features across all frames, the
spectrogram can be reduced to one feature vector, where
statistical values (mS - mean of random variable, sS -
standard deviation of random variable) calculated for each
frequency band are stacked vertically in a continuous fashion.
Using this method, a feature set of shape (k; n) will take
the (k A; 1) form after the reshaping operation, where A
is the number of statistics calculated. Used by [
2S11 6S21 4 ..
.
S12 S22
max Similar to the solution above, but with additional
minimum and maximum values calculated for each
frequency bin. Used by [
ponent Analyis is a basis transformation that results in a set of orthogonal basis vectors along which the variance of the data is maximal.
Calculating PCA involves the following steps: 1. Denote the reshaped dataset with X 2 RN d . 2. Standardization: subtract the mean, and divide by the standard deviation for each feature column. Each column will a have a mean of 0 and a standard deviation of 1.
The variance-covariance matrix consists of the variances of the variables along the main diagonal and the covariances between each pair of variables in the other matrix positions. The covariance matrix is symmetric, thus it’s diagonalizable.
The covariance matrix is calculated from X using the following formula:
C =
1 n 1
X T X ;
C 2 Rd d 4. Compute the eigenvectors and eigenvalues of the covariance matrix. Formally, the eigendecomposition of a matrix is described as:
C = V DV 1 where V is a square matrix whose ith columns represents the ith eigenvector of C. D is a diagonal matrix whose diagonal elements represent the eigenvalues of C.
These eigenvectors are called the principal components of the dataset, ie. the axis on which the data is more dispersed. 5. Sort the eigenvalues and the corresponding eigenvectors in decreasing order. 6. Leave the first d0 number of eigenvalues and their corresponding eigenvectors. V 0 = Vi j; 1 i d; 1 j d0
Xlow_dim = XV 0 ;
t-Stochastic Neighbor Embedding (TSNE) t-SNE is a
nonlinear dimensionality reduction method developed by
Laurens van der Maaten and Geoffrey Hinton in 2008 [
Using this probabilistic approach, the first main idea is that to convert Euclidean distances into conditional probabilities that reflect these spatial similarities. Let P be the joint probability distribution of the data in high dimensional space, represented by a normal distribution: p jji = åk6=i ex p( 1 kx(i) x(k) k2 ) 2si2 ex p( kx(i) x( j) 2
k ) (14) 2si2 Intuitively, this expresses the conditional probability of xi picking x j as its neighbor if neighbors were chosen in proportion to their density around xi.
Student t-distribution (with one degree of freedom) Q is used to model the distribution of the low dimensional data: qij j =
(1 + ky(i) åk6=i(1 + ky(i) y( j) 2 k ) 1 y(k)k2) 1
Due to difficult optimization, t-SNE introduces a modified version of the P distribution, a symmetric probability density function, defined as: pi j = pij j + p jji 2n (15) (16) (17) (18) t-SNE uses gradient descent to minimise the error between the two distributions and adds a decaying momentum term as an optimization. The authors introduce the Kullback-Leibler (KL) divergence as the cost (loss) function:
C = KL(PjjQ) = å pi jlog pi j i6= j qi j The gradient of the cost function is the following: ¶C ¶ yi = 4 å(pi j j qi j)(yi y j) (11) (12)
In essence, the t-SNE algorithm is based on the following steps: 1. Compute pairwise similarities p jji. 2. Let pi j = pij j2+np jji . 3. Randomly sample the low dimensional points from a Gaussian distribution with small variance (ie.: Y N(0; 10 4I)). 4. Repeat the following steps for T number of iterations: for t=1 to T do: (a) Compute low-dimensional similarities qi j. (b) Compute gradient ¶¶Cy . (c) Update y(t) = y(t 1) + h ¶¶Cy + a(t)(y(t 1) y(t 2)), where h is the learning rate, a the decaying weight coefficient of the momentum term.
An important hyperparameter of the t-SNE method is the perplexity. According to the authors, perplexity is the measure of the assumed number of neighbors, i.e. it balances the attention between the local and the global structure of the dataset. While typical values are situated between 1 and 50, the optimal perplexity value is a function of the dataset size, thus manual tuning is necessary.
Isomap Isomap is a nonlinear dimensionality
reduction method, belonging to the group of manifold
learning methods [
The steps of the Isomap algorithm are: 1. Construct a weighted neighborhood graph using kNN or e-radius and assign the Euclidean distances between the nodes to the edge weights. 2. Compute the shortest path between each node using the Floyd-Warshall or Dijkstra’s algorithm. Construct a distance matrix D from these pairwise geodesic distances. 3. Apply the multidimensional scaling (MDS) algorithm on the distance matrix calculated above. The following steps are part of the classical MDS algorithm. Firstly, double centering is applied to the squared distance matrix, using the centering matrix: H = I 1n eeT , where e is a N 1 column vector containing 1-s. where P contains the eigenvectors and L is a diagonal matrix containing the eigenvalues.
1
Xlow_dim = Pd0 Ld20 1 where Ld20 contains the square roots of the d0 largest eigenvalues, and Pd0 is the matrix containing the eigenvectors corresponding to the first d0 largest eigenvalues.
(SOMs) were introduced by Finnish professor Teuvo
Kohonen in 1982 [
The learning process can be summarized in the following way: 1. Randomly initialize all weights. 2. Pick one random sample from the dataset. 3. For each node of the map, calculate the Euclidean distance between the input vector and the node’s associated weight vector:
d0 d j(x(i) ) = å (xk(i) k=1 w jk)2 (22) where d0 is the dimensionality of the input data, x(i) is the input vector and w jk represents the weight associated to the connection between the j-th map node and the k-th element of the input vector. The node that produces the smallest distance is called the BMU (Best Matching Unit). 4. The weights of the BMU, as well as the weights of its neighboring nodes, are updated. The radius of neighbors to be updated is initially large, but it is reduced after each iteration.
wtj+k1 = wtjk + h(t)Tj;BMU (t)(xk(i) wtjk) (23) where h(t) is the decaying learning rate, BMU is the index of the Best Matching Unit and Tj;BMU (t) is the neighborhood function of the form:
Tj;BMU (t) = e dist( j;BMU)=2s(t)2 (24) If the distance between the two map nodes with indices j and BMU is larger, then the output of the neighborhood function will be smaller, and the j-th node will not get a significant update, as opposed to a closer node. The s (t) parameter shrinks with time, considering smaller and smaller neighborhoods. 5
For the evaluation of the class separability on the resulting 2D map, external cluster analysis metrics are used to analyze the correspondence between the ground-truth labels and the results of the clustering algorithm. 4. Calculate the eigendecomposition of B.
B =
HD2H
K-means is used for clustering, as it performed more
positively during preliminary testing, as opposed to other
methods like hierarchical clustering, DBSCAN or
OPTICS. The number of centroids was fixed to the number
of classes, namely 5. The algorithm ran 10 times with
different centroid seeds.
We concentrated on using external clustering evaluation
methods that compare the labels assigned by a clustering
algorithm with ground truth labels; similarly to [
RI =
T P + T N
T P + FP + T N + FN FM = r
T P T P T P + FP T P + FN (25) (26) (27) V-measure It is calculated as the weighted (here b = 1) harmonic mean between homogeneity (h) and completeness (c), which are also external cluster analysis metrics. Homogeneity is satisfied when every cluster contains only data points which are members of a single class. Completeness is satisfied when all points that are members of the same class, are also the members of the same cluster.
Vb = (1 + b ) h c
(b h) + c
The evaluation of the results with these supervised metrics measures the extent to which audio recordings of the same class get projected to points that are close to each other and form spatially coherent, independent clusters. 5.3
Five categories of sound samples were used from the UrbanSound8k dataset: car horn (150 samples), dog bark (150 samples), street music (142 samples), gunshot (150 samples), idling engine (150 samples).
The recordings of each category contain recordings of varying noise levels, environmental sounds and overlapping sounds. A specific hyperparameter-configuration notation is introduced for presenting the results, which is formatted in the following way: F-FE-NrBin-Reshape-DimRed-Hyperp. The meaning of the symbols and the possible values of the hyperparameters are described below:
F: value between 0 and 5. 0 means no filtering, 1-5 encode the cutoff frequency pairs identified in section 4.1.
FE: identifier of feature extraction method: STFT, Mel scaled STFT (mel), MFCCs or high-level features (feat).
NrBin: number of frequency bins: 13, 20, 40 bins. Only valid for the STFT, Mel-STFT, MFCC methods. Reshape: identifier of reshaping method as described in section 4.3:
2. Calculating mean and standard deviation across frames. 3. Calculating mean, standard deviation, minimum and maximum values across frames.
DimRed: Dimensionality reduction method, with 4 possibilities and possible further parameters for some methods (Hyperp): 1. PCA - no further hyperparameter was identified, that would significantly influence the outcome. 2. t-SNE - identified hyperparameter: Perplexity.
Values to be tested: 20, 40, 60. 3. Isomap - identified hyperparameter: Number of neighbors. Values to be tested: 20, 40, 60. 4. SOM - identified hyperparameter: Sigma. Values to be tested: 1, 2, 4.
All possible value combinations for the different hyperparameters described above generates 1500 different runs. Gridsearch was executed over the whole set of combinations. Each configuration was run 3 times to ease the effects of random components. 6
researchers didn’t find significant differences between the
performance of audio feature extraction methods e.g.:
[
Configuration 5-mfcc-40-2-tsne-20 4-mfcc-40-2-tsne-20 4-mfcc-40-3-tsne-20 4-mfcc-40-3-tsne-40 3-mfcc-40-3-tsne-20 5-mfcc-40-3-tsne-20 2-mfcc-40-2-tsne-20 0-mfcc-40-3-tsne-40 1-mfcc-40-3-tsne-20 0-mfcc-40-2-tsne-40
The top 3 results are illustrated on Figures 2, 3 and 4.
According to the results, the V-measure index is the closest to the mean of all albeit it goes without saying that the mean is heavily dependent on the selected indexes. The mean for the top results is very close to the value of the VMeasure index. Unfortunately the middle and the worst results are not correlating to the V-Measure index.
Configuration 4-stft-13-1-som-1 5-feat-6-1-isomap-40 4-mel-20-1-som-1 4-feat-6-1-som-1 0-mel-13-1-som-1 2-feat-6-1-som-1 0-feat-6-1-som-1 1-feat-6-1-som-4 2-feat-6-1-som-2 4-feat-6-1-som-2 7
As it has been pointed out, the MFCC-TSNE combination
achieved the best results. MFCC’s success is supported by
its widespread use in environmental sound classification
[
Considering dimensionality reduction methods, t-SNE
clearly emerged as a winner, both in a visual and
metrical sense. TSNE is famous for generally producing well
separated, good visualizations, as pointed out by its
authors [