The recent advances in machine learning (ML) and sound processing have considerably expanded the possibilities provided by such techniques. The next step are open-access tools in existing ML toolkits for general sound ML tasks. We present JSI Sound, one of the first tools of this kind, developed within the Orange data mining software. The input for the tool is a series of labelled recordings. The first step allows choosing proper filtering and segmentation of the input sound. In the next step, a series of frequency and time domain features are extracted using extensive feature extraction libraries. Once the features are extracted, the data is ready to be used with the standard ML widgets implemented in Orange. Up to date, we evaluated the tool on five different domains: emotion recognition from speech, chronic heart failure detection form heart sounds, and species classifications from biosounds including bee, bird, and frog species. The evaluation results show high performance on all five domains.
Artificial intelligence and machine-learning methods are slowly shifting from predefined symbols and features as input to real world signals, e.g., sounds and videos. A tool that enables fast prototyping of ML models using sound as input can significantly reduce the experimenting time both for experts and non-experts. Such tool can easily build baseline ML models for any sound-related ML task. The baseline models can be later used for comparison with more advanced sound ML approaches.
In zoology and biodiversity studies, an important task is to classify an individual animal from sound. In psychology and medicine, an important task is to recognize human emotions due to emotional disorders (e.g., depression and bipolar disorders). In medicine, an important task is to detect chronic-heart failure (CHF) which is a global pandemic currently affecting over 26 million of patients worldwide [Ambrosy et al., 2014]. Often, experts in the fields can deal with these tasks based solely on sound. However, in many situations experts may not be available. In addition, in studies dealing with biodiversity monitoring, researchers are often faced with vast amounts of data. This calls for introduction of automatic classification methods to help the experts with their work. Furthermore, non-expert may find solutions based on sound valuable when proper quality is accessed. In this paper, we focus on the sound. Mammals, birds, amphibians, some groups of insects (such as cicadas or crickets) produce (typically structured) sounds to communicate. Aide et al. [2013] worked on a real-time acoustic monitoring system for sound-producing animals from Puerto Rico and Costa Rica. In our previous work [Gradišek et al., 2017], we used a combination of ML approaches to classify bumblebees based on the unstructured flight buzzing sound produced by movement of their wings. Ganchev and Potamitis [2007] used a score-level fusion of classifiers with non-parametric (probabilistic neural network) and parametric (Gaussian mixture models) estimation of the probability density function for identification of a series of singing insects, namely crickets, cicadas, and katydids. They achieved 90 % accuracy for 307 species. Stowell and Plumbley [2014] used unsupervised feature learning to classify bird songs. Also, for bird songs, Cheng et al. [2012] tested several machine-learning methods for classification. Apart from speech and other forms of vocalization, human body produces sounds such as breathing sound and sound of heartbeats. A healthy heart produces different sounds compared to an unhealthy heart. Depressed human talks differently compared to a joyful one. Cao et al. [2015] used speech analysis and ML for emotion recognition. Stassen et al. [1991] analyzed speech characteristics in depressed people. Gjoreski et al. [2017] used sound analysis and a stack of ML classifiers to detect CHF.
Similar to the progress from ML systems to ML toolkits, in sound recognition tasks it would be reasonable to create a toolkit that would allow everyone with some basic computer knowledge and sound data to experiment with ML techniques - to see which filtering and segmentation methods, sound features, ML algorithms, etc., work best for a particular dataset. Here, we present our ML tool in Orange [Demšar et al., 2013; Demšar et al., 2004] that enables performing ML experiments on sound data.
The tool encompasses five filtering methods, two segmentation methods, and extraction of six types of sound features in frequency and time domain. After the feature extraction, the tool enables the use of the standard ML widgets implemented in Orange, including all of its data-manipulating widgets, feature-scoring/selection widgets, visualization widgets, ML algorithms, and evaluation widgets. We evaluate the tool on five different domains: emotion recognition from speech, CHF detection form heart sounds, and species classifications from biosounds, including bee (species of North American bumblebees), bird (species from the warbler family), and frog species (Slovenian frogs).
2
The overall methodology is presented in Fig. 1. It consists
of five steps: input sounds, preprocess sound, extract
features from the preprocessed sound, build ML models, and
evaluate models. The input consists of a series of labelled
sound recordings of unspecified length. It is desired that all
the recordings are acquired using the same sampling rate,
although this can be later adjusted in preprocessing. In the
preprocessing step, five filtering methods (Finite impulse
response - FIR, Butterworth, Chebyshev, Elliptic, and
Bessel filter) can be used for noise reduction. In the
segmentation step, the filtered signal is split into segments using
sliding-window technique. During the split, the method
keeps track about which segment belongs to which
recording. The segmentation step is important since the original
recordings are typically long, which can be computationally
demanding later in the feature extraction phase. In addition,
it is advised that the segments that are low in information,
i.e., the energy of the audio signal in the segment is lower
than the median energy in the recording, are discarded
Fig. 1 JSI Sound - Method Since the method keeps track about which segment belongs to which recording, it enables building segment-based or recording based models. 3
Orange is an open-source ML and data mining suite for data analysis through Python scripting and visual programming in Orange Canvas. For researchers in ML, Orange offers scripting to easily prototype new algorithms and experimental procedures. For explorative data analysis, it provides a visual programming framework with emphasis on interactions and creative combinations of visual components. The Orange suits allow developing widgets by third parties, thus we implemented our methodology as Orange widgets. After the installation, the widget appears in the list of standard widgets as “JSI Sound”. In JSI Sound, the widget Import Sounds allows to input sound recordings from a standard file browser to Orange. Upon successful loading of the recordings, the user can use the functions offered by the JSI Sound toolkit widgets, i.e., filtering, segmentation, and feature extraction. From the filtering options, FIR, Butterworth, Chebyshev, Elliptic, and Bessel filters are available, together with parameter settings for each filter. From the segmentation options, sliding-window technique with parameters (window size and step overlap) can be chosen. After the segmentation, one can also use the filtering option to filter each segment separately. However, it is recommended that this is done before the segmentation stage due to computational complexity. The last option is the feature extraction option, which calculates features from preprocessed signal (or individual segments). The user can choose from five different feature-extraction options (Emotions, MFCC, PLP, Chroma, and pyAudioAnalysis). After complete pre-processing (filtering, segmentation) and feature extraction, the user can exploit the standard ML widgets implemented in Orange, including all of its datamanipulating widgets, feature-scoring/selection, visualization, ML algorithms, and evaluation widgets.
Figure 2 shows how to use JSI Sound widgets in Orange. First, the audio recordings are split on two subsets (Train : Test, 70% : 30%) and are input via the sound input widget (named as Train and Test in Figure 2). Next, the recordings are filtered and segmented using the Filtering and Segmentation widgets. The parameters are chosen by checking the spectrogram of several recordings using the Spectrogram widget. Next, ML models (Logistic Regression, Random Forest, and Decision Tree) are trained on the training data and evaluated on the test data using the standard Orange widget, Predictions. The output of the Predictions widget is used to check the predictions of each model for the test data. These predictions are predictions from each segment of the input recordings, meaning if one recording has been split on ten segments in the segmentation phase, there will be ten predictions for the recording per model. The widget Min/Avg/Max allows for merging of the predictions per recording and training a meta-classifierto produce the final output for each recording. 4
Here, we present the classification results on five diverse datasets: three datasets for animal species recognition (Birds, Frogs, and Bumblebees), one datasets for emotion recognition, and one dataset for CHF. In all domains, the recordings are segmented using a 1 s window and MFCC feature extractor, where we obtained 39 features. The details for each dataset are presented in Table 1. The table shows number of different labels per dataset, the number of different recordings and the number of segments per dataset produced after the segmentation phase.
The experiments involve three stages: segment-based ML stage, recording-based feature-extraction stage, and recording-based ML stage. In the segment-based ML stage, six ML models are build using the algorithms: Logistic Regression, Naive Bayes, KNN, Decision Tree, Random Forest, and AdaBoost. The idea behind combining a variety of algorithms is that different algorithms can model different structures in the data. Each of the six ML models takes as input the segment-based feature vectors and outputs a probability for a segment for each class. In the recording-based feature-extraction stage, the output of the segment-based ML models is aggregated and provided as the input for the recording-based ML stage. The aggregation is performed using min, max, and average over the predictions of the segment-based ML models.
In the recording-based ML stage, a recording-based ML model is trained, which produces min, max, and average probability for the segments in one recording from test set, for each class from six different ML models. These ML models (meta-classifiers) are evaluated using 10-fold cross validation on the test data. The results are presented in Table 2. The high AUC on the CHF dataset is just a confirmation from our previous study Gjoreski et al. [2017] that this is a simple ML task. The high AUC on the Frogs dataset is a consequence of a small number of individuals per class (23). The experimental results on Frogs and Emotions datasets are a confirmation of the results already achieved in our previous studies [Gjoreski et. al., 2014; Gradišek et al.]. passes five filtering methods, two segmentation methods, and extraction of six types of sound features, including frequency and time domain. All of them are implemented as Orange widgets.
We tested the tool on five diverse datasets – three coming from animal sounds and two from human sounds. The experimental results presented here are of high accuracy, some of them reproduced the experimental results from previous studies, which proves the suitability of JSI Sound for sound ML tasks.
We thank dr. Tomi Trilar for recordings of birds and frogs, and prof. Candace Galen and dr. Nicole Miller-Struttmann for recordings of bumblebees.