real-time. The we4bee project has deployed 100 smart hives all over Germany. These hives are equipped with microphones, among other sensors. Beekeepers and bee researchers have observed the importance of sounds when monitoring bee hives, but audio can only be recorded in accordance with privacy laws. To prevent saving recordings of human voices, our aim is to deploy a pre-trained deep learning model on the Raspberry Pi 3B computer controlling the smart hive. This model has to classify recorded data in In this technical report, we document the process of setting up the software on the Raspberry Pi, the adaptations required for existing code to run in the new environment, and the necessity of modifying the trained models for deployment on the mini-computer. We find that in both standard operation conditions and under various artificial levels of high CPU and I/O load, the model's inference runs in real-time.
CEUR ceur-ws.org
CEUR Workshop Proceedings
k-NN Pred.
In our previous work [
The Siamese network has been trained on a total of 200 labeled samples of 60 s, of which we created random data pairs. The network was then trained to minimize the Euclidean distance between audio pairs from the same class (e.g., speech-speech) and to maximize the distance between pairs of diferent classes. From the trained network, the embeddings of the training data were then extracted from an intermediate layer and used to train a k-NN classifier to predict a sample’s class.
To be able to record audio in more than one beehive, we want to identify and discard audio recordings with human speech directly on the beehive’s Raspberry Pi. To this end, we selected an exemplary smart beehive and completed the following tasks: installing the TensorFlow Lite library on the Raspberry Pi; migrating our Python environment and scripts; migrating the model; and enabling real-time classification of incoming audio data. The proposed approach to real-time speech detection is visualized in fig. 1. The existing system, visualized on the left, records sensor and image data and uploads them directly. For the detection of speech, we use the pipeline visualized in the middle, where only no-speech data is stored for upload. The remainder of this report will delve into the necessary adaptions in more detail.
Since our earlier research was conducted using the TensorFlow library [
(b) Sample recording of bee humming.
(c) Total runtime distribution.
(d) Total runtime at different I/O loads. learning research, the 1 MB small TensorFlow Lite (TFlite) package focuses on inference and deployment and is suficient for our purpose.
We verified the successful installation by running an oficial audio classification tutorial [ 16]. In addition to the steps in the tutorial, we had to add our user to the audio group to gain access to the microphone. Following these configuration changes, we successfully executed the oficial audio classification tutorial and thereby confirmed the installation and functionality of TFlite.
After installing TFlite, we migrated the Python environment and adapted the scripts. The
methodology outlined in [
The last step was migrating the best-performing model from our earlier research. Bulbul
relies on the kapre library [18] for transforming raw audio input data to a spectrogram during
the forward pass. Two spectrograms of speech and bee humming are given in fig. 2a and fig. 2b,
respectively, which show diferent intensities in the lower frequencies. In fig. 2a, vertical lines
also indicate human speech. The audio conversion to spectrogram takes place in custom layers
requiring non-standard TFLite operations. To avoid installing full TensorFlow, we replaced the
spectrogram and magnitude-scaling layer of the original model with TFLite-compatible ones
and transferred trained weights. After conversion to the .tflite format, we confirmed that
the model performance did not sufer by re-running experiments from [
The smart beehive runs various sensor recording services, whose average CPU load over 15 min is 25.48 % (std 15.05). The memory usage generally is small, ranging from 5.78 % to 12.59 %, indicating that around 100 MB of the 1 GB RAM are occupied. These statistics show that the Raspberry Pi has enough resources available for real-time audio classification and uploading data without speech. To evaluate the boundary at which this is no longer feasible, we loaded the Bulbul and k-NN model, and separately timed the audio pre-processing, embedding extraction, and prediction over 117 files uploaded to our research beehive. For that, we disabled the sensor recording services and used the Linux commands stress to induce and nice to prioritize artificial CPU load. Despite these severe restrictions, the system scheduler ensures that our classification runs in real time. Further, we also evaluated the prediction performance under varying I/O loads using the stress --io n_threads command with {1..10} threads. The results in fig. 2d show that for more than 5 threads inducing I/O load, more outliers arise, and the average total runtime and its standard deviations increase. However, while these increases indicate that our script has more waiting time, the system is still capable of real-time inference.
To verify the performance in a controlled, realistic setting, we re-activated the sensor recording system and ran the inference five times in sequence, yielding 585 measurements in total. A histogram of the overall runtimes is given in fig. 2c, which shows that the majority is predicted in less than 60 s, with the k-NN having negligible influence. Of the 585 tested files, only 19 take longer than 60 s to classify. Of these delays, according to logs, 10 are caused by the camera recording, 4 by measuring the finedust concentration, and for 5 the source is unclear. Apart from this, the parallel recording of the sensor modalities had no negative impact on model runtimes. In summary, the results show that we only have around 20 min of audio data that remain unclassified over a period of roughly 10 h. On average, this translates to two audio snippets per hour that cannot be classified in real-time. Further data loss through connection failures is prevented by bufering up to 60 h of classified, non-speech data.
For evaluating the model performance on a diferent hive, we asked one male and one female volunteer to perform diferent activities (talking, playing music and singing, laughing and rough-housing, and staying silent for a fixed time period) at 1 m, 5 m 10 m distance to the hive. Our observations show that all activities are well-detected at all distances to the hive, including quiet speech.
Lastly, we let the automated audio classification run over one week, logging the predictions. The speech detected by our system can generally be mapped to real events, as consultation with the hive’s owners revealed. For example, one morning, between 7 and 7:15 am, speech is detected when they take their dog for a walk. In another instance, they prepare for travel.
In this technical report, we describe the process of deploying a speech detection model onto a Raspberry Pi 3B in a smart beehive. We show that, after making adaptions to the Python code and model architectures, we can deploy and run the model on the mobile computer. On this hardware, our setup can predict the class of a one-minute audio recording in less than 60 s, i.e., in real-time. Even when high CPU and I/O load simulate extreme scenarios, the setup is still capable of real-time inference. Lastly, using controlled activities with two volunteers as a test case, we showed that our model can predict human speech at various distances from the recording device. The next step is rolling out the models to all hives of the we4bee project. X. Zheng, TensorFlow: Large-scale machine learning on heterogeneous systems, 2015.
URL: https://www.tensorflow.org/, software available from tensorflow.org. [16] Tensorflow lite python audio classification example with raspberry pi, https://github.com/ tensorflow/examples/tree/master/lite/examples/audio_classification/raspberry_pi, 2022.
Accessed: 2023-07-13. [17] B. Bechtold, soundfile audio library, https://pypi.org/project/soundfile/, 2023. Accessed: 2023-07-13. [18] K. Choi, D. Joo, J. Kim, Kapre: On-gpu audio preprocessing layers for a quick implementation of deep neural network models with keras, in: Machine Learning for Music Discovery Workshop at 34th International Conference on Machine Learning, ICML, 2017.