Machine learning is reaching nearly every programming language and most kinds of devices. While the most popular language for developing machine learning application is Python, it has its own limits, for example, the partial compatibility with Android devices. When a mobile application needs to train a model, it is easier to achieve this with the device's native language like Java or Kotlin. There are many machine learning libraries for Java, but most of them lack Android support. This paper compares the resources needed to train random forest, support-vector machine and K-means models of the Weka, Tribuo and SMILE libraries. We developed an application to compare these libraries' implementations on datasets with various sizes. The results show that Weka is the suggested library for bigger datasets and complex models, as it is the least resource hungry.
In September 2020, Oracle announced Tribuo, their open source Java machine learning library under Apache 2.0 license. It features many commonly used algorithms like random forest, SVM, lasso, K-means, so it can solve prediction, classification, regression, clustering and anomaly detection problems. SMILE, the Statistical Machine Intelligence and Learning Engine is another machine learning library for Java. Its main advantage is performance compared to other libraries and algorithm support. Weka is a general purpose open source machine learning software with Java API, which is easy to use and has its own graphical interface. The common thing in these libraries is that they can be used with Java or Kotlin and there are many algorithms that all of them support. Because of this, their performance can be compared in the same environment, which will be an Android device with a mobile processor in it. Although these are not native Android libraries, they can work on these systems and their performance can be compared. Kotlin is a programming language for JVM, which became the preferred language for Android programming. Codes from Java can be transformed into Kotlin code, so it is easy to use Java libraries in this environment. Benchmarking mobile devices’ model training performance is a repeating task, because we can measure how much these devices evolved in years. In this paper, we present the Android machine learning ecosystem, the libraries, the challenge of porting machine learning libraries, and the results.
With the evolution of mobile devices and applications, it was inevitable to use
machine learning techniques for more personal user experience. Most applications
use pre-trained models to recognize voice, to take better pictures or to swap faces.
There are many disadvantages of training models on mobile, for example, the
energy consumption [
Java is not a popular language for machine learning, but it is popular for
application development, and because of the need of intelligent applications, it supports
many machine learning features with libraries. Each of these supports diferent
algorithms and datasets and each of them has advantages for specific systems. Weka
[
When we want to train and use machine learning models on Android, we can encounter many challenges. The main reason not to train models especially with large datasets on mobiles is that these operations need a lot of energy, so running applications would result in battery drain. While training is possible, it is limited by these devices’ memory capacity, because most mid-range smartphones usually have 4-6 gigabytes of memory, and this is not enough for processing larger datasets, not to mention that Android has limitations for memory usage. Another challenge for mobile machine learning developers is the architectural diference between the processors of computers and mobiles. Using Java libraries on these devices can be risky because Android does not have full Java support, so it is possible that some functions don not work or work, but with diferent results.
Currently, applications that want to use machine learning can choose from using TensorFlow Lite, or a library with Android Neural Network API support, or web services. With TensorFlow Lite developers can mostly use pre-trained models created with TensorFlow or they can train specific models for image and text classification. A popular choice is to use machine learning web services where the application sends data to the service and gets back the result.
Weka contains many Android incompatible code, mostly its graphical interface
and logging, but there are many more functions that use specific code parts, which
cannot be compiled on mobiles. Our strategy here was to remove everything that is
incompatible and see how the application can work with the newly compiled Weka.
The porting was successful, however sometimes minor errors occured during the
tests. The tested library was based on rjmarsan’s Weka-for-Android project [
SMILE also contained incompatible code, for example Java codes that were not supported by Android or functions with java.sql type. Because the number of these code parts were few, they were replaced by Android compatible ones. The result was good enough to run on mobiles, but it did not support all features, and some of these errors occured during runtime.
Oracle’s Tribuo library consists of many Maven artifacts, and some of them were not needed for this application. There are modules which cannot be compiled on Android, but the most important ones, namely tribuo-data, tribuo-classification-trees, tribuo-clustering-kmeans and tribuo-classification-sgd ran without modifications. Overally we can say that not all of Tribuo’s functions work on Android, especially the ones using third-party libraries, but we can do simple machine learning tasks with it on mobiles.
The aim of the Android application is to measure properties of machine learning libraries like memory usage or runtime. The libraries involved are the newly open sourced Oracle Tribuo, SMILE and Weka. The application runs the same test with each library, which means it trains models like SVM, Random Forest and K-means on multiple datasets with diferent sizes. The test uses the same algorithms and parameters for all libraries. The results and runtime properties are logged by the application.
The graphical interface is quite simple. When we tap on a library name, the
software will train a selected type of model on a selected dataset. The “ALL”
button is for running the training operation of all libraries parallelly. We can choose
from the Iris dataset [
Weka, SMILE and Tribuo can train machine learning models on Android and all of them can be used for simpler applications. As the software trained the same models with the same parameters and datasets these models’ performance was the same on the test datasets. The only diference between these libraries was in the runtime, memory usage, average CPU usage and energy consumption.
In Table 1 and Figure 2, we can see how fast the libraries finished the training of the models, where the prefix is the dataset (i is the Iris dataset and p is the Patterns dataset), and the second part is the name of the algorithm (where r is the random forest, s is the support-vector machine and k is the K-means).
As we can see, most of the time, Weka library was the fastest, except for Kmeans where SMILE was faster. For smaller datasets, both Tribuo’s and SMILE’s results are good, but for larger ones, they were much slower than Weka. Where there are zeros in the table, the software did not complete the training of the model, because the Android system killed the application due to its high resource need.
In Table 2 and Figure 3, we can see how much memory the algorithms needed in megabytes. The notation is the same as in Table 1 and Figure 2.
It is clear, that Weka used the least amount of memory, but in some cases Tribuo was wery close to it, like in iris-svm, iris-kmeans, and patterns-kmeans. There are special cases where the training did not finish, like Tribuo’s pattern-60000svm, pattern-rsvm and pattern-kmeans or SMILE’s pattern-60000-svm, patternsvm and pattern-kmeans where the memory need was extremely high compared to other cases. The highest value was the SMILE’s K-means training for the Pattern dataset where the application used 1.2 GB memory. For smaller datasets we can say that Tribuo and SMILE needed somewhat more memory, but this is not a huge diference.
In Table 3 and Figure 4 we can see the battery consumption of the application, when it trains the selected models. The table’s and figure’s notations are the same as earlier. The values range from 0 to 1, where 0 is the no energy need and 1 is the highest energy need.
Battery consumption is an important part of these measurements, because this means that a machine learning application could be maintained or it drains the battery that much, that the application is unusable. In Figure 4 we can see that Weka used the least amount of battery, the next one was Tribuo and the hungriest library was SMILE. For smaller tasks, Tribuo’s and Weka’s energy need were nearly the same.
In Table 4 and Figure 5 we can see the average CPU usage while training the models. The table’s and figure’s notations are the same as earlier. Average CPU usage shows how these libraries use this resource. When this value is below 50, it means that other applications can run parallelly while the training is running.
This paper showed that using machine learning libraries meant for Java can be used in Android application development if we have a specific task. However if an application uses models for image or text classification the recommendation is to use Tensorflow or web services. Specific tasks can be classifying or recommendation based on the users’ data, where it is necessary to train a local model.
Porting Weka, Tribuo and SMILE to Android devices is a somewhat challenging task, because each of them has some Java version or platform specific codes, which will not work on mobiles. To compare these libraries’ performance, we must select algorithms that each of them supports and has Android compatible implementation. For the comparison, we chose the random forest, SVM and K-means models and datasets with diferent sizes. The results show that Weka is the suggested library for bigger datasets and complex models, as it is the least resource hungry. It supports a wide range of algorithms, so every kind of machine learning task can be done with it. For smaller datasets or fewer complex models, Tribuo and Smile can be an option because they get updates frequently, so they can react faster to market needs.
An improvement can be extending these tests with other datasets, algorithms, or multiple devices. The results could be compared to results from computers instead of Android devices. As new machine learning libraries are released, they could be ported to mobiles and the results could be extended.