In this paper we present our results for the MediaEval 2018 Medico task, achieved with traditional machine learning methods, such as logistic regression, support vector machines, and random forests. Before classification, we combine traditional global image features and CNN-based features (early fusion), and apply soft voting for combining the output of multiple classifiers (late fusion). Linear support vector machines turn out to provide both good classification performance and low run-time complexity for this task.
The Medico task at MediaEval 2018 [
To address the research objectives described in Section 1, we chose
to train traditional machine learning algorithms only [
Early fusion of feature vectors created by diferent feature
extraction methods increases the dimensionality of the feature space
and improves the likelihood that binary classification problems are
linearly separable. On the other hand, increasing feature
dimensionality will reduce the run-time eficiency of machine learning
algorithms. To cope with this trade-of, we selected a sensible
combination taken from the following feature sets of the given endoscopic
image dataset: LIRE – traditional global image features extracted
using the LIRE library [
Preliminary cross-validation experiments on the training dataset
indicated that LIRE and GoogLeNet features performed significantly
better than SurgicalAction features, consistently across diferent
classifiers. Moreover, early fusion (concatenation) of LIRE and
GoogLeNet features (2209-dimensional) further improved
crossvalidation scores, but no additional improvement was observed
when combining all three feature sets. Therefore, we decided to use
only the 2209-dimensional feature set for all further experiments
producing detection runs for the Medico task. Each feature
dimension was centered (by subtracting the mean) and scaled to the range
[
Due to the low number of training samples (see Fig. 1) compared to the dimensionality of the feature space, traditional linear classifiers have the potential to provide a sensible trade-of between efectiveness (classification performance) and eficiency (run-time complexity). We therefore included two linear classifiers, logistic regression (LR) and linear support vector machine (LSVM), into our experiments, but also chose two non-linear classifiers for comparison, namely random forests (RF) and kernel support-vector machine (KSVM) with a radial basis function (RBF) kernel. Multinomial logistic regression and random forests implicitly support multi-class classification, whereas support-vector machines (SVMs) were used in a one-versus-rest (OVR) ensemble to support multiple classes.
Table 1 lists the classifiers used for our submitted runs, including information on decision boundaries, tuned hyper-parameters, multiclass strategy, support for class probabilities, and obtained crossvalidation (CV) scores on the development set. Hyper-parameters include the regularization parameter C (lower values mean stronger regularization), the width γ of the RBF kernel, and number n and maximal depth k of decision trees for random forests.
Hyper-parameters of chosen classifiers were optimized
independently using grid search in the parameter space and 4-fold
crossvalidation on the development set provided by task organizers. As
objective function for optimization the mean MCC [
To combine the output of several classifiers (late fusion),
various well-known ensemble methods exist in the literature [
Note that only two of the chosen classifiers naturally provide
class probabilities for prediction (LR and RF), whereas SVMs do
not. However, with additional run-time cost, confidence scores
produced by SVMs can be transformed into class probabilities using
Platt scaling [
We submitted five runs with predictions for the test set (8740 images) to Medico task organizers for evaluation: one for each classifier listed in Table 1, and a late fusion run combining the output of RF, KSVM, and LR classifiers. Table 2 lists some of the evaluation metrics obtained by oficial evaluation of our runs as well as mean prediction times per image (in milliseconds) on the test set measured on commodity PC hardware (Intel Core2 E8400 CPU @ 3 GHz, 8 GB RAM, no GPU usage). Note that prediction times do not include feature extraction and model loading times, but refer to the time span needed to perform feature scaling and prediction of classes and class probabilities (if applicable), including the total time needed to apply an ensemble of classifiers (RF, SVM in OVR mode, RF-KSVM-LR fusion). Prediction times have been measured three times and the average value is reported in the table. Since feature extraction and prediction were performed with diferent software and in batch mode, investigation of real-time processing capabilities of the proposed approach for online video processing would need further experiments.
All submitted runs display comparable classification performances on the test set, except for RF, which shows a slightly degraded performance. We explain this by a possible over-regularization due to limiting the maximal tree depth to 30 during training.
Remarkably, LSVM displays a slight advantage with respect to classification performance in comparison to all other runs, including the late fusion approach (RF-KSVM-LR). Moreover, the slight advantage of LSVM turns into a substantial gain when taking also run-time eficiency into account. The high computational costs of KSVM and of RF-KSVM-LR are mostly due to expensive Platt scaling.
When comparing test results to cross-validation scores on the development set (see Table 1), it may come as a surprise that classification performance has improved on the test set. We explain this efect by the imbalance and sparsity of training data (Fig. 1), as the stratified sampling strategy for selecting the folds during cross-validation often leads to incomplete training folds (missing rare classes).
In addition to approaches used for run submission, we also
experimented with a hierarchical classifier following the nested
dichotomies approach [
In conclusion, this paper has provided a compelling example of the usefulness of traditional machine learning techniques when combined with CNN-based feature extraction methods for predicting a predefined set of diseases and findings in endoscopic images of the GI tract of the human body. Experiments revealed that both linear support-vector machines and multinomial logistic regression are able to deliver good classification performance at a low run-time complexity in a high-dimensional feature space, learning eficiently from an imbalanced and sparse training set. A more detailed analysis of our approach and a comparison to other submissions will be possible when ground truth labels of the test set and results of all participants have been published.