Active learning facilitates the training of classifiers by selectively querying the user in order to gain insights on unlabeled data samples. Until recently, the user had limited abilities to interact with an active learning system: A sub-selection was presented by the system and every sample within had to be annotated. We propose an alternative and graphical solution to active learning called MapView where the user may profit from a different interpretation of the underlying data. Experiments underline the usability and advantages of our approach during the training of a classifier from scratch.
providing information on the original features in the graphical visualization as well as confidences and proposed class labels from the classifier. This gives the user a more complete picture of the scenario. Also all samples are presented, allowing relabeling of samples and even adding new classes during training.
Our proposed method is independent from the classifier choice as long as the classifier result contains an
additional probability measure about how certain the classifier is in its decision. We take advantage of a
Random Forest classifier [
We propose a graphical approach to support the user in training a classifier via active learning. The scenario is multi-class classification with pool-based sampling with uncertainty sampling as query strategy.
The main information displayed in the graphical representation is: current labels, current predicted labels,
certainty of the prediction and sample coordinates in feature space. The last providing the user with a feature
analysis view. This kind of view is described as enhancing the user’s understanding of predictions and trust in
the underlying classifier in [
As classifier, we select Random Forests (see [
Within the scope of our classifier, we use the term event for any item in our database (this can be e.g., an image). Let Ei be the i-th event in our database, consisting of i = 1, ..., N samples. Each event Ei has an F -dimensional feature vector xi = {x1, ..., xF }. The goal is to assign one of K class labels to the image. For this purpose, for each event Ei a class vector ci = {c1, ..., cK } exists, with cj ∈ [0; 1], j = {1, ..., K} representing the probability of belonging to class j.
Since our aim is to embed the high-dimensional events on a user-friendly 2D map, it is necessary to perform
dimensionality reduction. Dimensionality reduction means finding an embedding e : Rd → Rc that transforms
high-dimensional data from its original d-dimensional space into representations with a reduced number of
dimensions c (with c d) [
There exist a variety of dimensionality reduction methods (see [
E = P 1
X (di∗j − dij )2 i<j di∗j i<j
di∗j
where di∗j is the distance between the i-th and j-th event in the original space, and dij the distance in the
embedding. As distance measure the Euclidean distance was selected. The minimization is performed iteratively
with a random initialization and a heuristic approach as proposed by [
K X kj · λj j=1 with PK
j=1 λj = 1. In our application area, the number of cluster centers is typically smaller than the dimension d of the feature vectors. Hence, for each feature vector, we computed a least squares solution of the overdetermined equation system in Eq. 2. In general, the number of cluster centers need to be restricted to min (d, K) in order to guarantee a unique solution of Eq. 2. By choosing λ1 = 1 − PK j=2 λj , one can reduce the constraint on the lambdas by solving: xi − k1 ≈ (k2 − k1, . . . , kK − k1) · ... | {kz } λK λ2 which is equivalent to finding a least squares solution of Eq. 2. Since the matrix k is independent of the feature vectors xi, its pseudoinverse k+ enables an efficient computation of the barycentric coordinates of all feature vectors: λ1 ... λK λ1
+ = k · (xi − k0) = 1 − PK j=2 λj After embedding the cluster centers kj via SM, the embedded feature vectors xi∗, i = {1, ..., N } of all events are computed using Eq. 2 by substituting kj with the embedded cluster centers k∗. j
We implemented a simple user interface depicting the proposed MapView of a given dataset (see Fig. 1). In our setting, all events are displayed as points on a 2-dimensional map. Instead of presenting the user the raw view of a pre-selected set of interesting instances (e.g., unlabeled images with pre-computed features), the user gets a graphical overview. Within this view, events are displayed as points with the transparency of the color proportional to the certainty of the classifier – i.e., the more certain a classifier is about classifying a specific event, the less visible (and thus, more transparent) the corresponding point is on the map. This results in a highlighting of interesting samples that is intended to attract the user’s attention, while samples with a high classifier certainty are blurred out. We also decided to show both the manually assigned label (area of the circle) as well as the classifier’s prediction (circular ring) pretending the label is unknown. The user can interact with the MapView via different ways: (i) zooming in/out and translating, (ii) getting a preview of a selected event (e.g., image thumbnail and features), (iii) assigning a class label to the event (via a pop-up window that shows a preview of the underlying raw image or features), and (iv) applying a (re-)training of the classifier. 3
For our experiments, we used an activity recognition dataset containing accelerometer and gyroscope sensor
signals collected by waist-mounted smartphones (see [
We attempted to train a classifier from scratch by excluding the class labels for our active learning approach and only using them for labeling the query samples. We started with a random labeling of one sample per (2) (3) (4) (5) (a) Legend (b) Initial map
(c) Result after one labeled event per class (d) Result after 6 labeled events per class class and a subsequent training of the classifier. This step is depicted in Fig. 1(b) with each color representing a class label (see Fig. 1(a)). The result after training the classifier on these labeled samples is shown in Fig 1(c). Obviously, the certainty of the classifier is very low at this point as nearly all samples are depicted opaque. Afterwards, we labeled five uncertain samples per class and again trained the classifier (the result is in Fig. 1(d)).
The results are easily interpretable and match our assumptions: As one can see, the active activities (walking, walking down, walking up) form a cluster in the lower left region of the map, whereas passive activities (laying, sitting, standing ) are centered on the upper right half of the map, with laying being even more separated from the other two classes. Postural transitions (lie to sit, lie to stand, sit to lie, sit to stand, stand to lie, stand to sit ) are spread in between the other two clusters. The average accuracy as shown in Tab. 1 evaluates the classifier’s performance after the initial setup of only one labeled event per class, and after 5 additional samples. We also evaluated the 12 classes by dividing them into passive and active classes as described before, as well as into 3 postural transition classes of the pairs lie to sit/lie to stand, sit to lie/sit to stand, stand to lie/stand to sit. Apparently, after labeling only 5 additional events per class, the average accuracy increases up to 60%. When considering the combined groups with only 1 labeled event per (original) class, the accuracy even rises to 97%.
The advantage of our method is that it is faster to calculate the embedding only for the cluster centers instead of all samples of the dataset. In addition, the map also allows a graphical interpretation of the dataset and the classifier’s behavior.
Per group 1 labeled sample per class 0.47 0.97
samples per class 0.60 0.98
The proposed method implements a graphical representation of an underlying dataset for classification via active learning. Instead of presenting the user a sub-set of interesting samples and requiring their annotation, the samples are depicted as 2D points on a map (called ‘MapView’) with the color corresponding to the class label. This results in several advantages such as (i) using active learning to avoid the annotation of a large amount of data, (ii) gaining insight into the high-dimensional feature space of the data within the 2D view (e.g., simple identification of cluster samples or outliers), (iii) getting feedback of the classifier’s certainty in labeling the samples (samples about which the classifier is uncertain are plotted opaque, whereas the more certain a classifier is, the more transparent the 2D point representation), and (iv) straightforward interpretation of the classifier result and improved understanding of the classifier’s decision (e.g., understanding why the classifier made or is uncertain its decision). In the future, we will consider feature selection before the embedding process in order to reduce the high-dimensional space to the most promising features. Additionally, other dimensionality reduction or embedding methods might be worth experimenting with.
This work was funded via the projects ‘rollerNIL’ within the call ‘Produktion der Zukunft’, and ‘MVControl’ within the call ‘IKT der Zukunft’. Both programs are supplied by the Austrian Research Promotion Agency (FFG), and promoted by the Austrian Federal Ministry of Transport, Innovation and Technology (BMVIT). [10] J.W. Sammon. A Nonlinear Mapping for Data Structure Analysis. IEEE Transactions on Computers,
C-18(5):401–409, 1969.