Binary classification is challenging when dealing with imbalanced data sets where the important class is made of extremely rare events, usually with a prevalence of around 0.1%. Such data sets are common in various real-world problems. Examples are medical diagnostics where the disease is considered a rare event, or multiple types of fraud detection where regular transactions are the most prevalent. The events are categorized as either predicted positive or predicted negative against a certain threshold. In large imbalanced data sets, it is expensive to verify, through human raters, all the predictions made by the classifier. As a result, only predicted positive events are rated. Additionally, in most industrial applications, it is useful to combine multiple classifiers to make a decision. We developed a machine learning pipeline which, combined with expert knowledge, decides on an optimal threshold. ROC curves are reformulated to true positive (TP) versus false positive (FP) curves. We propose two solutions to select an optimal threshold by maximizing the area under the curve (AUC): a graph-based approach and an analytic approach. The graph-based approach constructs a graph to select an optimal path in the threshold space and the analytic approach minimizes an energy function. Our results agree with the Google team’s manual attempts to choose the operating point in their private data sets and binary classifiers while providing a rigorous mathematical framework. Our solutions built on the Google team’s expert knowledge efforts which identified the marginal precisions used in our methods. To further evaluate the performance of our algorithm, we split 3 public data sets into training and testing sets and used them to train five different classifiers. The results show improvement of the f-1 measure by 1.5%, the precision was improved by an average of 5.1% and, the recall was reduced by 1.6% on average. Depending on the purpose of the classification, we show how to reverse this trade-off.
Co-First authors: Both authors contributed equally to this work.
Our study focuses on data sets with extreme rare events,
where the minority class accounts for 0.01% to 0.19% of
the population. Such data is very common in many
applications including fraud detection where fraudsters constitute
a small percentage of users, medical diagnostics, face
detection, money laundering, and intrusion detection. In such
data sets, the distribution of positive to negative classes is
extremely imbalanced and it is typically more important to
correctly detect the positive class
In the rest of this paper, we provide some background about binary classifiers and their performance, we summarize related work and describe the problem statement in detail. Then we propose solutions for single classifier thresholding and for an ensemble of classifiers where two methods are proposed; the analytic and graph approach. Finally, we explain two experiments to benchmark our results and evaluate the performance of our algorithm, one using public data sets, and the other one using private Google data sets.
A binary classifier is a model that classifies its input into two classes; a positive and negative class. In this paper, we call the classifier’s input a document, and its output a score. The higher the score, the more the classifier is confident that the document is in the positive class and vice versa. In the threshold binary model, a threshold number is picked and compared to the output score of the classifier to decide which class is predicted. If the score is larger than the selected threshold, the document is classified in the positive class. If the score is lower than the threshold, the document is classified in the negative class.
To evaluate the performance of the classifier, the documents are often presented to human raters to determine the true positives, the false positives, and the true and false negatives. A higher threshold will cause the predicted class to become more precise on average while losing some recall and a lower threshold will cause the predicted class to become less precise on average while gaining some recall. Note that: Precision =
T P + F N This results in a trade-off where a very low threshold might yield many false positives and a high threshold might yield many false negatives. This trade-off is usually visualized with a receiver operating characteristic (ROC) curve that plots the true positive rate (TPR), defined as versus the false positive rate (FPR), defined as
T P R = F P R =
T P T P + F N
F P
F N + T P
;
;
as suggested in
False positive rate 1.00
ROC and precision-recall analysis are widely used in
imbalanced data set classification, as shown in various types
of rare event data sets in
The iso-performance line method maximizes the
classification accuracy, which places more weight on the majority
class, therefore overlooking the minority class in which
we are interested the most. This method is therefore not
applicable in our case, its pitfalls are discussed in detail in
In the context of credit card assessment, an MCDM
(Multiple Criteria Decision Making) was proposed in
Ensemble classifiers are common methods to combine
multiple classifiers, each one looking at different features of
a dataset. In
Different thresholding methods were suitable for a range of applications, we identified few very important challenges, described below, that are faced when dealing with rare event data sets, which were not addressed before, in our knowledge.
In an extreme rare event problem, if one positive event happens in every ten thousand samples, one would have to label a million samples to detect 100 positive events. Like the data we are dealing with from Google, many datasets will lack information about the true and false negatives, because it is very expensive to have humans spend time rating such a large amount of data. Therefore, it makes sense that the only data classified by humans is the data that scored higher than a given score on the classifier i.e the positive class. Unfortunately, not knowing the true and false negatives, therefore, calls for analysis methods that do not require them.
The second important challenge to address occurs when using multiple classifiers to evaluate a single document. It often happens that we need to look at different aspects of the document using different classifiers before making a decision. For instance, one classifier can be assigned images and another classifier can look for text. Each classifier will output a different score and after a threshold is chosen, it can be compared to each of the score and a decision can be made about whether the document belongs to the positive or negative class. The difficulty arises in finding a threshold that optimizes all the models jointly. This takes us from a singledimensional optimization problem to a multi-dimensional one. In the following sections, we will explore the methods developed to address these challenges in the context of extreme rare event data sets.
For the sake of completeness, we begin with the simple case of one classifier where we start defining the framework of our methods.
Let Xi be the independent and identically distributed (iid) random variable representing the score of positive documents. The number of true positives at a threshold t is the number of positive documents greater than the score t. Hence, for each threshold t we have TPR as:
n 1 X IXi t; n
i=1 1 m where n is the total number of positive samples and IXi t is the indicator function. Similarly, let Yi be the iid random variable of the score of negative documents, then FPR is given as:
X IYi t; m
i=1 where m is the total number of negative samples. The ROC curve is then parametrized as m 1 m
n ! X IYi t; n1 X IXi t ; t 2 R; i=1 i=1 and is expected to converge to (P (Yi t); P (Xi t)). The trade-off optimization problem then becomes one of finding a threshold t that maximizes P (Xi t) and minimizes P (Yi t). However, as mentioned in the introduction, we are addressing the situation where only the portion of the documents having scored higher than a certain threshold are rated. In other words, the distributions are left-censored, we only have IXi t; IYi t for t 0 and we only know the total number of sample m + n but do not have knowledge of either m or n. Hence, we are unable to plot an ROC curve.
Since we lack information about the true and false negatives, we use a reformulation of the ROC curve that plots TP instead of TPR and FP instead of FPR , this removes the normalization to probabilities that a normal ROC does by dividing by the positives or negatives. The reformulation is referred to as the positive-negative (PN) curve as defined in [8]. In our paper, we also refer to it as a partial ROC curve. The appearance is the same between the PN and ROC curve, but the chance-line (diagonal) in the ROC becomes the break-even line as known in information retrieval, a more detailed explanation is given in [8].
Mathematically, m
n X IYi t; X IXi t ; t 2 R 0; i=1 i=1 ! where m and n are the total number of positives and negatives, respectively, having scores greater than 0 for instance.
Given a classifier, a low threshold T can be set, the documents with a score higher than T can be sent to human raters to determine the TP and FP, then in the resulting data set, the threshold is raised to see how the TP and FP change. Note that as the threshold keep getting higher, we would have fewer data points in the sub-sample and therefore a wider confidence interval. Using the partial ROC curve implies that we can not use the rated data to simulate the TP and FP count had the model threshold been set even lower than T.
With the partial ROC curve, we can formulate our optimization problem as follows: Given any point on the curve, if this point was the operating point and we wanted to get one additional true positive document, we can determine the cost that we need to pay in terms of a raw number of false positives.
In many situations, this formulation can relate to a business goal. For example, the positive class could be unwanted documents that a company needs to filter out without losing the wanted documents. Thus, they can provide a number to quantify how many good documents they are willing to give up in order to filter out one additional bad document. This can be answered by analyzing monetary costs for instance. Based on this, a target derivative FT PP can be chosen, again this represents the amount of false positives gained vs the amount of true positives gained at a certain threshold. The target derivative is based on expert knowledge, which was provided for us by Google in our application.
A partial ROC curve can be described as a parametric function
f (t) = (F P (t); T P (t)); where f : T ! R2; where T is the threshold space. A fit can be applied to find the points where the derivative has the desired value. In the result section, we present the fitting methods that worked best for typical partial ROC curves.
In this case, the PN curve (partial ROC) becomes an Ndimensional manifold, ddFT PP becomes a more complicated notion since a local direction must be picked for a unique derivative to exist.
In general, one can define the function as f (~t) = (F P (~t); T P (~t)); f : T^ ! R2; (1) where T^ is the threshold space described by the cross product of all the threshold values for each classifier. We can consider the derivative at a point to be: lim x!0
TP FP u u ; where u is a vector that describes the direction at which the derivative is being taken.
We describe a path (s) = (g(s); h(s)) for 0 s 1 over T^ space, that is, a function : [0; 1] ! T^.
We impose the following restrictions on for the resulting curve to behave like a partial ROC curve: (0) = (inf T1; inf T2; : : : ; inf Tn) (1) = (sup T1; sup T2; : : : ; sup Tn) f (t) and g(t) must be monotonous
The first two conditions are so that the curve starts and ends at the correct points, and the last point is so that the amount of true positives and false positives decrease with respect to our new “threshold”.
To pick the best classifier, should maximize the area under the curve: argmax
T P dF P Z
([0;1])
We propose two algorithms to find , one is based on minimizing a function (the analytic approach) and the other one is based on building a graph (the graph approach).
Before we give a step by step example implementation for both methods, we introduce the following simple base algorithm for both methods to find the optimal operating point. Algorithm 1 Base algorithm for finding the optimal operating point Input: Set of points P representing (T P; F P; T 1; T 2; :::), target derivative d Output: Optimal operating point (T 1; T 2; :::) 1: C FindOptimalCurve(P ) 2: Find point q where ddt C(q) = d
In what follows, we explain the FindOptimalCurve function used in Algorithm 1 in the context of both the analytic and graph approach.
The core of this approach is the usage of a family of functions for which we optimize the parameters in a way that maximizes the area under the corresponding PN curve. This approach estimates the integral yielding the area under the curve using the simple trapezoidal method. The algorithm implementation is summarized in Algorithm 2.
We describe the implementation of this algorithm in more detail using two thresholds, as an example, in the following steps: Algorithm 2 Analytic implementation of FindOptimalCurve Input: Set of points P representing (T P; F P; T 1; T 2; :::), a family of curves fp defined on [0; 1] ! T Output: A curve C 1: Fit a function F^(T 1; T 2; :::) as the best fit for the points (F P; T 1; T 2; :::) 2: Fit a function T^(T 1; T 2; :::) as the best fit for the points (T P; T 1; T 2; :::) 3: Let gp (T^; F^) fp 4: q argmaxpAU C(gp(t))dt 5: return f(t; fq (t))8t 2 [0; 1]g
2. Plotting the three-dimensional plots (TP, T1, T2) and (FP, T1, T2) and fitting a surface through each one of them using an interpolation technique. We create an n n grid on the threshold plane with each point on the grid corresponding to a pair of thresholds and a value of TP and FP. This step allows finding the estimators T^P and F^P at each point of the surface. 3. Turning the grid into a linear space; since the data is normalized at this point, the domain will be from 0 to 1. We can also specify the granularity i.e the size of the partitions in this domain. In the next steps, the granularity will also indicate the number of AUC samples that are calculated in the process.
This step is the start of the parametrization used in this approach. If s denotes the parameter indicating the order of the partitions, then both thresholds are a function of s which is then used as an index in a separate parametric function to output the corresponding TP and FP values separately.
E(p~) = AUC(F^P p~; T^P p~) where:
p~ is a family of parametric functions defined from [0,1] to T1 T2 with p~ being the parameter. In our algorithm, we used a parametric function that is graphically close to the resulting discrete path and that passes through the bounds (0,0) and (1,1): p~ = xp (2) AUC is a function that calculates the area under the curve using the trapezoidal formula, this is an estimation of the integral form of the calculation of the area under the curve described by: Rf(x; p~(x))g TP dFP 5. Minimizing the function below using the Nelder-Mead minimization function.
log(log(E))
Note that we made the previous function negative so we can use a minimization method to maximize the AUC. We also composed E with two logarithms in order to make it converge faster, otherwise, it might get stuck on local minima. 6. Creating a function that determines the curve in the TPFP space that corresponds to the resulting path within the threshold space using the parameterization mentioned in Step 3. 7. Finding the point s where ddFT PP w.r.t. the parameterization s equals the target marginal precision, by estimating the derivative of f . 8. Finding the optimal operating point (T1,T2,TP,FP) given a target marginal precision ie. the values of t where (t) = s. It can be interpolated at the value of s given by the closest marginal precision to the target.Ridders method can be used for root finding to get the best parameter s.
The goal of the graph-based approach is again to find the continuous path on the threshold space that maximizes the area under the curve in the TP-FP space, now with the constraint that every point is only allowed to move through its neighbors.
We defined the node, edge, and weight of the graph shown in Figure 2, for 2-dimensions, as follows:
Node v: represents a pair of thresholds, having attributes of (T1,T2,TP,FP) Edge e: the connection between the nodes is established if and only if one of their thresholds is equal or their other threshold is consecutive in the discrete set of thresholds. Weight of edge e = (v1; v2): this is equal to the negative of the area under the segments (trapezoid) (FP(v1); TP(v1)) and (FP(v2); TP(v2)) as shematized in Figure 3. Mathematically, the weight is defined as: w(e) =
Then the total weight of edges in a path is the negative of the area under the curve swept out in the TP vs FP plot. In another words, by fixing end points, we can find a better TP vs FP curve by finding a path that has a minimal weight. So, it suffices to find the shortest path from v0 to v1 ,with (t1(v0); t2(v0)) = (0; 0) , (t1(v1); t2(v1)) = (1; 1) . Intuitively, it finds a shortest path from the left bottom corner to the right upper corner in the thresholds space T1 T2 . The edge is directed towards up or right to avoid bent paths in the corresponding TP vs FP curve.
After we find the set of nodes in the shortest path, we connect the corresponding points and re-parametrize the path. Note that it would be continuous but not smooth. The algorithm for this method is the following: Algorithm 3 Graph based implementation of FindOptimalCurve Input: Set of points P representing (T P; F P; T 1; T 2; :::) sampled in a grid with respect to the thresholds Output: A curve C 1: Create a graph G connecting points to neighbors with a greater coordinate than them 2: Let the origin be A0 and the point with the biggest coordinates be A1 3: P FindOptimalPath(G; A0; A1) 4: Let f (t) : [0; 1] ! F P T P such that it passes through the path P 5: return f(t; f (t))8t 2 [0; 1]g
Below we explain this approach in detail, starting from step 3 of the analytic approach for a 2-dimensional example, since the first two steps are similar. 1. Transforming the threshold space into an n n grid with each point on the grid corresponding to a pair of thresholds and a value of TP and FP. 2. Calculating the weight of the graph as the negative area under the segment formed by 2 points v1; v2 in the TP vs FP curve using equation 3 3. Determining the weight of the edge connecting the node of two consecutive points.
Note that the predicted positives have to be non-negative, hence the segment is above the x-axis. The weight is non-positive therefore a lower weight indicates a higher area under the segment. 4. Store the (TP; FP; T1; T2) tuple in every node as extra information. The edge can then be defined as a 3-tuple of (nk; nl; wk;l), where nk and nl are each node and wk;l is the negative area under the curve of each node.
The nodes are connected if they are within the same neighborhood. A function can do so if the one of the thresholds is the same between two neighbours or the thresholds are consecutive to each other in the discrete set of thresholds.
The path selected is therefore just a collection of nodes from the left bottom to right upper corner in thresholds space and the total weight of the path corresponds to the negative area under the curve swept out by it in the TP FP space.
Note that in order to compute the shortest path between the lower-left corner (source node) to the top right corner node, the Bellman-Ford algorithm can be implemented since it accepts edges with negative weights unlike the usual Dijkstra’s algorithm. 5. Connecting the nodes to obtain a continuous path in the threshold space and determining the curve in the TP-FP space that corresponds to the selected path. It does so using the same parametrization explained in step 3 of the analytic approach. 6. Getting the marginal precision, refer to step 7 of the analytic approach. 7. Determining the optimal operating point given a target marginal precision, refer to step 8 of the analytic approach.
To evaluate the performance of the method, we use rare event data sets where the amount of false negatives and true positives can be calculated, in order to compare the precision and recall of two standard classifiers with our methods on the resulting ensemble of classifiers. This was done using three rare event datasets123 whose true and false positive and negative values are known.
Each dataset was cleaned by changing empty values for the average value in the column, turning categorical datasets into one-hot encoded vectors, and encoding cyclical values like days of the weeks into points in a unit circle. These datasets were then passed through a PCA procedure (Principal Component Analysis) to reduce the dimensionality. The data sets were then split into a training set T and a holdout testing set H, where T represents 75% of the data set and H represents the other 25%.
Each classifier was trained using the first 20% of the components of the PCA. We need to simulate a joint optimization of two classifiers where each one of them makes decisions over a separate but related data set. Thus, one of the classifiers was assigned the remaining even components and a second classifier of the same type was assigned the remaining odd components.
The hyperparameters, for each classifier, were tuned by maximizing the average accuracy in a cross-validation test. A cross-validation test was done by splitting the training set T into 5 different cross-validation groups: T1; T2; :::T5, where for each training set Ti, a classifier is trained using the set Sj6=i Tj and the score for the ith fold is denoted by si and is obtained by evaluating the accuracy of the classifier on Ti. The hyperparameters with the highest average value of s1; :::; s5 are picked.
The classifiers used for the testing were Linear SVMs, Random Forests, Decision Trees, k-nearest neighbors, XGBoosting and Naive Bayes.
We explored multiple classification techniques to determine whether our methods improve their performance. We relied on (Pedregosa et al. 2011) for a compilation of classification algorithms. In all classifiers, we estimated the hyperparameters through a grid search with cross-validation. For the k-nearest neighbors classifier, we tested all the values of k from 1 to pn where n is the number of points in the training data, as well as the L1 and L2 metrics, the default behavior for classification, is to pick the class with the biggest number of nearest neighbors assigned to them. The decision tree classifier used the grid search coupled with the Gini and entropy criterion. We optimized the maximum depth parameter for all integer values between 1 and 500 as well as without setting a maximum depth, the default classification method is to follow the instructions for the tree. The random forest classifier also used the Gini and entropy criterion with the number of estimators between 50 and 500, with steps of 10 between them, the default behavior for classification is to let each decision tree in the forest vote, and the class with the most votes wins. For the extreme gradient booster (XGboost), we used a maximum depth from 1 to 20, a learning rate from 0.1 to 1, with steps of 0,1, and a number of estimators from 50 to 500 with a step of 10, the default classification behavior is the same as random forests. For the linear SVMs, we again used the grid-search with crossvalidation to look for the hyperparameters between using L1 and L2 norm for the penalty, the default threshold used was 0. For SVMs, we used the hinge or squared hinge loss functions to train the classifier with a dual true or false and using all values of the regularization parameter from 0 to 1 with a step of 0.1, the default threshold is 0. We also used a Naive Bayes classifier with no optimized parameters, whose default classification behavior is to choose the class with the highest probability.
A grid was made using the result of the cross product of the thresholds of equally spaced points in each dimension. An input would be considered to have a true value if either of the scores produced by each classifier surpassed their own threshold. This means that we are taking the union of the classification. Using the scores outputted by the classifiers, we calculated the amount of false positives and true positives for each pair of thresholds in the grid. This was the input for Algorithm 1. We then calculated the precision and recall from the testing set.
To evaluate their performance, the F1 measure was used instead of the accuracy score. This latter is not suitable for rare event data sets where a difference in accuracy of 0.01% can make a huge difference in both the precision and recall. The F1 measure is calculated by
F1 = 2
pr p + r ; where p is the precision and r is the recall.
Figures 4, 5 and 6 show the F1 measure, precision, and recall, respectively, for each experiment. An experiment was defined as a combined pair of a dataset and a classifier type. When either the precision or recall were extremely low, the other quantity is ”undefined”. In this case, taking the union of the classifiers’ decisions results in classifying every document as positive, which makes the ensemble classify every document as positive.
Overall, the precision was increased on average by 5.1% while the recall was reduced by an average of 1.6% and F1 measure increased by an average of 1.5%.
Table 1 and Table 2 of the appendix show the results for all the simulated experiments with all the mentioned classifiers. Undefined values are showed as - on the tables. As the F1 measure of the classifiers that do not use the proposed methods increases, the F1 measure of the ensemble resulting from the proposed methods also increases. This trend is shown on figure 7. It also plots the identity line to show that the proposed methods improve the F1 measure of all the points above this line.
The Google data contains 22 data sets each consisting of about 10,000 rows and 4 columns of data. The two first columns represent two thresholds of two different classifiers and the next two represent the number of true and false positives for each pair of thresholds as determined by human raters, assuming all documents over the threshold is considered as part of the positive class. These datasets, have a prevalence of the positive class of about 0.01% of the total number rows.
The result of running Algorithm 2 and 3 is shown in Figure 10 for an example data set. The figure represents the threshold paths obtained using both the analytic and graph approach, as well as the location of the operating point.
Using the parameterization from Equation 1, we can use the threshold paths to determine the optimal partial ROC curve that was selected by each of these algorithms.
Next, we can determine the optimal operating point given a target marginal precision as determined by a business goal set by Google’s expert knowledge efforts prior to this work. The result of running Algorithm 1 is shown for 3 different methods in Figure 8 in the threshold space and correspondingly in Figure 9 in TP vs FP space.
The analytic and graph methods mentioned in figures 8 and 9 are the proposed methods, while the benchmark is a fine-tuned operating point used by Google based on separating the false positives into small intervals called buckets, and picking the highest TP for each bucket, then adjusting this manually to fit the goals of the optimal operating point. While this latter is good as a benchmark for our algorithms, it can not be used as another optimization method since it lacks mathematical rigor and produces an erratic behavior in the threshold space which, without fine-tuning, would imply the non-existence or non-uniqueness of the derivative of the partial ROC curve needed to find the operating point. Our methods make a good attempt at approximating the operating point selected by the benchmark method. These points are also plotted on their respective paths in Figure 10.
We used an or (union) in combining the decisions of the classifiers but it can be substituted by an and operation depending on whether the application prioritizes positive classification or negative classification. Our methods also assume that the ROC curve for both of the classifiers has a unique decreasing derivative which can be expected from a tuned classifier.
Both the analytic and graph methods were presented as although both useful, there are different disadvantages to each one of them. The analytic method is heavily dependent on the family of functions used to optimize over, and is prone to get stuck in an optimal minimum. On the other hand, it is uncertain whether the graph method is prone to overfitting to the data or not, however, unlike the analytic method, there is no way to mitigate it. A possible alteration to be done to mitigate overfitting with the graph method would be to hold a series of simulations where some connections between the nodes are deleted, and then picking the average path between the resulting set of paths.
Both algorithms scale reasonably well with respect to the amount of data. The graph algorithm scales at a rate of O(nk), where n is the number of rows and k is the number of classifiers. This is good when the number of dimensions is low, however, it scales exponentially with the number of classifiers in the ensemble. The analytic approach works the opposite way since it depends on an optimization problem, it does not scale very well with respect to the number of rows, however, it scales linearly with the number of dimensions. We compared the running time of the algorithms using a computer with an i7-6700 and 32GB of RAM for all the Google data sets combined. The analytic approach ran on average 10 times faster than the graph approach; for n = 10; 000 and k = 2, the analytic method takes 60 seconds while the graph method takes 6 seconds in our conditions.
Finally, the experiments were done using an automated method to tune the classifiers, notably the grid-search with cross-validation in order to obtain a large number of samples across different accuracies. However, this could be improved since it does not reflect a real-world application of this process in which the hyperparameters would be estimated with the help of additional fine-tuning methods that depend on the specific application of the data set.
The proposed algorithm finds optimal thresholds for the output of binary classifiers used for rare event datasets. The method works by finding the best possible path in the TP - FP space, which is the one that maximizes the area under the curve of the projected partial ROC curve. This curve is then used to find the optimal operating point corresponding to a given target derivative. The latter represents the tradeoff of lost false positives to gained true positives which was obtained from the Google team’s expert knowledge efforts prior to this work. Two optimization methods are proposed: an analytical method, which optimizes parameters of an energy function and a graph method, which performs the optimization through finding the path in a graph to maximize the AUC.
Our results were very close to the optimal operating points that were picked manually by the Google team. We were, therefore, able to automate their process of finding an optimal threshold while providing a mathematical framework and using their target derivatives(marginal precisions). Moreover, we compared the F1 measure, precision, and recall of combining five classifiers and three datasets while either relying on our thresholding methods and taking the union of the decisions or using the classifiers individually with their default classification behavior. The result improved the precision while lowering the recall less significantly. This tradeoff could be reversed, if desired, by simply changing the ensemble decision between the individual classifiers from ’or’ to ’and’. We discussed important limitations and possibilities for future work. Notably, optimizing different metrics other than AUC individually or in combinations would be worth exploring. Moreover, we optimized two binary classifiers jointly but the theoretical framework described in the methods section extends to N dimensions. Using this in practice will require optimizing over multiple parameters in the analytic approach and extending the graph to an N-dimensional lattice. Exploring this possibility through practical applications might open great avenues for the general optimization of ensemble classifiers.
We would like to thank our teammates Ching Pui Wan and Joanne Beckford, NSF grant DMS-0931852, Payam Refael and his team from Google, our academic mentor Aliki Mavromoustaki, and all of the IPAM staff at UCLA for making this work possible.
Machine Learning
Ensemble k-nearest neighbors