In this work, we introduce a framework for synthetic data generation for academic performance prediction formulations. A common problem in these academic performance prediction dataset is that outcomes/grades are not distributed evenly, leading to class imbalance. This poses a challenge for predictive machine learning algorithms to learn important characteristics at the edges of the target class distribution. We present a general framework for synthetic embeddingbased data generation (SEDG), a search-based approach to generate new synthetic samples using embeddings to correct the detriment e ects of class imbalances. We compare the SEDG framework to traditional synthetic data generation methods and relate the framework in relation to deep generative models. In our results, we nd SEDG to outperform the traditional re-sampling methods for deep neural networks and perform competitively for common machine learning classi ers at the student performance task in several standard performance metrics.
In the educational domain, academic performance
prediction approaches are at the crux of degree planning and early
warning systems. Classi ers that seek to predict the
performance of a student given input features can be biased
because these datasets are often skewed centrally to the mean
as a relative grading scale is often employed. The limited
number of examples belonging to outlier students, speci
cally those demonstrating success and failure (needing
interventions/help), are not su ciently represented. As these
predictive models are useful for identifying at-risk students
or helping students plan their degree and career pathways,
this imbalance directly con icts with this purpose. We
refer to these examples belonging to the minority classes. In
contrast, there exists an overwhelming number of students
with average grade scores. We refer to these examples
belonging to the majority classes. This evident disproportion
results in classi ers failing to understand and learn how to
classify within the minority classes, which was supported in
Copyright ©2021 for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0)
[
Deep learning methods have also exhibited advances in
robustness, e cacy, and allowing end-to-end training for SDG
methods as seen in [
In this paper, we introduce an novel framework for synthetic embedding-based data generation methods, called SEDG, to target the e ects of class imbalances on academic performance prediction tasks. In Section 2, the academic performance dataset, the performance metrics, and the classi ers used will be described. Further details of SEDG framework and deep generative models will be discussed in Section 3. In Section 4, the results are presented and evaluated.
In this section, we will discuss the details of the Student
Performance Dataset from [
The dataset provided by [
It is important to consider an appropriate performance
metric for datasets with class imbalance, as [
We will use three traditional machine learning classi ers (support vector machines, random forest, gradient-boosted decision trees/XGBoost) and an neural network architecture which we will denote as NNModel.
Support vector machines (SVM) [
L(p; q) =
8x 1 X p(x) log(q(x)); jxj = N N (1)
We will describe the SEDG approach step by step, dividing the discussion into 4 chronological parts: sample selection, feature selection, feature modi cation and synthetic sample usage. In each step, we o er various design considerations that we have tested. For future works, we encourage new variations and designs of this general framework. We will then write on deep generative models in the context of the SEDG framework.
We must rst consider how to select the samples from the original dataset to base the synthetic samples on. We can set how many samples k we want to select to be xed constant or stochastically chosen within a de ned range.
Naively, we can randomly select k samples from the entire dataset D, all with equal probability. Once the samples are selected, we place them into a sample pool, S, that will be used to generate the new synthetic samples. This approach will be referred to as random sample selection.
We can de ne a sample selection approach that partitions the dataset D and select a member Mi of the partition P = fM0; M1:::g to sample from. The member selection can occur at a probability p or be deterministic. The member will be stochastically sampled from, with or without replacement. Consider we want k samples from the datasets, we can implement a partition P , where the equivalence relation is de ned by the samples' associated class. We set each member of the partition Mi 2 P to be associated to a selection probability pi proportional to their cardinality pi / jMij. The selection probability represent how likely the member will be selected. We choose to de ne the surjective mapping from the partition P = fM0; M1:::Mng to the probability distribution p = fp0; p1:::pmg with a hashmap for simplicity. A special case would be to deterministically select the member Mi based on the cardinality, where jMij jMjj for all j 2 [0; n]. Once a member Mi is selected, we sample m examples from the set, where m k. The value of m, similar to the value of k, is a hyper-parameter. The m samples are stochastically selected from M by performing random sample selection, and we add this to our sample pool S. We repeat this process until jSj = k. This approach will be referred to as partition-based sample selection. We note that this approach is dependent solely on the dataset, and remains entirely independent on the classi er.
We formulate a sample selection approach to select samples based on the classi er's performance, similar to concept of boosting. Speci cally, we can associate higher selection likelihoods for samples the learner has the largest margin for error or uncertainty. In this context, we can consider classi cation accuracy to quantify error and uncertainty. Thus, we can place the highest probability of selection to the samples that were misclassi ed, and rank the correctly classi ed samples based on the classi er's certainty of the prediction. So, given a classi er C and an loss function L(C; s), where s 2 S is a sample belonging to the training set S, we let the selection probability pi for xi to be greater than the selection probability pj for xj if and only if L(C; xi) > L(C; xj). The loss function returns a scalar value that represents the correlation between the target and prediction values. This approach will be referred to as performance-based sample selection. We note that this approach, in contrast to partitionbased approach, is dependent on the classi er, thus also requires trained model for its operation. Consequently, the performance-based sample selection method is more computationally costly.
We now can formulate a selection method that incorporates from both partition and performance sample selection approaches. In this paper, we use the class-AUC score to do so. We rst partition as described above and use the class-AUC score as our uncertainty metric to develop a selection probability distribution to select the member to sample from. Thus, we implement a class partition to rst select a member to sample from, and we incorporate the AUC score from the classi er to develop the probability associated to member selection. We call this the partition-performance sample selection approach.
At a more granular level, we can think of every sample Si 2 S from the dataset D, where S D to be made up of features. For example, a sample Si = ff1i; f2i; f3i:::; fng can i be de ned to be a set of features where jSij = n. We de ne equality as Si = Sj if and only if fki = fkj, 8k 2 Z. After sample selection, we can modify the selected samples to generate new samples. Thus, given some mapping from Si to Si0 where : ff1i; f2i; f3i:::g ! ff1i0 ; f2i0 ; f3i0 :::g would provide us new samples Si0, where Si0 6= Si. But in order to modify these features, we must consider how to select which features to modify e ciently as to appropriately increase variance in the dataset without creating class overlaps. But, to modify the samples, we must rst select the features of the sample that we wish to adjust. We will discuss three approaches to select features, which we will call random, imbalance-based, and importance-based feature selection approach. We can label the imbalance-based and importance-based feature selection methods together and classify them as weighted feature selection.
The random selection approach, similar to random sample selection, stochastically sample a subset of feature f i Si = ff0i; f1i; f2i:::fki g, where jf ij jSij, with all features with equal probability. The number of feature selected k = jf j can be xed or randomly chosen per sample.
We can weigh the feature selection based on a metric based
in feature value imbalances, as seen in Figure 2. We
introduce a metric that quanti es this imbalance by using the
following method, which di ers from past works [
The feature importance approach creates a probability distribution p based on feature importance, which will be used to select the features. Feature importance provides insight into the relationship between the predictive system and the data, and how much signi cance each feature in the data may have on the overall system's performance at the given task at hand. We will consider three approaches to calculate and obtain the feature importance: mean decrease impurity importance, permutation importance, drop-column importance.
Mean decrease, or gini impurity, importance methods rely
on the use of tree-based classi ers, and is calculated by
aggregating the gini decreases of each feature. This is criteria
used to determine the splits at every tree. However, [
We now consider how to modify the selected features optimally. We de ne optimality as maximizing the variance between the synthetic samples from original samples while minimizing the class overlap in the new dataset. Our objective is to maximize the improvements in the minority classes, since that is the de cient area of performance. To modify the features, we can naively inject noise to the continuous features and replace discrete features with sampling, but we wish to formulate a more targeted approach. In this paper, we will focus on embedding-based modi cations, a searchbased method where we will leverage learned embeddings to o er insights in how to optimally modify the features. For future works, we encourage di erent methodologies to feature modi cation.
Embeddings are mappings from the raw feature domain to a
domain that can be more useful and understandable for some
classi er to perform the task at hand. In other words, we
can think of embeddings as optimized pre-processing
transformation. We consider two methods of embedding
generation in Section 3.3.1 and Section 3.3.2: weights from
transfer learning (WTL) and latent representation from
autoencoder (LRA). There are many other embedding generation
approaches [
We now formulate a method for feature modi cation using embeddings. Recall a sample S can be thought of as a set of features fs. These features can either be discrete and continuous. We will consider both cases, and discuss how we can handle each case separately. Assume a set of features fd S that was selected for modi cation is homogeneously discrete, for each feature fd;i in fd, then there exists a set of possible feature values fd;i can take on yi = fyi;0; yi;1; :::yi;kg. We can sample a subset ci of yi, where jcij jyij and if fd;i = yi;m then yi;m 62 ci. If the search space for unique feature values yi is small enough, we can try all possible values for fd;i, however if it is large, then we could sample a subset as mentioned. Given the embedding mapping i : yi ! Ei, we calculate the similarity score between i(fd;i) and i(yi;j ) for all yi;j 2 ci, and replace the feature value with the one with the highest similarity score. We repeat this process for every features in fd. In other form, given a embedding mapping : S ! E, where the input is the entire sample, we will again follow the same procedure, however we must compare (S) and (S0) where S = ff0; f1:::fi; :::g and S0 = ff0; f1:::fi0; :::g where fi0 2 ci. When we repeat this process for each feature in fd, we can choose to replace S to the updated S0 for the proceeding feature modi cations, or choose not to replace S. To place higher emphasis on lower class overlaps, we can place a soft constraint on the similarity score using the mean similarity score from most similar, or randomly selected, samples to S from di erent classes. To place higher emphasis on class variance, we can set a threshold value for the similarity score, or set a soft constraint on the similarity score using the mean similarity score from different samples from the same class as S. Now we assume a set of features fc S that was selected for modi cation is homogeneously continuous. For each element fc;i in fc, we de ne a range with a xed step yi = fyi;0; yi;1; :::yi;kg where fd;i 2 (yi;0; yi;k) and yi;0 < yi;k. Then, we proceed similarly to the discrete feature case for feature modi cation.
The WTL approach consists of learning the embedding mapping through training an model F on some relevant task. For example, we choose the task to be classi cation of stu= F = F r
L(y; y^) r F LF (y; y^)
Auto-encoders are parameterized functions learn to compress and decompress the input data using unsupervised learning. The architecture thus has a bottleneck structure. Let the set of layers l = fl0; l1:::lng be the layer that make up the autoencoder, and the cardinality jlij for i 2 [0; n] be the number of parameters in layer li. Given 0 a < c n and b 2 (a; c), there exists an unique lb such that jlbj < jlaj and jlbj < jlcj, where we will refer to lb as the bottleneck layer. We de ne = fl0; l1:::lbg and D = flb+1; lb+2:::lng. The aim of the autoencoder is to accurately build a reconstruction of the input given this decompression. Inference on auto-encoders can be seen in Equation 4, where x^ is the prediction, and x is the input data. dent performance. We note that both and F are parameterized functions by their own independent weights and F . Thus, inference on the model can be seen in Equation 2, where y^ is the prediction, and x is the input data.
E^ = (x)y^ = F F ( (x)) (2) Following the speci cation from the example above, a classier F and the embedding mapping can be optimized using the same loss function L, de ned in Equation 1. With gradient learning, the weight updates can be seen in Equation 3. (3) (4) (5) E^ = (x) x^ = D(E^) Auto-encoders are optimized given the reconstruction loss L, and the target is the input data x. Typically, the loss function L will compare the input x to the predicted value x^ directly, using a function like mean-square error or cross entropy. If gradient learning is used, the weight updates can be seen in Equation 5.
= D = D r
L(x; x^) r D L(x; x^) Once the auto-encoder is optimized, we can export the compressing subset of layers as the embedding mapping. We call this procedure LRA.
We also consider variational auto-encoders (VAE), which follows the same inference and optimization methods, however, we represent the bottleneck layer lb as a probability distribution, often a Gaussian N ( ; ). Thus, the input of layer lb+1 will be samples from the distribution from lb. The loss function used is the empirical lower bound (ELBO) loss, which combines the reconstruction loss normally used in auto-encoders and the Kullback{Leibler divergence loss term KL(P (zjx); PD(z)), where is a mapping that uses the layers fl0; l1; :::lbg, D is a mapping that uses the layers flb+1; lb+2; :::lng and z is the latent space representation, or the output of the . Thus, z will be considered to be the embedding of the sample.
Once we obtain the synthetic samples, we now must decide how the samples will be utilized. We evaluate two considerations: cold or warm start and iterative or non-iterative. Initially, the model learns on the training set Dtrain. Then, the set of synthetic samples Ds;t is appended to the training set Dtrain. We can choose to either re-initialize the weights of the model, which we will call cold-start, or keep the weights from the previous training cycle, which we will call warm-start. By doing cold starts, we remove the bias from the previous dataset, but lose what was learned from that dataset. We consider whether to repeat the process of creating a new Ds;t+1 from the newly trained model, making the process iterative, or end the training cycle entirely, making the process non-iterative. If the process is iterative, we can choose to dropout and replace the previous Ds;t partially or wholly with Ds;t+1, or append Ds;t+1 to the current Ds;t at each step using a nite queue system. Another consideration is whether to re-sample the distributions used for sample and feature selection.
In this section, we will discuss deep generative models (DGM) in context with the SEDG framework.
Generative models can model the conditional probability P (xjy), the joint probability P (x; y) or the prior P (x). A DGM is expressed using a deep neural network parameterized by learnable weights . Equivalently, DGM are mappings f : y ! x, f : x ! x, or f : x; y ! x. In context of SEDG, DGM can be viewed as an end to end approach to generating synthetic samples, bypassing the need for Section 3.2, and 3.3. For our purposes, the objective is to learn how to generate new samples following the de nition of optimality discussed in Section 3.3. We can set the input of the DGM to either be random noise, or samples selected using methods from Section 3.1. A potential issue is the possibility of the generative model learning to map the input to itself without any variance, especially for traditional autoencoders and given the nature of the optimization. This paper does not propose a proven solution to this issue. In fact, we address this issue by simply preventing over- tting with early stopping with heuristics and holding out a subset of the training dataset, discussed in further detail in Section 4. Now, we will discuss two main approaches to training DGMs: unsupervised and adversarial training.
Unsupervised training refers to learning the prior P (x), thus in more practical terms, optimizing the model using only the input x. An example of unsupervised training is training auto-encoders, where we treat the input as the output as well. Instead of exporting the embedding function, as seen in Section 3.3.2, we treat the auto-encoder as the generator model f : x ! x^, where x^ is the synthetic sample. In this paper, we will solely evaluate DGMs using auto-encoders and VAEs.
Adversarial training adds onto unsupervised learning with the discriminator model, which learns to discern real or fake data. At a high level, the generator model attempts to trick the discriminator model by learning how to generate samples that are realistic. We will consider the traditional approach and the conditional approach.
The traditional approach is to train the generator and
discriminator models separately using unsupervised and
supervised learning respectively. The training for the generator
model will follow the procedure discussed in Section 3.3.2,
however we aggregate the discriminator's inaccuracy and the
reconstruction loss, scaled by and 1 respectively, where
j j 1, to obtain our generative model's loss. The
training of the discriminator will optimize the classi cation loss
between real and fake data using the dataset and the
generator model. The conditional approach follows most of what is
stated in the traditional approach, however we aim to have
both the generative and discriminative models conditioned
on the class label, y. Similar to practices discussed in [
In this section, we will evaluate the methods discussed in Section 3 using the student performance dataset and classi ers described in Section 2. We will compare our results with other balancing methods from past works, such as random oversampling, random under-sampling, SMOTE, Tomek Links, extended nearest neighbors, and combinations methods of these approaches. We see that if the classi ers are trained on the dataset normally, optimizing the cross entropy loss, we obtain performance seen in Figure 5, which follows our hypothesis of disproportion performance between minority and majority classes. We seek to mitigate these effects of class imbalances with the SEDG framework we have proposed.
We now will discuss our results from our experiments using the SEDG and DGM methods mentioned in Section 3 and using the NNModel and the traditional models mentioned in Section 2.3 as the classi er. For all of the experiments below, we set the number of synthetic samples that will be generated to 100 samples. All recorded improvements are on the testing set, which encompasses 40% of the dataset. We de ne % improvement P I in Equation 6, where M (ysyn; y^syn) is the score from some performance metric M from the dataset with the synthetic samples, and M (y; y^) is the score from the performance metric M from the original dataset.
P I = M (ysyn; y^syn)
M (y; y^) (6) We consider using the NNModel de ned in Section 2.3 as our classi er, and we will test our embedding-based SDG methods and DGMs on the student performance dataset. In Figure 6, we nd that the SEDG method outperforms all traditional sampling methods for all performance metrics when using NNModel as the classi er. Even given 50 independent trials to run, many of the classic sampling method perform poorly, some even harming the performance. Most notably, the class-AUC score demonstrate a signi cant positive di erence in targeting the minority classes when using SEDG methods.
Now we consider the traditional classi ers seen in Section 2.3, and how SEDG methods a ect their performances. First, we show that when we transfer the embeddings from the deep models, to act as data pre-processing mappings to these learners. We nd their performance increases noticeably, as seen in Table 1. This supports and allows us to proceed to use embedding-based SDG methods without worries of the embeddings being incompatible to these classi ers. In Figure 7, we nd that the SEDG method, while it does not excel in accuracy and macro-AUC score % improvements for traditional machine learning classi ers, it is able to target the minority classes more e ectively in class-AUC score than all other methods. Additionally, compared to the other sampling methods, the results show when considering all the performance metric, SEDG and DGMs performs the best as other approaches that may excel in one metric often fail to replicate the same success in other metrics.
Here, we discuss design considerations made in Section 3 from what was demonstrated in these trials. We found partition-performance based sample selection demonstrates highest improvements in classi cation accuracy over other sample selection methods. However, based on the macroAUC and class AUC scores, there doesn't seem to be a clear sample selection method that outperforms the others, all performing comparatively targeting the minority classes than the traditional sampling methods.
We also remark that DGMs demonstrates very minimal improvements in predictive accuracy and macro-AUC score when using the NNModel classi er. We assume this to be related to the issue of over- tting, which we accommodated for by employing early stopping with basic heuristics: if the majority of training data ( 50%) share 80% of the features to the target sample, then we end training. We suggest for future works to handle this issue of over- tting in the context of SDG more e ciently.
We further con rm that weighted feature selection is more e ective than random feature selection in all performance metrics. The feature imbalance approach performs better in all metrics, with permutation importance and drop-column importance, the two feature importance methods, performing similarly in predictive accuracy. We nd that for both accuracy and macro-AUC score, using the embedding matrix from the NNModel outperform the other methods, with VAE being a close second in terms of macro-AUC score. For class AUC scores, we nd that VAEs and embedding matrix from NNModel improve the scores of the minority classes the most.
In this section, we investigate the synthetic samples more in-depth for each class to improve our understanding for student performance by showing how likely certain features are allowed to change without much signi cant changes to the data distribution, similar to feature importance. However, with synthetic samples, we can also see how we can change these features, thus seeing possible feature value candidates. In Figure 8, we use a VAE as DGM to create 100 synthetic samples for each class to show how likely each feature in each class is subject to change, and what feature values is reasonable. For example, we can see for low performing students, the likelihood of changing absences is low, with greater number of absences being a possible candidates. We also see that some features, such as travel time, school supplies, and famrel, are highly susceptible to change, inferring to their lack of distinction between possible candidates. Thus, we can leverage this method for feature understanding at a class level. Its usage can inform educators further on di erent performing students.
In this paper, we proposed a general framework for embeddingbased SDG and investigated DGMs for academic performance tasks. We tested the SEDG approach and DGMs against standard re-sampling methods, and found greater improvements in our proposed approaches in all performance metrics when we use our NNModel as the classi er and a more comprehensive improvement when we use traditional machine learning classi ers. We also introduced a technique for greater interpretibilty and insight into the dataset using a DGM by looking at the synthetic samples generated and seeing how likely each feature is modi ed and to what values it can take on for each class.
We acknowledge NSF #1757064 Grant for support this work.