Overview

In machine learning, imbalanced datasets are no surprise. If the datasets intended for classification problems or other problems related to discrete predictive analytics have an unequal number of samples for different classes, then those datasets are said to be imbalanced.

Classes having comparatively fewer instances than others are said to be a minority with respect to the classes having a comparatively larger number of the samples (majority). Training ML models with imbalanced datasets often causes the models to develop a certain bias towards the majority classes.

The SMOTE algorithm, short for Synthetic Minority Over-sampling Technique, addresses this issue imbalanced classes. It is based on nearest neighbors (determined by the Euclidean distance of data points in the feature space).

The feature values of nearest neighbor samples are used to interpolate synthetic feature values to retrieve a certain predefined percentage of additional synthetic samples (over-sampling).cApplying this algorithm to the samples of the training datasets that belong to the minority classes finally ends up with a more balanced training set and the removal of the model bias.

Apache Spark ML does not support SMOTE out of the box.

The SMOTE algorithm is implemented as an Apache Spark ML Transformer and also made available as a plugin component for CDAP data pipelines.

SMOTESampler

@Plugin(type = SparkCompute.PLUGIN_TYPE)
@Name("SMOTESampler")
@Description("A preparation stage for either building Apache Spark based classification or "
+ "regression models. This stage leverages the SMOTE algorithm to extends a training "
+ "dataset containing features & labels with synthetic data records.")
public class SMOTESampler extends SMOTECompute {

    ...

}

Parameters