Ensemble of Regressor Chains (ERC)

This review is a complimentary review of the previous article.

Regressor Chains or RC is an ensemble model built over a chain of classifiers [1]. To train this model, we have to select a random order of target variables and build a separate model for each of them. In the ERC [2], we have the RC with this significant difference that each model trains over a transformed data set, which means that we augmented the input with the output variables.

You may also be interested in the implementation of the ERC
To see the related code, please refer, to here.

ERC model training and prediction

We can conclude that the main idea here is to generate a meta input for the ML regressor model and for this matter they used the output as an input. This only works for training as we already have the real value, but for the prediction, the ERC must rely on the estimated values. But as an assumption in ML, the input and output variables should be independent. To deal with this issue, they proposed an out-of-sample of the targets during training. Therefore, they implemented the k-folds cross-validation for the estimated values and used them in the training part.

Training

• Define the input dataset.
• Transform the dataset.
• Build m models for m regression outputs.
• Apply new k-fold cross-validation on the augmented input data.

j = 0
X_train = x_train
while j < y_train.shape[1]:
i += 1
X_train = input(X_train, y_train, j)
if j+1 < y_train.shape[1]:
Y_train = y_train[:, j+1]
else:
break
cv_results = cross_val_score(estimator=model, X=X_train, y=Y_train,
cv=10, scoring='r2')
scores.iloc[:, i] = np.mean(cv_results)
mapping = {scores.columns[i]: 'D\'_target_' + str(j+1)}
scores = scores.rename(columns=mapping)
j += 1

Note that the base leaner could be any single regression model.

References

• [1] Read, J., Pfahringer, B., Holmes, G., & Frank, E. (2011). Classifier chains for multi-label classification. Machine Learning, 85(3), 333–359.
• [2] Spyromitros-Xioufis, E., Tsoumakas, G., Groves, W. et al. Multi-target regression via input space expansion: treating targets as inputs. Mach Learn 104, 55–98 (2016). https://doi.org/10.1007/s10994-016-5546-z.