Introducing a Multi-target regression

this review, I will review a well-known paper called Multi-target regression via input space expansion: treating targets as inputs [1].

Note that I am introducing Grigorios Tsoumakas, et al work, and they wrote the paper. This review is simply an introduction and summary of their original work.

I will try to explain this paper simply with related examples.

Introduction to Multi-target prediction

MTR or Multi-output regression or Multivariate regression refers to the models that try to predict multivariate outputs with the related input data.

MTR and MLC predict multiple variables using the dependent variables

The focus of this study is implementing the MLC models on regression problems.

Main idea

As with most machine learning models, by having the dependant and response variables, the goal here is to construct a function that maps the input into the output, where the output in this case would be a set of vectors.

Two different approaches in training the models

The single Target or ST[2, 3] considers m single model to predict the vector of the regression output. Moreover, the Stacked Single Target or SST was introduced. SST has two stages of prediction. In the first stage, it predicts m vectors with m models. And in the latter stage, a set of m’ meta-models are learned for each target by transforming the training set D’. In the transformed training set, it uses estimated values for the output space.

SST model with two training stages

Another introduced model is RC or Regressor Chains which is based on its classification idea or MLC[4]. The idea is to build separate models on a random permutation of targets.

RC model training

Conclusion

SST and ERC are two new Multivariate methods by extending two well-known multi-label classification methods.

Both methods are treating other estimated values as further input variables.

References

• [1] Spyromitros-Xioufis, E., Tsoumakas, G., Groves, W. et al. Multi-target regression via input space expansion: treating targets as inputs. Mach Learn 104, 55–98 (2016). https://doi.org/10.1007/s10994-016-5546-z.
• [2] Luaces, O., Díez, J., Barranquero, J., del Coz, J. J., & Bahamonde, A. (2012). Binary relevance efficacy for multilabel classification. Progress in Artificial Intelligence, 1(4), 303–313.
• [3] Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5(2), 241–259.
• [4] Read, J., Pfahringer, B., Holmes, G., & Frank, E. (2011). Classifier chains for multi-label classification. Machine Learning, 85(3), 333–359.