Ensemble learning has become a foundational paradigm in modern machine learning due to its strong association with improved predictive stability, generalization, and robustness in complex data environments. By combining multiple base learners, ensemble methods address fundamental limitations of single-model approaches, particularly in the presence of noise, high dimensionality, and model uncertainty. This study presents a comprehensive and integrative analysis of prominent ensemble learning strategies, including bagging, boosting, random forests, weighted voting mechanisms, stacked generalization, and efficiency-driven ensemble construction using data envelopment analysis (DEA). Drawing upon established theoretical frameworks and empirical findings, the article synthesizes classical ensemble methods with advanced weighting and optimization techniques to clarify their comparative strengths, limitations, and practical applicability. Emphasis is placed on how margin theory, variance reduction, classifier diversity, and efficiency optimization jointly contribute to ensemble effectiveness. Furthermore, the study examines the role of tunable parameters, hyperparameter optimization, and weighted aggregation in enhancing ensemble performance across classification and regression tasks. By systematically reviewing methodological developments and identifying unresolved challenges, this work provides a unified perspective that supports informed ensemble design in applied machine learning research. The analysis contributes to the literature by bridging statistical learning theory with efficiency-based optimization frameworks, highlighting opportunities for future research in adaptive, interpretable, and resource-aware ensemble systems.

Ensemble learning, bagging, boosting, random forests, weighted voting, data envelopment analysis, stacked generalization

1. Yoav Freund and Robert E. Shapire. Experiments with a new boosting algorithm. In Lorenza Saitta, editor, Machine Learning: Proceedings of the Thirteenth International Conference (ICML’96). Morgan Kaufmann, 1996.

2. Leo Breiman. Random forests. Machine Learning Journal, 45:5–32, 2001.

3. Breiman, L. Bagging predictors. Machine Learning, 24(2), 123–140, 1996.

4. David Meyer, Friedrich Leisch, and Kurt Hornik. The support vector machine under test. Neurocomputing, 55:169–186, 2003.

5. Robert E. Schapire, Yoav Freund, Peter Bartlett, and Wee Sun Lee. Boosting the margin: a new explanation for the effectiveness of voting methods. In Machine Learning: Proceedings of the Fourteenth International Conference (ICML’97). Morgan Kaufmann, 1997.

6. Breiman, L. Random forests. Machine Learning, 45(1), 5–32, 2001.

7. Li, H. B., Wang, W., Ding, H. W., & Dong, J. Trees Weighting Random Forest Method for Classifying High-Dimensional Noisy Data. IEEE Conference on E-Business Engineering, 2010.

8. Pham, H., & Olafsson, S. Bagged ensembles with tunable parameters. Computational Intelligence, 35(1), 184–203, 2019.

9. Robnik-Šikonja, M. Improving Random Forests. ECML 2004.

10. Kuncheva, L. I., & Rodríguez, J. J. A weighted voting framework for classifier ensembles. Knowledge and Information Systems, 38(2), 259–275, 2014.

11. Shahhosseini, M., Hu, G., & Pham, H. Optimizing Ensemble Weights and Hyperparameters of Machine Learning Models for Regression Problems. arXiv:1908.05287, 2019.

12. James, G., Witten, D., Hastie, T., & Tibshirani, R. An Introduction to Statistical Learning. Springer, 2013.

13. Wolpert, D. H. Stacked generalization. Neural Networks, 5(2), 241–259, 1992.

14. Dua, D., & Graff, C. UCI Machine Learning Repository. University of California, Irvine, 2019.

15. Jones, E., Oliphant, T., & Peterson, P. SciPy: Open source scientific tools for Python, 2001.

16. Peykani, P., Mohammadi, E., Saen, R. F., Sadjadi, S. J., & Rostamy-Malkhalifeh, M. Data envelopment analysis and robust optimization: A review. Expert Systems, 2020.

17. Donate, J. P., Cortez, P., Sánchez, G. G., & De Miguel, A. S. Time series forecasting using a weighted cross-validation evolutionary artificial neural network ensemble. Neurocomputing, 109, 27–32, 2013.

18. Zheng, Z., & Padmanabhan, B. Constructing ensembles from data envelopment analysis. INFORMS Journal on Computing, 19(4), 486–496, 2007.

19. Peykani, P., & Mohammadi, E. Window network data envelopment analysis. International Journal of Industrial Mathematics, 12(1), 89–99, 2020.

20. Hong, H. K., Ha, S. H., Shin, C. K., Park, S. C., & Kim, S. H. Evaluating the efficiency of system integration projects using DEA and machine learning. Expert Systems with Applications, 16(3), 283–296, 1999.

21. Jerrold H. Zar. Biostatistical Analysis. Prentice Hall, 1998.