Improving the Reliability of Bank Customer Churn Prediction via Calibration and Uncertainty Quantification
DOI:
https://doi.org/10.52171/herald.373Keywords:
Banking, Machine Learning, Deep Learning, Random Forest, XGBoost, Monte Carlo Dropout, CalibrationAbstract
This study looks into how machine learning models for customer churn prediction in the banking industry use uncertainty quantification and calibration analysis. With an emphasis on predictive accuracy, probabilistic calibration, and profitability, the study contrasts three sophisticated models: Random Forest (Deep Ensemble), XGBoost, and a Neural Network with Monte Carlo Dropout. The assessment took into account both traditional and reliability-focused criteria, such as AUC, F1-score, Brier score, projected Calibration Error (ECE), and projected profit. The Random Forest model produced the most profit (118,000 AZN) with an AUC of 0.842 and an F1-score of 0.777. XGBoost displayed a moderate calibration variation (ECE = 0.088) but somewhat increased the F1-score to 0.794. On the other hand, while having a lower F1-score (0.631), the NN + MCDO model achieved the greatest AUC value (0.863) and the best calibration consistency (ECE = 0.026). The results indicate that uncertainty-aware deep learning improves probabilistic reliability and interpretability, whereas ensemble-based models are more effective for profit optimization in banking decision systems.
References
1. Tran H.D., Le N., Nguyen V.H. Customer churn prediction in the banking sector using machine learning-based classification models. Interdisciplinary Journal of Information, Knowledge & Management, 18, 87–105, 2023. https://doi.org/10.28945/5086
2. Ashraf R. Bank customer churn prediction using machine learning framework. Journal of Applied Finance & Banking, 14(4), 1–5, 2024.
3. Badalova A.N., Guliyeva S.H. Application of Machine Learning Methods for Classification of Agricultural Crops. Herald of the Azerbaijan Engineering Academy, 14(2), 106–116, 2022. https://doi.org/10.52171/2076-0515_2022_14_02_106_116
4. Imani M., Joudaki M., Beikmohammadi A., Arabnia H.R. Customer churn prediction: A systematic review of recent advances, trends, and challenges in machine learning and deep learning. Machine Learning and Knowledge Extraction, 7(3), 105, 2025.
https://doi.org/10.3390/make7030105
5. Xu X., Kou G., Ergu D. Profit-based uncertainty estimation with application to credit scoring. European Journal of Operational Research, 325(2), 303–316, 2025. https://doi.org/10.1016/j.ejor.2025.03.007
6. Li Y., Yan K. Prediction of bank credit customers churn based on machine learning and interpretability analysis. Data Science in Finance and Economics, 5(1), 19–34, 2025.
https://doi.org/10.3934/DSFE.2025002
7. Peng K., Peng Y., Li W. Research on customer churn prediction and model interpretability analysis. PLoS ONE, 18(12), e0289724, 2023. https://doi.org/10.1371/journal.pone.0289724
8. Breiman, L. Random forests. Machine Learning, 45(1), 5–32, 2001. https://doi.org/10.1023/A:1010933404324
9. Chen, T., & Guestrin, C. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794, 2016. https://doi.org/10.1145/2939672.2939785
10. Gal, Y., & Ghahramani, Z. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML), 1050–1059, 2016
11. Fawcett, T. An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874, 2006. DOI: https://doi.org/10.1016/j.patrec.2005.10.010
12. Powers, D.M.W. Evaluation: From precision, recall and F-measure to ROC. Journal of Machine Learning Technologies, 2(1), 37–63, 2011
13. Brier, G.W. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3, 1950
14. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K.Q. On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning (ICML), 1321–1330, 2017
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Musa Rahimov
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
