Dynamic Stability and Learning Rate Selection in Deep Neural Networks Using an Empirical Indicator

Abstract

Learning rate selection in deep neural networks is often based on empirical procedures that do not always provide a clear understanding of training dynamics or allow the search space to be reduced efficiently. This study analyzes such dynamics through an empirical stability indicator built from the temporal evolution of the loss function, used as a proxy for training sensitivity. The study combines dynamic analysis, predictive performance evaluation, and a comparison between traditional selection and a dynamic criterion. To this end, it examines the relationship among learning rate, relative stability, and performance in multilayer networks on MNIST and FashionMNIST, and then extends the analysis to architectures with varying depth, the CIFAR-10 dataset, different optimizers, a multi-dataset evaluation including tabular datasets, and a multi-seed validation. The results show that the learning rate systematically modifies the relative stability of training and that the best predictive performances tend to concentrate in intermediate regions of the indicator. In the multi-dataset experiments, the dynamic criterion reduced the search region by 30% to 60% while maintaining final performances comparable to those obtained through exhaustive exploration. Overall, the findings suggest that dynamic analysis can complement hyperparameter selection and provide a useful tool for studying learning stability in deep neural networks.