Understanding the Initial Condensation of Convolutional Neural Networks (CSIAM Transactions on Applied Mathematics, 2025)
Previous research has shown that fully-connected neural networks withsmall initialization and gradient-based training methods exhibit a phenomenon knownas condensation [T. Luo et al., J. Mach. Learn. Res., 22(1), 2021]. Condensation is a phenomenon wherein the weight vectors of neural networks concentrate on isolated orientations during the training process, and it is a feature in the non-linear learning processthat enables neural networks to possess better generalization abilities. However, theimpact of neural network architecture on this phenomenon remains a topic of inquiry.In this study, we turn our focus towards convolutional neural networks (CNNs) to investigate how their structural characteristics, in contrast to fully-connected networks,exert influence on the condensation phenomenon. We first demonstrate in theory thatunder gradient descent and the small initialization scheme, the convolutional kernelsof a two-layer CNN condense towards a specific direction determined by the trainingsamples within a given time period. Subsequently, we conduct systematic empiricalinvestigations to substantiate our theory. Moreover, our empirical study showcasesthe persistence of condensation under broader conditions than those imposed in ourtheory. These insights collectively contribute to advancing our comprehension of thenon-linear training dynamics inherent in CNNs.