Convergence of Backpropagation with Momentum for Network Architectures with Skip Connections (Journal of Computational Mathematics, 2021)
We study a class of deep neural networks with architectures that form a directed acyclicgraph (DAG). For backpropagation defined by gradient descent with adaptive momentum,we show weights converge for a large class of nonlinear activation functions. The proofgeneralizes the results of Wu et al. (2008) who showed convergence for a feed-forwardnetwork with one hidden layer. For an example of the effectiveness of DAG architectures,we describe an example of compression through an AutoEncoder, and compare againstsequential feed-forward networks under several metrics.