Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks (Communications in Computational Physics, 2020)
Deep networks, especially convolutional neural networks (CNNs), havebeen successfully applied in various areas of machine learning as well as to challengingproblems in other scientific and engineering fields. This paper introduces Butterfly-net,a low-complexity CNN with structured and sparse cross-channel connections, togetherwith a Butterfly initialization strategy for a family of networks. Theoretical analysis ofthe approximation power of Butterfly-net to the Fourier representation of input datashows that the error decays exponentially as the depth increases. Combining Butterfly-net with a fully connected neural network, a large class of problems are proved tobe well approximated with network complexity depending on the effective frequencybandwidth instead of the input dimension. Regular CNN is covered as a special casein our analysis. Numerical experiments validate the analytical results on the approximation of Fourier kernels and energy functionals of Poisson's equations. Moreover,all experiments support that training from Butterfly initialization outperforms trainingfrom random initialization. Also, adding the remaining cross-channel connections,although significantly increases the parameter number, does not much improve thepost-training accuracy and is more sensitive to data distribution.