Linear Constrained Rayleigh Quotient Optimization: Theory and Algorithms (CSIAM Transactions on Applied Mathematics, 2021)
We consider the following constrained Rayleigh quotient optimization problem (CRQopt):where $A$ is an $n×n$ real symmetric matrix and $C$ is an $n×m$ real matrix. Usually, $m$ « $n$.The problem is also known as the constrained eigenvalue problem in literature since itbecomes an eigenvalue problem if the linear constraint $C$T$v=b$ is removed. We start bytransforming CRQopt into an equivalent optimization problem (LGopt) of minimizing the Lagrangian multiplier of CRQopt, and then into another equivalent problem(QEPmin) of finding the smallest eigenvalue of a quadratic eigenvalue problem. Although these equivalences have been discussed in literature, it appears to be the firsttime that they are rigorously justified in this paper. In the second part, we presentnumerical algorithms for solving LGopt and QEPmin based on Krylov subspace projection. The basic idea is to first project LGopt and QEPmin onto Krylov subspaces toyield problems of the same types but of much smaller sizes, and then solve the reducedproblems by direct methods, which is either a secular equation solver (in the case ofLGopt) or an eigensolver (in the case of QEPmin). We provide convergence analysis for the proposed algorithms and present error bounds. The sharpness of the errorbounds is demonstrated by examples, although in applications the algorithms oftenconverge much faster than the bounds suggest. Finally, we apply the new algorithmsto semi-supervised learning in the context of constrained clustering.