Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning

Riemannian submanifold optimization with momentum is computationally challenging because ensuring iterates remain on the submanifold often requires solving or approximating difficult differential equations. We simplify such optimization algorithms for a class of structured symmetric positive-definite matrices with the affine invariant metric. We propose a generalized version of the Riemannian normal coordinates which preserves the metric and dynamically trivializes the problem into a Euclidean unconstrained problem. We use our approach to explain and simplify existing approaches for structured covariances and develop efficient second-order optimizers for training large-scale NNs without matrix inverses.