Sketched Newton--Raphson - Télécom Paris
Article Dans Une Revue SIAM Journal on Optimization Année : 2022

Sketched Newton--Raphson

Résumé

We propose a new globally convergent stochastic second-order method. Our starting point is the development of a new sketched Newton-Raphson (SNR) method for solving large scale nonlinear equations of the form F (x) = 0 with F : R p → R m. We then show how to design several stochastic second-order optimization methods by rewriting the optimization problem of interest as a system of nonlinear equations and applying SNR. For instance, by applying SNR to find a stationary point of a generalized linear model, we derive completely new and scalable stochastic second-order methods. We show that the resulting method is very competitive as compared to state-of-the-art variance reduced methods. Furthermore, using a variable splitting trick, we also show that the stochastic Newton method (SNM) is a special case of SNR and use this connection to establish the first global convergence theory of SNM. We establish the global convergence of SNR by showing that it is a variant of the online stochastic gradient descent (SGD) method, and then leveraging proof techniques of SGD. As a special case, our theory also provides a new global convergence theory for the original Newton-Raphson method under strictly weaker assumptions as compared to the classic monotone convergence theory.
Fichier principal
Vignette du fichier
sketch_newton-raphson.pdf (721.13 Ko) Télécharger le fichier
Origine Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-04182653 , version 1 (17-08-2023)

Identifiants

Citer

Rui Yuan, Alessandro Lazaric, Robert M Gower. Sketched Newton--Raphson. SIAM Journal on Optimization, 2022, 32 (3), pp.1555 - 1583. ⟨10.1137/21m139788x⟩. ⟨hal-04182653⟩
21 Consultations
34 Téléchargements

Altmetric

Partager

More