Sketch In, Sketch Out: Accelerating both Learning and Inference for Structured Prediction with Kernels - Equipe Signal, Statistique et Apprentissage Access content directly
Journal Articles Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (AISTATS) Year : 2024

Sketch In, Sketch Out: Accelerating both Learning and Inference for Structured Prediction with Kernels

Abstract

Surrogate kernel-based methods offer a flexible solution to structured output prediction by leveraging the kernel trick in both input and output spaces. In contrast to energy-based models, they avoid to pay the cost of inference during training, while enjoying statistical guarantees. However, without approximation, these approaches are condemned to be used only on a limited amount of training data. In this paper, we propose to equip surrogate kernel methods with approximations based on sketching, seen as low rank projections of feature maps both on input and output feature maps. We showcase the approach on Input Output Kernel ridge Regression (or Kernel Dependency Estimation) and provide excess risk bounds that can be in turn directly plugged on the final predictive model. An analysis of the complexity in time and memory show that sketching the input kernel mostly reduces training time while sketching the output kernel allows to reduce the inference time. Furthermore, we show that Gaussian and sub-Gaussian sketches are admissible sketches in the sense that they induce projection operators ensuring a small excess risk. Experiments on different tasks consolidate our findings.
Fichier principal
Vignette du fichier
sisokr_paper.pdf (816.18 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04001898 , version 1 (23-02-2023)
hal-04001898 , version 2 (06-05-2024)

Identifiers

Cite

Tamim El Ahmad, Luc Brogat-Motte, Pierre Laforgue, Florence d'Alché-Buc. Sketch In, Sketch Out: Accelerating both Learning and Inference for Structured Prediction with Kernels. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics (AISTATS), 2024. ⟨hal-04001898v2⟩
104 View
56 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More