Playable Video Generation - Télécom Paris Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Playable Video Generation

Willi Menapace
  • Fonction : Auteur
Sergey Tulyakov
  • Fonction : Auteur
Aliaksandr Siarohin
  • Fonction : Auteur
Elisa Ricci
  • Fonction : Auteur

Résumé

This paper introduces the unsupervised learning problem of playable video generation (PVG). In PVG, we aim at allowing a user to control the generated video by selecting a discrete action at every time step as when playing a video game. The difficulty of the task lies both in learning semantically consistent actions and in generating realistic videos conditioned on the user input. We propose a novel framework for PVG that is trained in a self-supervised manner on a large dataset of unlabelled videos. We employ an encoder-decoder architecture where the predicted action labels act as bottleneck. The network is constrained to learn a rich action space using, as main driving loss, a reconstruction loss on the generated video. We demonstrate the effectiveness of the proposed approach on several datasets with wide environment variety. Further details, code and examples are available on our project page willi-menapace.github.io/playable-video-generation-website.

Dates et versions

hal-04204969 , version 1 (12-09-2023)

Identifiants

Citer

Willi Menapace, Stéphane Lathuilière, Sergey Tulyakov, Aliaksandr Siarohin, Elisa Ricci. Playable Video Generation. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, Nashville (Tenessee), United States. ⟨hal-04204969⟩
9 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More