Playable Environments: Video Manipulation in Space and Time - Télécom Paris Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Playable Environments: Video Manipulation in Space and Time

Willi Menapace
  • Fonction : Auteur
Aliaksandr Siarohin
  • Fonction : Auteur
Christian Theobalt
  • Fonction : Auteur
Sergey Tulyakov
  • Fonction : Auteur
Vladislav Golyanik
  • Fonction : Auteur
Elisa Ricci
  • Fonction : Auteur

Résumé

We present Playable Environments - a new representation for interactive video generation and manipulation in space and time. With a single image at inference time, our novel framework allows the user to move objects in 3D while generating a video by providing a sequence of desired actions. The actions are learnt in an unsupervised manner. The camera can be controlled to get the desired viewpoint. Our method builds an environment state for each frame, which can be manipulated by our proposed action module and decoded back to the image space with volumetric rendering. To support diverse appearances of objects, we extend neural radiance fields with style-based modulation. Our method trains on a collection of various monocular videos requiring only the estimated camera parameters and 2D object locations. To set a challenging benchmark, we introduce two large scale video datasets with significant camera movements. As evidenced by our experiments, playable environments enable several creative applications not attainable by prior video synthesis works, including playable 3D video generation, stylization and manipulation. Further details, code and examples are available at https://willi-menapace.github.io/playable-environments-website

Dates et versions

hal-04204984 , version 1 (12-09-2023)

Identifiants

Citer

Willi Menapace, Stéphane Lathuilière, Aliaksandr Siarohin, Christian Theobalt, Sergey Tulyakov, et al.. Playable Environments: Video Manipulation in Space and Time. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, New orleans, USA, United States. pp.116366. ⟨hal-04204984⟩
8 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More