Machine learning is being used to drive wider adoption and professional development
Vid2Vid Cameo – which is available through a demo on the NVIDIA AI Playground – can be used to capture facial movements and expressions from video shot on any type of camera, including smartphones, and applied in real time to animate an avatar, character or painting. Pose Tracker 3D body-pose estimation software enables the capture of full-body movements like walking, dancing and performing martial arts.
Tools like these open facial animation and performance capture up to a much wider user base, including online meetings and other forms of remote B2B communication. With refinements involving customisation, professional animators can wield both of the above technologies in unison to transfer their own movements to virtual characters for use in a wide range of projects, including during concept design and previz, to convey look and movement for early versions of digital characters.
Vid2Vid Cameo needs just two elements to generate a talking-head video: a still image of the avatar or painting to be animated, plus footage of the original performer – speaking, singing, head rotation etc.. It uses a GAN (generative adversarial network) trained on 180,000 videos to identify 20 points on a human face. These points are extracted from the video stream of the performer and applied to the avatar or digital character.
The demo below transfers a performance of Edgar Allan Poe’s “Sonnet — to Science” to a portrait of the writer by artist Gary Kelley.
Visit this NV Labs page on Github to read the original academic paper ‘One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing’.
Pose Tracker, an Extension in the NVIDIA Omniverse 3D design collaboration and world simulation platform, is used for pose estimation and tracks whole body movement to capture complex motion. Download NVIDIA Omniverse for free and get started with step-by-step tutorials.
Source: NVIDIA blog