In an article for Virtual Producer, as well as providing an introduction to generative AI and the possibilities it provides to content creators, Noah Kadner highlights a number of tools that are going to set the world of filmmaking alight. Here are some that you might not yet be familiar with.
Cuebric by Seyhan Lee “takes AI-generated art and slices it into ready-to-use Unreal Engine environments for virtual production”.
A product of its Toronto AI lab, NVIDIA Get3D “generates 3D meshes with complex topology, rich geometric details, and high-fidelity textures”.
Weta Simulcam “learns the basic geometry of the set from this lidar data and can then predict that geometry for compositing”.
“We infer an object disparity in one camera, turn it into depth, and then project that back into space and observe it with the hero camera. The computer-vision camera is very fast at acquiring the image, so we’re a frame ahead and ready to composite.”
Dejan Momcilovic, motion-capture supervisor, Weta FX
Respeecher, “a voice cloning/machine learning technology was used for Darth Vader’s voice in the Obi-Wan series for Disney+ and for younger Luke Skywalker’s voice in The Book of Boba Fett, among many others”.
Vall-E, “can closely simulate a person’s voice when given a three-second audio sample. Once it learns a specific voice, VALL-E can synthesize audio of that person saying anything—and do it in a way that attempts to preserve the speaker’s emotional tone”.
OpenAI’s Jukebox “generates music, including rudimentary singing, as raw audio in various genres and artistic styles”.
Nuke CopyCat “copies sequence-specific effects, such as garbage matting, beauty repairs, or deblurring, from a small number of frames in a sequence and then trains a network to replicate this effect on the entire sequence- saving a lot of manual effort and time”.
Electric Sheep’s ML Rotoscoper is “the next step of efficiency for rotoscoping … using object detection and image matting techniques through machine learning”.
If you are interested in using generative AI and other forms of machine learning in your creative processes, Mondatum provides a range of services around the development of artificial neural networks (ANNs), machine learning model training and the use of AI to solve problems and streamline processes and workflows. For more information, get in touch.
Source: Virtual Producer