Researchers at the University of Trento, the Institut Polytechnique de Paris, and Snap, Inc. have devised an AI system that uses a framework they call Clustering for Action Decomposition and DiscoverY (CADDY) to learn a set of discrete actions from a large library of video clips that can be used to create new “playable” videos in a gaming-like experience. Click here to view the team’s academic paper, and here for the project website.
Using CADDY and a real-life video of a tennis player, users can select Left, Right, Forward, Backward, Hit the ball, or Stay to prompt the system to create videos capturing that action.
“Our experiments show that we can learn a rich set of actions that offer the user a gaming-like experience to control the generated video. As future work, we plan to extend our method to multi-agent environments. CADDY automatically discovers the most significant actions to condition video generation and can produce playable video generation models in a variety of settings, from video games to real videos.”
Extract from the research paper
This technology could find itself being used inn the worlds of corporate video production and commercials, particularly for small to medium-sized business that don’t have the money to invest in expensive expertise. As well as opening up new creative possibilities, CADDY offer the poootenntial to eliminate the need for (expensive) re-shoots.
Source: VentureBeat