Originally developed to generate stills, a new Firefly Video Model has been announced. It can add audio or video frames from an existing video clip and generate new video and 3D animation clips at 1080p resolution and up to five seconds in length using reference images or text prompts of up to 1,000 words
Adobe say it “excels at generating videos of the natural world … like landscapes, plants or animals”. and suggests it could be ideal for creating fire, smoke and dust that could then be composited into visual effects or motion graphics projects. Amongst the video clips in this Adobe blog post there are examples of generated footage resembling 3D, hand-drawn and stop-motion animation.
You can get the tools here as well as in the Adobe Creative Cloud. It’s in public Beta on the Firefly web app and in Beta builds of Adobe’s Premiere Pro video editor., as Generative Extend.
Adobe says Firefly is “commercially safe and is only trained on content we have permission to use”.
As with other, similar models, it struggles to maintain temporal and stylistic consistency. Its limitations mean it can only be (reliably) used to supplement conventionally shot material, as AI-generated stock footage and B-roll and to paper over cracks, a missing establishing shot, for example.
Adobe Firefly features
Generative Extend – can extend existing video clips, generating new frames at the start or end, and generate extra audio ‘room tone’.
Image to Video – can generate clips based on reference stills and even from concept illustrations.
Text to Video – can generate video from text prompts and optionally specify camera properties like camera position, zoom extent and common camera motion types.
Using built-in controls, you can select the aspect ratio of the video generated (currently only 16:9 or 9:16), and choose from a range of shot size presets, ranging from extreme close up to extreme long shots, camera angle, including aerial shots, and camera moves, including handheld.
Payment for using the Firefly Video Model is credit-based. Generating video initially uses 20 credits per second of footage, which will rise to 100 credits per second. A $9.99 / month Firefly Standard plan gives you 2,000 video credits a month. A $29.99 / month plan ups that to 7,000 credits a month. Credits are also included as part of Creative Cloud subscriptions.
The Firefly Video Model is available now available via the Firefly web app. Text to Video and Image to Video are both in Beta.
Source: cgchannnel.com