Google has introduced Lumiere, an AI-based video generator described as a “spatio-temporal diffusion model for realistic video generation.”

This technology appears to be capable of creating videos depicting realistic and diverse movements, making it one of the most advanced text-to-animal AI video generators demonstrated yet. The presentation showed how from a text I could create images of different animals with different styles.

Lumiere stands out from other video generation models due to its unique architecture. Unlike existing models that synthesize distant keyframes followed by temporal super-resolution, Lumiere outputs the entire temporal duration of a video in one go. This approach allows for global temporal consistency, making the resulting videos fluid and coherent.

The spatio-temporal aspects of the video are handled simultaneously by Lumiere, allowing you to create videos from start to finish in one continuous process. This eliminates the need to stitch together small parts or frames, resulting in a more efficient and simplified video generation experience.

When available Lumiere offers the following features:

1. Text to video generation.
2. Image to video conversion.
3. Style-based video generation.
4. Text-based video editing.
5. Video inpainting capabilities.