But how do AI images and videos actually work? | Guest video by Welch Labs

Over the last few years, AI systems have become astonishingly good at turning text props into videos . At the core of how these models operate is a deep connection to physics . This generation of image and video models works using a process known as diffusion, which is remarkably equivalent to the Brownian motion we see as particles diffuse , but with time run backwards, and in high-dimensional space . As we'll see, this connection to physics is much more than a curiosity . We get real algorithms out of the physics that we can use to generate images and videos . And this perspective will also give us some really nice intuitions for how these models work in practice . But before we dive into this connection, let's get hands-on with a real diffusion model . While the best models are closed source, there are some compelling open source models . This video of an astronaut was generated by an open source model called WAN 2.1 . We can add to our prompt and have our astronaut hold a flag, hold a laptop, or hold a meeting . If we cut down our prompt to just an astronaut, we get this . And if we cut down our prompt to nothing, we interestingly still get this video of a woman . If we dig into our WAN model's source code, we'll find that the video generation process begins with this call to a random number generator . Creating a video where the pixel intensity values are chosen randomly . Here's what it looks like . From here, this pure noise video is passed into a transformer . This is the same type of AI model used by large language models, like ChatGPT . But instead of outputting text, this transformer outputs another video that now looks like this . Still mostly noise, but with some hints of structure . This new video is added to our pure noise video, and then passed back into the model again, producing a third video that looks like this . This process is repeated again and again . Here's what the video looks like after 5 iterations, 10, 20, 30, 40, and finally 50 . Step by step, our transformer shapes pure noise into incredibly realistic video . But what exactly is the connection to Brownian motion here ? And how is our model able to use text input so expressively to shape noise into what our prompt describes ? In this video, we'll impact diffusion models in 3 parts . First we'll look at a 2021 OpenAI paper and model called CLIP . As we'll see, CLIP is really two models, a language model and a vision model , that are trained using a clever learning objective that allows them to learn this really powerful shared space between words and pictures . Experimenting with this space will help us get a feel for the high dimensional spaces that diffusion models operate in . But learning a shared representation is not enough to generate images