Part X · Other kinds of models

Ch 40. Painting from noise

Not one piece at a time

GPT-2 writes text one token at a time (Ch 11). Many image generators work differently: they start with the whole picture as random noise and clean it up a little at a time, everywhere at once. This is a diffusion model.

GPT-2 writes text one token at a time, left to right (Ch 11). A picture could be made the same way, one patch after another, but many image generators work differently: they start with the whole picture as random noise and clean it up a little at a time, everywhere at once. This is a diffusion model.

Training works backwards. Take a real picture and add noise, bit by bit, until nothing of it is left. A network learns to look at a noisy picture and guess the noise that was added.^[1] To make something new, start from pure noise and keep removing the noise it predicts.^[2]

Training works backwards from the result. Take a real picture and add a little random noise, then a little more, step by step, until nothing of the picture is left. A network is trained to look at a noisy picture, told how many steps of noise it has had, and guess the noise that was added.^[1][1] To make something new, start from pure noise and repeatedly subtract the noise the network predicts. Each step leaves something a little more like the training pictures.^[2][2]

The noise, written down

The toy adds noise in 50 steps, each a little bigger than the last: the share $β_{t}$ grows from 0.001 to 0.2. You don’t have to add it step by step. With ${\overline{α}}_{t}$ the product of all the $(1 - β)$ s up to step $t$ , the point after $t$ steps is

x_{t} = \sqrt{{\overline{α}}_{t}} x_{0} + \sqrt{1 - {\overline{α}}_{t}} ε

where $x_{0}$ is the clean point and $ε$ is fresh random noise.^[2] After all 50 steps $\overline{α}$ is about 0.0045, so less than 7% of the original point is left. Training picks a random step, makes $x_{t}$ this way, and scores the network by the squared error between its guess and $ε$ .

Try it with dots

Here, each “picture” is a single dot, and the training pictures are dots scattered along a shape. Train the tiny network to predict the noise, then paint 400 new dots, each starting from pure noise:

A real image has millions of numbers. Here, each “picture” is a single dot with just two numbers, its position, and the training pictures are dots scattered along a shape. Train the tiny network (two layers of neurons like the ones in Ch 6) to predict the noise, then paint 400 new dots, each starting from pure noise:

Each dot is drawn independently, from its own random start, yet together they land on the shape. The network never stored the training dots; it learned which way to nudge a noisy dot to make it more like them. Before training, the nudges are random, and painting gives a smudge.

One step back

To paint, each step takes the network’s guess $\hat{ε}$ and removes the share of it that this step would have added, then adds a little fresh noise $σ_{t} z$ , none at the very last step:

x_{t - 1} = \frac{1}{\sqrt{1 - β_{t}}} (x_{t} - \frac{β_{t}}{\sqrt{1 - {\overline{α}}_{t}}} \hat{ε}) + σ_{t} z

This is the sampling rule from the paper the toy follows.^[2] Run 50 times, from step 50 down to 1, it turns a cloud of random dots into the shape.

From dots to pictures

Real image generators do the same with far bigger networks and every pixel’s colors. Many work on a compressed version of the image and steer each step with your text description.^[3]

Real image generators do the same thing with far bigger networks and with every pixel’s colors instead of two numbers. To save work, latent diffusion models remove the noise from a compressed version of the image, and they can steer each step with a text description, so the noise becomes the picture you asked for.^[3][3] The original recipe took 1,000 small steps to make one picture;^[2][2] the toy above uses 50.

Recipe card

Recap

A diffusion model learns to predict the noise that was added to a training example.
To make something new, it starts from pure noise and removes a little predicted noise at each step.
Unlike GPT-2’s one-token-at-a-time writing, it refines the whole picture at every step.

$x_{t} = \sqrt{{\overline{α}}_{t}} x_{0} + \sqrt{1 - {\overline{α}}_{t}} ε$ gives any step’s noise in one go; painting removes it one step at a time.

Next question: an agent picks actions by writing text. Could a model predict the actions directly?

Sources

Jascha Sohl-Dickstein et al. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML 2015.
Jonathan Ho et al. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020.
Robin Rombach et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022.

Sources

Jascha Sohl-Dickstein et al. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML 2015.
Jonathan Ho et al. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020.
Robin Rombach et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022.

Sources

Jascha Sohl-Dickstein et al. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML 2015.
Jonathan Ho et al. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020.
Robin Rombach et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022.