Ch 39. Seeing pictures

A transformer doesn’t care what its tokens stand for; it just needs a row of vectors (Ch 5). A picture can become one by being cut into square patches, each turned into a vector, just like a token. This is the vision transformer.^[1]

A transformer doesn’t care what its tokens stand for. All it needs is a row of vectors (Ch 5). Text becomes that row by splitting into tokens. A picture can do the same by being cut into square patches, each turned into a vector by multiplying its pixel values by learned weights, plus a position, just like a token. The rest is the familiar stack of blocks. This is the vision transformer.^[1][1]

A patch, in numbers

Each 32×32 patch has 3 colors per pixel: 32 × 32 × 3 = 3,072 numbers. One learned 3,072 × 768 weight matrix turns it into 768 numbers, the same matrix for every patch, and a position vector is added. One extra learned vector goes at the front, a class token, so the transformer reads 50 vectors, and at the end the class token’s vector stands for the whole picture.^[1] CLIP then maps that vector, and the text side’s summary of a caption, into a shared space of 512 numbers.

CLIP was trained on 400 million pictures with their captions from the internet, learning to put each picture close to its own caption.^[2] So words and pictures land in one shared space. Pick some words and see which pictures land closest:

CLIP was trained on 400 million pictures paired with their captions from the internet. It has two parts, one that reads pictures and one that reads text, and training pushed each picture’s vector close to its own caption’s vector and away from the others.^[2][2] The result is a single space where words and pictures can be compared. Pick some words and see which pictures land closest:

Comparing a caption with pictures

For each caption, CLIP takes the cosine similarity with each picture’s vector and multiplies by a learned temperature, 100 for this model, before the softmax. The similarities themselves sit close together: “a red circle” scores 0.271 with the red circle and 0.268 with the red square. Multiplied by 100, that 0.003 gap gives the circle 42% and the square 32%. In training, each batch held 32,768 picture–caption pairs: every picture had to pick out its own caption from all 32,768, and every caption its picture.^[2]

A multimodal chat model combines the two ideas: a vision model turns the picture into vectors, and the language model reads them alongside your words.^[3] To the language model, the picture is just more tokens.

A multimodal chat model puts the two ideas together. A vision model like CLIP’s turns the picture into patch vectors, a small layer converts those into the same kind of vectors as text tokens, and the language model reads them in its context alongside your words.^[3][3] To the language model, the picture is just more tokens.

Put a model that sees inside the loop from Ch 29 and you get a computer-use agent. The harness takes a screenshot and gives it to the model with the task. The model answers with an action, like a click at a spot on the screen or some text to type. The harness does it, takes a new screenshot, and goes round again, as in this made-up run:

screenshot 1 → click(412, 230)
screenshot 2 → type("cat rhymes")
screenshot 3 → key("Enter")
screenshot 4 → done

It’s hard. To click a button, the model has to work out where it is in pixels, which Anthropic had to train its model to do.^[4] On a test of 369 real computer tasks, people finished about 72% and the best model 12%.^[5] And everything on the screen reaches the model, so a web page can hide instructions in it, as in Ch 34.^[4]

Put a model that sees inside the loop from Ch 29 and you get a computer-use agent, which works any program the way a person does. The harness takes a screenshot and gives it to the model with the task. The model answers with a tool call for an action: move the pointer and click at a position, type text, press a key. The harness carries it out, and the next screenshot is the tool’s result. This run is made up:

screenshot 1 → click(412, 230)
screenshot 2 → type("cat rhymes")
screenshot 3 → key("Enter")
screenshot 4 → done

The hard part is pointing. To click a button, the model has to give its position in pixels, and Anthropic had to train its model to count pixels across the screen, a skill it compares to counting the letters in a word (Ch 16). Seeing the screen as a series of screenshots, like a flipbook, the model can also miss a message that appears and vanishes between them.^[4][4]

OSWorld tests this on 369 tasks in real desktop and web apps, each checked by a script that inspects the computer afterwards. People finished over 72% of them and the best model of early 2024 about 12%, failing mostly at finding the right place on the screen and at knowing how the apps work.^[5][5] Later that year Anthropic reported 14.9% for its first computer-use model.^[4][4] And everything on the screen reaches the model, so a web page can carry instructions just as the poisoned page in Ch 34 did; Anthropic names this prompt injection as a main risk of computer use.^[4][4]

What a screenshot costs

Anthropic’s models read a picture in 28 × 28-pixel patches, one token each, so a $w\times h$ screenshot costs $\lceil w/28\rceil \times \lceil h/28\rceil$ tokens.^[6] A 1280 × 800 screen is 46 × 29 = 1,334 tokens, against the 49 patches of CLIP’s 224-pixel pictures above. Every step adds one, and each call re-reads them all, as in Ch 29: after ten steps, a harness that keeps every screenshot sends 13,340 tokens of them with each call.