Ch 36. Reading both ways

Every model so far has filled in a blank at the end of the text. Put the blank in the middle and GPT-2 has a problem: it only reads the words before the blank. In The cat sat on the ___ of the fridge, the clue comes after the gap.

Every model so far has filled in a blank at the end of the text. Put the blank in the middle and GPT-2 has a problem: it only reads the words before the blank. In The cat sat on the ___ of the fridge, the clue comes after the gap, where GPT-2 can’t look.

A different kind of transformer, the encoder, reads the whole text at once. The best known is BERT, trained by hiding some words and guessing them from the words on both sides.^[1] Here are real guesses from BERT and GPT-2:

A different kind of transformer, the encoder, reads the whole text at once. The best known is BERT, trained by hiding 15% of the tokens in its training text and guessing each one from the words on both sides.^[1][1] Here are real guesses from BERT and GPT-2:

The sentence about the dog shows the gap at its widest. In “The ___ wagged its tail and barked”, GPT-2 has only “The” to go on, and its top guess is a new line, at 1.4%. BERT reads the rest and says dog, at 61%. Notice too that BERT’s tokenizer split wagged into “wa” and “##gged”, where ”##” marks a piece that continues the word before it.

The difference is one setting in attention (Ch 9). In GPT-2, each token can only look at the tokens before it. In BERT, every token can look at every other token. Switch between the two:

The difference comes down to one setting in attention (Ch 9). In GPT-2, a decoder, each token can only look at itself and the tokens before it. In BERT, every token can look at every other token. Switch between the two:

The mask, written down

Attention scores every pair of tokens and turns each row of scores into weights with a softmax. The only difference between the two models is a mask $M$ added to the scores first:

For GPT-2, $M_{ij}$ is 0 when token $j$ comes no later than token $i$, and $-\infty$ otherwise, so later tokens get a weight of exactly 0. For BERT, every $M_{ij}$ is 0. Over the 10 tokens of the fridge sentence, GPT-2’s mask allows 55 of the 100 pairs and BERT’s all 100.

BERT pays for that in training. Only the 15% of tokens it hides give it something to predict, while GPT-2 learns from every token. And of the hidden tokens, BERT swapped 80% for a [MASK] token, 10% for a random token, and left 10% unchanged, so that it couldn’t count on seeing [MASK], which never appears in real text.^[1]

The price of looking both ways is that an encoder needs the whole text before it starts, so it can’t write one token at a time. Encoders are used for understanding instead: search, spam filters, and embeddings like the map in Ch 5.

The price of looking both ways is that an encoder needs the whole text before it starts. It can’t write a story one token at a time, because the tokens after the one it’s writing don’t exist yet. So encoders are used for understanding: search, sorting email into spam or not, and embeddings. The embedding model behind the map in Ch 5 is a small encoder of the same design as BERT.