Ch 37. Many experts

GPT-2 small is dense: all 124 million of its weights work on every token, so a bigger dense model is slower for every token. A mixture of experts gets around this. It splits the MLP in each block into several smaller ones, the experts, and adds a small router that sends each token to only the best few.^[1]

GPT-2 small is dense: every one of its 124 million weights takes part in predicting every token. Making a dense model bigger makes every token more expensive to process. A mixture of experts gets around this. Take the MLP inside each block (Ch 10) and split it into several smaller MLPs, the experts. In front of them goes a small router: for each token, it scores every expert and sends the token only to the best few.^[1][1]

Each token’s result blends the outputs of its chosen experts. The other experts don’t run for that token at all.

Each token’s output is a blend of its chosen experts’ outputs, weighted by the router’s scores. The other experts don’t run at all for that token, so their weights cost nothing. Different tokens go to different experts, so over a whole text every expert gets used.

The router, written down

For a token’s vector $x$, the router multiplies by a weight matrix $W_g$ to get one score per expert, keeps the top $k$, and turns those into weights with a softmax. The output is the weighted sum of the chosen experts’ outputs:

That’s Mixtral’s layer, with $k=2$.^[2] The widget takes a softmax over all eight scores and then rescales the top $k$ to add up to 1, which gives the same weights.

Only the MLPs are split; attention and the embeddings are shared by every token. That’s why Mixtral uses 13 billion weights per token, not a quarter of 47. If each expert has $E$ weights and $S$ are shared, then $S+8E=47$ billion and $S+2E=13$ billion, so each expert holds about 5.7 billion and about 1.7 billion are shared.

Training needs a push to spread the work. A router that favors a few experts trains them more, which makes it favor them even more, so Shazeer and his colleagues added a loss term that penalizes uneven use.^[1] DeepSeek-V3 balances its experts without an extra loss term.^[3]

Many large open models work this way. Mixtral 8x7B has 47 billion weights but uses only 13 billion for each token.^[2] DeepSeek-V3 has 671 billion and uses 37 billion.^[3]

This is how many large open models are built. Mixtral 8x7B has 8 experts in each block and sends each token to 2 of them: it has 47 billion weights in total, but each token uses only 13 billion, about a quarter.^[2][2] DeepSeek-V3 goes further: 671 billion weights in total, 37 billion per token, about 6%.^[3][3]

You might imagine one expert for grammar and one for cooking, but trained experts don’t split the work that neatly. In Mixtral, they followed things like code syntax more than topics.^[2]

It is tempting to imagine one expert for grammar, one for science, and one for cooking, but trained experts don’t divide the work that neatly. When the Mixtral team checked which experts handled text on different topics, they found no obvious pattern. What showed up instead was syntax: the word self in Python code, or the spaces that indent a line, kept going to the same experts.^[2][2]