Ch 45. Picking a model

A few parts come up often:

• The size is the number of weights, in billions (Ch 12). It tells you roughly how much memory the model needs.
• E and A mean the model stores more weights than it uses for each token. E2B costs about as much per token as a 2.3-billion-weight model, and 26B-A4B uses under 4 billion of its weights per token (Ch 37).^[1]
• it means tuned to chat and use tools; without it, the model only continues text (Ch 15).
• assistant is a small drafter that speeds up a bigger model (Ch 14).
• Distill names a small model trained on a big model’s examples; the size is the small one’s.^[2]
• qat, q4_0, and w4a16 are about how many bits each weight uses (Ch 13).^[3]
• diffusiongemma writes by cleaning up noise, like the picture models in Ch 40.^[4]

A few parts are worth a closer look:

• The size is a count of weights, in billions (Ch 12). It tells you roughly how much memory the model needs and, for an ordinary model, how much work each token takes.
• E and A change what the size means. E2B is an effective size: Gemma 4 E2B stores 5.1 billion numbers, but most of the extra are per-layer tables that are only looked up, like the token table in Ch 5, so each token costs about as much as a 2.3-billion-weight model.^[1][1] 26B-A4B is a mixture of experts (Ch 37) with 25.2 billion weights, of which about 3.8 billion (8 of 128 experts, plus one shared) run for each token. It runs almost as fast as a 4-billion-weight model, but all 25.2 billion have to fit in memory.^[1][1]
• it means instruction-tuned, for chat and tools; a name without it is a base model that only continues text (Ch 15). Base models are a common starting point for training your own version.
• assistant is a drafter for speculative decoding (Ch 14), not a model to use on its own.
• Distill names a teacher and a student. DeepSeek-R1-Distill-Qwen-1.5B is a 1.5-billion-weight Qwen model tuned on examples made with DeepSeek-R1, a far bigger reasoning model (Ch 17). The size is the student’s, not the teacher’s.^[2][2]
• qat, q4_0, and w4a16 are about bits per weight (Ch 13). Gemma 4 comes in four QAT versions: unquantized 16-bit QAT weights for people who round them with their own tools, Q4_0 in a GGUF file for llama.cpp, a mobile version with some 2-bit layers, and 4-bit weights with 16-bit activations for the vLLM serving software.^[3][3][4][4] transformers, ct, and gguf only say how the file is packaged.
• diffusiongemma is built on the 26B A4B model but writes by denoising a block of 256 tokens at a time instead of one token per pass, a cousin of the picture models in Ch 40.^[5][5]

Some things are never in the name, like how much text the model can read at once, whether it can see pictures or hear audio, and its license. Those are on the model card.^[1]

Some things are never in the name. The model card lists how much text the model can read at once (128K tokens for E2B and E4B, 256K for the larger models), what it can take in (all of them read images; only E2B, E4B, and 12B hear audio), whether it has a thinking mode (Ch 17), and its license.^[1][1] Even the size shown on a model’s page can disagree with its name: it counts the numbers stored in the files, so the page for gemma-4-E2B shows 5B.

A full bill, worked out

Take Qwen2.5-1.5B-Instruct, the model in the agent chapters. It has 1.54 billion weights, stored in 16 bits: 1.54 × 2 = about 3.1 GB.^[6] Its KV cache holds 14,336 numbers per token (Ch 26), also 2 bytes each, so a 4,000-token chat adds about 0.11 GB and a full 32,768-token context about 0.94 GB. Rounded to 4 bits, the weights shrink to about 0.8 GB, but the cache doesn’t unless it is rounded too, so with a full context it would be the bigger part of the bill.