HuggingFace's greedy decoding isn't always greedy

I’m building an LLM inference engine from scratch: continuous batching and a paged KV cache. The exit criterion for the model layer is a golden test: my engine and stock HuggingFace, greedy decoding, token-identical for 50 tokens on real Qwen2.5-1.5B-Instruct weights. Greedy is deterministic, so any divergence means a bug.

It diverged at token 3.

Three forwards outvote a wrapper

The first three tokens matched, then my engine said is (token 374) and generate() said can (token 646). Early divergence usually means something structural: a RoPE position off by one, or a GQA head grouping bug. So before touching code I checked which side moved, by computing the same next-token logits three independent ways:

A margin of 0.27 is not a floating-point tie. All three forward passes, including HuggingFace’s own cached path (the thing generate() supposedly wraps), agreed with each other to within 4e-05 and disagreed with generate(). When three forwards outvote the wrapper, the model math is fine and the wrapper is doing something to the logits.

The mechanism: processors vs warpers

It was: Qwen2.5-1.5B-Instruct ships a generation_config.json in the checkpoint, and it contains repetition_penalty: 1.1.

The part worth internalizing is why do_sample=False doesn’t save you. transformers splits generation-time logit manipulation into two categories:

• warpers: temperature, top-k, top-p. Part of the sampling machinery, disabled when do_sample=False.
• processors: repetition penalty, no-repeat-ngram, min-length. These edit the scores before token selection, and they apply regardless of the sampling mode.

So do_sample=False gives you argmax, but over logits that have already been modified. The repetition penalty divides positive logits by the penalty and multiplies negative ones (the CTRL-paper formulation transformers implements). Here, is appears in my prompt (“…paged attention is”) and its logit was +21.16. Positive, so it gets divided: 21.16 / 1.1 = 19.24, which drops it below can’s untouched 20.89. The “greedy reference” was greedy over different numbers than the model produced.

To be clear about whose choice is whose: Qwen shipping the penalty is legitimate, and generation_config.json exists for exactly this. The trap is that generate() applies it silently under a flag whose name promises otherwise, and that trap is checkpoint-agnostic: swap in any model that ships any processor (no-repeat-ngram, min-length, plenty do) and the same silent divergence fires.

Repro

Ten lines, checkable (transformers 5.13.1, float32, CPU):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig

MODEL = "Qwen/Qwen2.5-1.5B-Instruct"
m = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.float32).eval()
tok = AutoTokenizer.from_pretrained(MODEL)
ids = tok("The key idea behind paged attention is", return_tensors="pt").input_ids

print(tok.decode(m.generate(ids, max_new_tokens=10, do_sample=False)[0, ids.shape[1]:]))
m.generation_config = GenerationConfig()  # neutralize the checkpoint's defaults
print(tok.decode(m.generate(ids, max_new_tokens=10, do_sample=False)[0, ids.shape[1]:]))

Output:

 that the brain can only hold a limited amount of
 that the brain is not a passive receiver of information

Same model, same weights, same do_sample=False. Two different “greedy” continuations.

The fix, and where it goes

One line, in the test fixture, not the engine:

hf.generation_config = GenerationConfig()  # pure argmax, no penalties

After that: 50/50 tokens identical, and the golden test means what it says.

Nothing changes in the engine itself, and that’s a deliberate position, not a gap. A checkpoint’s generation_config.json is the model author’s suggested client-side defaults: useful for a chat demo, not part of the model’s contract. A serving engine executes the sampling parameters the request asked for, exactly. It doesn’t silently inject a checkpoint’s opinions into every request; silently injecting them is precisely what made generate() an unreliable reference here.

To be fair to transformers: this is arguably working as designed. The checkpoint asked for the penalty and generate() honored it. The trap is the expectation gap: nothing about do_sample=False suggests “…but scores may still be edited first,” and the failure mode it produces is the worst kind: coherent output, plausibly greedy, quietly not.

The takeaway

If you’re validating an inference engine against HuggingFace (which is everyone who builds one), “compare against generate()” means nothing until you’ve neutralized the checkpoint’s shipped generation config. Otherwise you can burn days concluding your engine is broken when the reference is the thing that’s lying. The defensive check is two lines:

print(model.generation_config)
# suspicious: repetition_penalty, no_repeat_ngram_size, anything non-default

If there’s a repetition_penalty in there, your greedy isn’t greedy.