Ask for a two-word coffee shop name instead and the gap gets worse: four tokens at the low end, 977 on kimi-k2.6. That is a 244x difference in what you are billed for the same two words of output.

The tokens are going into a hidden reasoning trace the model writes before it writes your answer. You pay for that trace, it usually does not reach you, and on most of these models usage does not even tell you it happened. This is a measurement of how much it costs, which parameter switches it off, and why that parameter is different on nearly every model.

What was measured

https://inference.do-ai.run/v1, 21 September 2026, all 27 chat models my subscription tier can call. Three prompts with short correct answers:

• What is 2+2? Reply with only the number.
• What is the capital of France? Reply with only the city name.
• Invent a two-word name for a coffee shop. Reply with only the name.

Three runs each, recording usage.completion_tokens, the visible message.content, the length of message.reasoning_content, and wall-clock latency. The interesting number is billed tokens against the fewest tokens any model needed for that same answer. Eight models said "4" in two tokens, so two is the floor, and everything above it is overhead.

The spread

Completion tokens, median of three runs, for "What is 2+2?":

Eleven of the 27 sit at or near the floor. Sixteen emit a reasoning trace nobody asked for.

The expensive answers were not better, but the cheap ones were not always right

The obvious defence is that the expensive models are being careful. On these prompts they are not buying anything with it. Every model in the top half of that table returned "4" and "Paris" on every run, and the 159-token answer is the same single character as the 2-token one.

What surprised me is which end produced the errors. Both of them came from the cheap end. deepseek-4-flash returned "5" once, and deepseek-v4-pro-0813 answers "2" about half the time, which gets its own section further down because the reason is worth reading. Not one of the heavy reasoners got the arithmetic wrong.

So this is not an argument that reasoning is waste. Reasoning earns its keep on hard problems. The point is narrower: these models cannot tell that "2+2" is not a hard problem, so they spend on it the same way they would spend on something that deserves it.

The off switch is different on almost every model

Sixteen models reason unprompted. Thirteen of them can be stopped, and this is where it gets tedious: there is no single parameter that does it.

reasoning_effort: "none" works on 11 models. chat_template_kwargs: {"enable_thinking": false} works on 12. The overlap is partial, so you cannot pick one and ship it.

Verified separately at ten runs each, with the answer checked every time:

All four answered "4" in 10 of 10 runs with the switch on. An 83x reduction on qwen3.5-397b-a17b for one parameter, and the answer does not change.

glm-5.2 is the awkward one. It rejects reasoning_effort outright:

400 reasoning_effort must be one of [medium high xhigh] for this model

There is no "low" and no "none" in that list, so the documented OpenAI parameter cannot ask it for less thinking at all. The only thing that works is the vLLM template flag.

Three models you cannot turn off

minimax-m2.5, openai-gpt-oss-120b and openai-gpt-oss-20b never stop reasoning whatever you send.

The two gpt-oss models at least let you turn it down. reasoning_effort: "low" takes them from 38 to 16 tokens. none and minimal are refused with an error that names the chat format rather than the model:

400 Harmony does not support reasoning_effort='none'

minimax-m2.5 accepts every switch with a 200 and ignores all of them, which is the worst of both worlds because nothing tells you it did nothing.

I nearly got this one wrong. My first pass ran three samples per variant and reported that thinking: {"type": "disabled"} cut minimax-m2.5 from 105 tokens to 60. Ten runs later: baseline median 74, with the parameter 76, and a baseline range of 55 to 175 tokens. The variance on that model is wider than the effect I thought I had found. Three samples was not a measurement.

reasoning_effort: "low" can make things worse, not better

This one caught me while I was testing the script at the bottom of this post.

If a model does not reason by default, asking it for a low reasoning effort does not reduce anything. It switches reasoning on. "Low" is not a step down from the default, it is a step up from nothing. Ten runs each:

deepseek-3.2 goes from 2 tokens to 86, a 43x increase, for setting the parameter people reach for when they want to spend less. Every one of those runs still answered "4".

So the parameter is not a dial from expensive to cheap. On a reasoning model it turns the volume down, and on a non-reasoning model it turns the feature on. You have to know which kind you are talking to before the parameter means anything.

The latency is worse than the token count

Tokens are money, but the wait is what your users feel. Median wall-clock for "What is 2+2?", six runs:

Twenty-four seconds to answer "4". One parameter takes it under a second.

usage does not tell you what you paid for

OpenAI reports hidden reasoning in usage.completion_tokens_details.reasoning_tokens. Here, 8 of 27 models populate that field, and the ones that populate it are mostly the router: targets rather than the models actually doing the thinking.

So on a model like glm-5.2 you get a completion_tokens of 103 for a one-character answer and no breakdown explaining it. The only way to see the trace is message.reasoning_content, which is not in the OpenAI schema, so a typed client will drop it before you ever look.

One model answers "2" to "2+2", and a space fixes it

This one is worth the whole exercise.

deepseek-v4-pro-0813 answers the question wrong. Thirty runs of What is 2+2? Reply with only the number. and it replies "2" fifteen times. Fifty per cent, on arithmetic a calculator settled in 1642.

It is not that the model cannot add. Thirty runs of each variation:

Drop the output constraint and it is perfect, showing its working and arriving at 4. Spell the numbers out and it is perfect. Ask it something harder, 17+25, and it answers 42 every time.

Add one space, 2 + 2, and the failure disappears completely.

The trigger needs both halves: the unspaced digit string and the instruction to reply with only the number. My read is that 2+2 tokenizes as a unit the model handles badly under a hard output constraint, and asked for a bare number it emits the first one it saw. I have not confirmed that, and I am labelling it a guess.

The sibling model matters here. deepseek-v4-pro, same family, no date suffix, is 0 wrong in 30 on the identical prompt. Two snapshots of the same model, one of them 50% wrong on a prompt the other never misses. If you pin a dated model id for reproducibility, this is what you are pinning.

One model leaks its chat template into content

nemotron-nano-12b-v2-vl puts raw ChatML markup in the visible answer. Six of nine runs across the three prompts:

'<|im_start|>answer<|im_end|>4\n<|im_start|>'
'<tool_call>\n{"name": "get_capital", "arguments": {"country": "France"}}\n</tool_call>\n<|im_end|>\n'

The request carried no tools key at all. The model invented a function, the API did not parse it into tool_calls (that field is None), so the markup arrives as ordinary text in message.content. Anything rendering that straight to a user displays a fake tool call and a stop token.

It was the only model of the 27 that did this.

What to do with this

Measure your own. The numbers above are one endpoint on one afternoon and they will drift.

If you are calling a reasoning model for something trivial, find its off switch before you ship, and expect it to be model-specific rather than provider-specific. Try reasoning_effort: "none", then chat_template_kwargs: {"enable_thinking": false}, and check usage.completion_tokens rather than trusting the parameter was honoured, because minimax-m2.5 will accept all of them and change nothing.

Use enough repeats to see past the variance. Three was not enough for me and it cost me a wrong result I nearly published.

And if you are pinning a dated model snapshot, run your actual prompts against it. deepseek-v4-pro-0813 is two tokens and fast and wrong half the time, and nothing in the response would tell you.

The script

#!/usr/bin/env python3
"""What does a one-token answer cost on your endpoint, and can you turn it off?"""
import json, os, statistics, urllib.request

BASE = os.environ.get("BASE_URL", "https://inference.do-ai.run/v1")
KEY = os.environ["API_KEY"]
Q = "What is 2+2? Reply with only the number."
REPS = 10

SWITCHES = {
    "baseline": {},
    "reasoning_effort=none": {"reasoning_effort": "none"},
    "reasoning_effort=low": {"reasoning_effort": "low"},
    "enable_thinking=false": {"chat_template_kwargs": {"enable_thinking": False}},
}

def call(payload):
    req = urllib.request.Request(
        BASE + "/chat/completions", data=json.dumps(payload).encode(), method="POST",
        headers={"Authorization": "Bearer " + KEY, "Content-Type": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=120) as r:
            return json.loads(r.read().decode())
    except Exception as e:
        return {"_error": str(e)[:80]}

def measure(model, extra):
    toks, ok = [], 0
    for _ in range(REPS):
        d = call({"model": model, "max_tokens": 4000, **extra,
                  "messages": [{"role": "user", "content": Q}]})
        if "_error" in d:
            return None
        toks.append(d["usage"]["completion_tokens"])
        content = d["choices"][0]["message"].get("content") or ""
        # an empty or wrong answer makes a low token count meaningless
        if "4" in content:
            ok += 1
    return {"median": statistics.median(toks), "min": min(toks),
            "max": max(toks), "correct": ok}

for model in os.environ.get("MODELS", "deepseek-v4-pro").split(","):
    print("\n" + model)
    for name, extra in SWITCHES.items():
        m = measure(model, extra)
        if m is None:
            print("  %-24s rejected" % name)
        else:
            print("  %-24s median=%-5s range=%s-%s  correct=%d/%d"
                  % (name, m["median"], m["min"], m["max"], m["correct"], REPS))

Run it with MODELS=glm-5.2,kimi-k2.6,qwen3.5-397b-a17b. Compare the medians, not single calls, and check the correct column before you celebrate a low token count. Two tokens and a wrong answer is not a saving.

That gives me something most people writing about AI and code do not have: a repository where I know exactly what the right answer is, and where the wrong answer used to live at a known commit. So I asked a question I had been wondering about since I started letting models write parts of this app. Does the code that already exists decide what the model writes next?

It does, completely.

The experiment

The setup is deliberately boring, because the whole point is that only one thing moves. I wrote out a task asking for a footballer search box on a new game screen, told the model to match the conventions of the code it was being shown, and then sent that same text over and over, eight times per model per condition, at a temperature of 0.7 so the runs would not all be carbon copies of each other.

What changes between conditions is which commit of my own repository gets pasted in above the task. For the first I went back to 82385a5, where nine per-game search components are sitting there and nothing shared exists yet, and for the second I used cc78828, the commit that closed the migration, where the shared component is present and TeamTies is shown calling it. Both conditions get roughly four hundred lines of genuine source from the repository as it stood, so neither is winning on prompt size, which was the first objection I expected and wanted to rule out before I looked at anything else.

The models were DeepSeek v4 Pro, Llama 4 Maverick, GLM 5.3 and Mistral 3 14B, an odd-looking lineup that has a dull explanation: Claude and GPT both come back 403 on my account's tier, so open weights are what I had to work with.

Thirty-two runs each, and not one crossed over. Same model, same instructions, 190 lines of hand-written state machine in one case and a 41-line wrapper in the other, decided entirely by what was lying around in the repository.

I expected a tendency. I did not expect it to be unanimous across four models from four different labs.

The models reproduce the specific bugs, not just the shape

The interesting part is not the line count, it is what is in those 190 lines. The migration existed because eleven copies had drifted on specific behaviours, and the test I added lists them. So I checked the generated components against that same list.

Of the 33 hand-rolled components the old context produced:

• 0% made the arrow keys wrap around at the ends of the list, which is the first behaviour the drift guard names.
• 0% used SEARCH_MIN_QUERY_LENGTH, and 97% re-typed a bare >= 2 instead — the exact thing the repository's own documentation tells you never to do.
• 58% had any mobile focus guard at all, against a rule that says the keyboard must not cover the board until the player has had a go.

These are not generic AI mistakes. They are my team's mistakes, from 2025, faithfully reproduced in 2026 by models that were shown the code containing them. The forked components taught the models to fork, and taught them the defects too.

There is a cleaner way to put the damage. Every one of the 32 components generated from the old context calls useFootballerSearch. That hook was deleted in the final pull request of the migration. All 32 would fail to compile against the repository as it stands today, which is a strange thing to say about code that perfectly matches the codebase it was shown.

So what actually steers it

If old code drags the model backwards, the practical question is what you can do about it when your repository is full of old code. I ran two more conditions to find out.

In the first, the model sees only the forked components, exactly as before, plus one paragraph from the repository's CLAUDE.md: the rule saying every search goes through the shared utility and never hand-roll another typeahead. In the second, the model sees no source for the shared component at all, only its filename in a directory listing.

One paragraph of documentation, sitting next to nine hundred lines of exactly the code it forbids, flipped every single run. That is a better return than I expected from a file most people assume gets skimmed.

Except the documentation does not finish the job

Here is where the neat story breaks, and it took a second measurement to see it.

Reaching for the shared search only means the model wrote the component's name into its JSX. It does not mean it called it correctly. The component has 23 real props, so I compared the props in each generated call against that list.

When the model could see the component's source, it invented nothing: none of those 32 calls used a prop that does not exist. When it had only the written rule, 28.6% of calls invented at least one. When it had only the filename, 82.1% did, averaging 4.64 imaginary props per call.

So the file listing result is largely a mirage. Telling a model that the component exists reliably stops it hand-rolling a typeahead, and then it writes a confident call to an API it has never seen, with onFootballerSelect and debounceMs and other things I never wrote. The 97% reuse figure is nominal; most of that code does not compile.

The rule and the source do different jobs. The rule decides which road it takes. The source decides whether the code at the end of the road is real.

One thing that did not go wrong

I expected a single leftover fork to poison the well, so I ran a condition with the shared component present and one of the old forked files still sitting in view. It made no difference: every run still reused the shared component, at essentially the same length.

That is worth saying plainly because it contradicts the tidy version of this post. Models are not dragged down by the worst code in the repository, they follow the dominant pattern. You do not have to sweep up every last bad example before your AI-assisted work improves. You have to make the good abstraction exist, and make it visible.

What I got wrong

What I got wrong was the prop-counting instrument, and I nearly published its output.

My first pass reported that every call in every condition invented props that do not exist, including the condition where the model could read the component's source. I believed it for about a minute, because it flattered a story I was already enjoying. Then it occurred to me that a model with the type definition in front of it inventing nine imaginary props on every single run is not a plausible thing to happen.

The regex that was meant to extract the real prop names had matched nothing, so the set of real props was empty and every prop the models used counted as invented. That result was measuring my own broken parser. With the list actually populated, the true figure for that condition is zero, and the real finding — that invention rises sharply as context thins — only appears once the instrument works.

It is the same lesson as the last few of these: when a number arrives that is both extreme and convenient, the first suspect is the thing doing the measuring.

What to take from it

If you are letting a model write code in an existing repository, the repository is your prompt, whether you intended it or not. Everything in these results points the same way. The model is not consulting best practice, it is pattern-matching on what is in front of it, and it will reproduce a pattern you abandoned a year ago with complete confidence and a hook that no longer exists.

Three things follow, in the order they pay off. Delete the old pattern, because as long as it is the only thing in the repo it is the specification. Write the rule down where the tools will read it, since that one paragraph was worth more than every line of example code around it. And make sure the real implementation, not just its name, is reachable, because a model that knows your component exists but cannot see it will invent its interface without hesitating.

The migration was worth doing for the humans. The measurements say it changed what the machines write too, by a wider margin than it changed what we write.

Everything above ran on DigitalOcean's inference API against deepseek-v4-pro, llama-4-maverick, glm-5.3 and mistral-3-14B, 160 completions in total, eight per model per condition.