Testing a local LLM one question at a time hides the failure: what we measured on a 64 GB Mac

We run local models inside coding-agent setups, where the model reads files with read-only tools and answers questions about a codebase. The usual way to check such a model is to ask one question and see whether the answer is right. We wanted to know whether that check is enough.

We found the gap by accident. A loop let the agent process swallow its standard input, so questions 2 to 6 reached the model as one large prompt instead of five separate ones. The results looked different enough that we turned the bundled condition into a formal test.

This post reports that test. It is a small observation about how to test, not a ranking of models.

A one-question test would never have shown the gap.

Everything ran on one machine, one model at a time.

A mixture-of-experts model uses only part of its parameters for each token (about 3B here); a dense model uses all of them. We do not publish file names, setting names or code from the codebase, so the tasks are described in general terms.

Six questions, each with an exact answer that a person had verified.

First we asked the six questions one per run. Both models answered all six correctly.

The MoE model needed 137.3 seconds in total and the dense model 417.4 seconds, so the dense model was 3.0x slower. On quality, though, the test showed no difference at all.

If we had stopped here, we would have concluded that the two models are equally reliable for this work.

Then we put the same six questions into one context and ran that bundle three times per model: 18 sub-answers per model.

The gap that one-question tests hid shows up immediately.

MoE: 9 of 18 (50%), no fully correct run (0 of 3), mean time 170.7 s. Dense: 17 of 18 (94%), fully correct runs 2 of 3, mean time 362.2 s, which is 2.12x slower than the MoE model.

In the MoE model's run 1, the process exited silently after 187.7 seconds: zero bytes on standard output and standard error.

A count of correct answers does not show how a run goes wrong, so we also recorded the failure modes.

For a role that delegates judgement, we consider a silent empty run the more harmful failure.

Speed claims usually quote decode speed: how many tokens per second the model writes. A real agent run also has prefill, the model reading everything it is given, here the files the agent opens. We measured both on the dense model, with the same prompt, warm, thinking off.

The MLX engine decoded about 2.1x faster than GGUF. On the bundled six-question run, though, wall-clock time was about the same: a mean of 481.7 s for GGUF and 492.2 s for MLX, with no meaningful difference. Most of the time in a real agent run goes to prefill (reading files), not to decode.

Both variants answered correctly on the questions not affected by a later code edit (see the limits below).

The speed ratio between the MoE and dense models also depends on the prompt: 4.8x in this test (40.0 versus 8.4 tok/s) and 7x in an earlier one. Please do not quote 7x as a general number.

1. Test agents with the load shape they will really face: long, bundled context, not only one question at a time.
2. Record failure modes, not only a score: silent empty output, dropped items, shallow reasoning, fabrication.
3. Treat decode tokens per second as a poor proxy for task time when prefill dominates.
4. Score with a script against a human-verified key, and keep the raw logs.
5. State the setup and the limits. Do not turn n=3 into a leaderboard.

• This is a small test: 6 questions x 3 runs x 2 models, one machine, one codebase, n=3 per cell. It is an observation about test design, not a model ranking.
• The software has moved on. The Ollama stack used here has since been replaced in our setup by llama.cpp. Model families are named as tested, and results may differ today. We claim nothing about current versions.
• The speed condition was run after a code edit changed the answer to one question (question 5), so that condition is compared on the other five questions only.
• The hardware is Apple Silicon, not NVIDIA. Do not mix these numbers with numbers from other people's setups.
• Scoring was exact match against a human-verified key. A first visual pass missed one fabrication in the dense model's run 2, and a script corrected it: always score with a script.

You do not need our code to apply the same idea.

1. Write tasks that have an exact answer a person has verified, so a script can score them.
2. Run each task alone, then bundle them into one long context the way your real runs look.
3. Repeat each condition several times and keep the per-run results.
4. Record the failure type for every wrong run: empty output, dropped item, shallow reasoning, fabrication.
5. Measure real wall-clock time, and measure decode speed separately.
6. Score with a script and keep the raw logs.
7. Write down the setup and the limits next to the results.

MoE (mixture of experts)

A model that uses only part of its parameters for each token.

Dense model

A model that uses all of its parameters for each token.

Prefill

The stage where the model reads everything it is given, such as the files an agent opens.

Decode

The stage where the model writes its answer, token by token.

tok/s

Tokens per second: how fast decode runs.

Wall-clock time

The real time from start to finish of a run, prefill and decode together.

Exact match

A sub-answer counts only if it equals the human-verified key exactly.