Model-Card Sampling vs Temperature 0 on a Korean Document-QA Benchmark: qwen3.8 27B and Nanbeige4.3, 320 Questions Rerun

Every number here comes from runs on my own machines. The benchmark and judging are the same as in Local RAG scored on 300 Korean questions. (한국어판: 모델 카드 권장 샘플링으로 다시 재 보니)

Summary

• All main runs on this benchmark used temperature 0. After a Hugging Face comment that qwen3.8 should be run at its model card's sampling, I reran qwen3.8 27B and Nanbeige4.3 33B on all 320 questions at their cards' recommended values.
• No accuracy difference showed up. qwen3.8 27B went from 71.2% to 72.2% (thinking off) and from 77.3% to 77.9% (thinking on). Paired on the same questions, both are within chance (p = 0.68, 0.75).
• Nanbeige4.3 with thinking rose from 66.2% to 70.6%, but the gain came from the larger output cap and context, not from sampling. Leaving out the 29 questions that hit the 8,192-token cap in the first run, it is 194 vs 196 correct. Those 29 went from 4 to 15 correct at temperature 0 once only the cap and context were raised.
• Higher temperature produced more answers that never finished. With thinking on, Nanbeige answers cut at 20,480 tokens: about 3 at temperature 0 (estimated), 10 at 0.6, 11 at 1.0.
• Seven answers cut at 20,480 were sampled again with a 40,960 budget. All seven stopped on their own, and all in fewer than 20,480 tokens. The cap was not too small; those samples had run away.
• On this benchmark I found no sign that temperature 0 cost points. What distorted comparisons was the output cap, which counted cut answers as wrong, more than the sampling.

Why rerun

I posted the Nanbeige4.3 33B results in the model's discussion tab. One reply suggested the same settings as Qwen (temperature 1.0, top_p 0.95, no token cap); a second recommended temperature 1.0 / top_p 0.95 / top_k 20 for both models, with qwen3.8 at exactly its card values. Fair point. This benchmark ran every model at temperature 0 for reproducibility, and most model cards do not recommend that. If qwen3.8 27B lost points to temperature 0, the comparison in the Nanbeige post and the leaderboard order could change.

So both models were rerun on all 320 questions (300 with answers in the documents plus 20 unanswerable ones I wrote) at the recommended values.

Model	Sampling	Source
qwen3.8 27B thinking off	temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5	Qwen3.8-27B model card
qwen3.8 27B thinking on	temperature 1.0, top_p 0.95, top_k 20	Qwen3.8-27B model card
Nanbeige4.3 33B thinking on and off	temperature 0.6, top_p 0.95	"Reasoning and chat" row of the Nanbeige4.3-33B card
Nanbeige4.3 33B thinking on (extra)	temperature 1.0, top_p 0.95, top_k 20	the value suggested in the discussion

The Nanbeige card gives two rows: "agentic and tool-use tasks" at temperature 1.0 / top_p 0.95 (max new tokens 65,536) and "reasoning and chat tasks" at 0.6 / 0.95 (262,144). Neither row sets top_k. Document QA is closer to reasoning and chat. I first ran the suggested 1.0 with top_k 20, then reread the card and ran all 320 questions again at 0.6. Both results are below.

Setup

Item	Details
Questions, retrieval, prompt	Same as before; every model gets the same top-5 pages
Runtime	Ollama on an RTX 3090; seed 42 whenever temperature > 0
Output cap	Thinking off: 1,024 tokens (unchanged). Thinking on: 20,480 tokens for both models (was 8,192), context 32,768 (was 16,384)
Judge	qwen3.6 27B (88.5% agreement with Allganize's verdicts, checked in the first post). No second judge for these reruns

For thinking on, the cap and context were raised together with the sampling, so the effects have to be separated. For qwen3.8 this was not needed: its longest thinking at temperature 0 was 4,558 tokens, so no answer had reached the 8,192 cap, and thinking off kept the same cap. Nanbeige is separated below.

Results

299 questions with known answers. "Better / worse" counts questions only the rerun got right vs only temperature 0 got right; p is an exact McNemar test.

Condition	Temperature 0	Card sampling	Better / worse (p)
qwen3.8 27B, thinking off	71.2%	72.2%	13 / 10 (0.68)
qwen3.8 27B, thinking on	77.3%	77.9%	6 / 4 (0.75)
Nanbeige4.3 33B, thinking off	64.9%	65.6%	18 / 16 (0.86)
Nanbeige4.3 33B, thinking on	66.2%	70.6%	23 / 10 (0.035)
Nanbeige4.3 33B, thinking on, suggested 1.0 / top_k 20	66.2%	71.2%	25 / 10 (0.017)

Nanbeige thinking off is the full 320-question card-sampling rerun from the earlier post.

• qwen3.8 27B moved by less than a point. For single runs at temperature > 0 that is noise. qwen3.8 27B with thinking stays the top local model on the leaderboard.
• Only Nanbeige with thinking rose significantly. With five comparisons, p = 0.035 is borderline. The next section takes the 4.4 points apart.
• Refusals on answerable questions ("not found in the document" when the gold page was among the 243 retrieved) rose with card sampling: 15 to 19 for qwen3.8 thinking off, 5 to 13 with thinking on. It did not show up as an accuracy difference.
• On the 20 unanswerable questions, qwen3.8 refused all 20 in every condition and Nanbeige with thinking refused 19 in every condition.

Where Nanbeige's 4.4 points came from

At temperature 0, Nanbeige with thinking hit the 8,192-token cap on 29 questions. Counting those 29 and the other 270 separately:

Questions	Temp 0, cap 8,192	Temp 0, cap 20,480	0.6, cap 20,480	1.0, cap 20,480
270 that finished within 8,192 at temp 0	194	(not rerun)	196	197
29 cut at 8,192 at temp 0	4	15	15	16

All cap-20,480 columns use a 32,768 context.

• On the 270, sampling changed nothing. 0.6 vs temperature 0, paired: 12 / 10 (p = 0.83).
• On the 29, the cap and context changed the score. At temperature 0 with only the cap and context raised, 4 became 15; 0.6 also got 15.
• 0.6 vs 1.0 is within chance too (211 vs 213 correct, 11 / 13, p = 0.84). The reply argued that lower temperature is for coding and 1.0 might suit this benchmark better; here it made no difference. At 1.0, though, refusals on answerable questions rose to 34, vs 24 at 0.6.
• The 70.6% published earlier is the temperature-0 run with only the 29 cut questions rerun at 0.6. The full 0.6 run also lands on 70.6% (211 correct), but by coincidence: 194 vs 196 on the 270, and 17 vs 15 on the 29. The same 29 questions at the same 0.6 gave 17 and 15 correct in two runs, which shows how large single-run noise is.

Cut answers: would more tokens finish them?

Some answers were still cut at 20,480. I counted an answer as "repeating lines" when more than half of its last 8,000 characters were lines that appear three or more times.

Nanbeige, thinking on	Cut at 20,480	Of which repeating lines
Temperature 0 (rerun of the 29 cut at 8,192)	3	2
0.6, top_p 0.95	10	5
1.0, top_p 0.95, top_k 20	11	9

The temperature-0 row covers only the 29 rerun questions. The other 291 finished within 8,192 tokens at temperature 0, so I assume they would also finish within 20,480, which gives an estimated 3 for all 320. If that holds, higher temperature raised unfinished answers from about 1% to about 3%. Nine of the answers cut at 1.0 had finished at temperature 0 in 216 to 7,297 tokens.

The seven cut answers without repeating lines (five at 0.6, two at 1.0) were run again with a 40,960 budget (context 53,248). These are fresh samples with the same settings and seed, not continuations.

Question	Sampling	Cap 20,480	Cap 40,960	Tokens at 40,960
q_39	0.6	cut, wrong	stopped, correct	3,182
q_104	0.6	cut, wrong	stopped, wrong	1,554
q_129	0.6	cut, wrong	stopped, correct	13,669
q_156	0.6	cut, wrong	stopped, wrong (second attempt)	9,679
q_234	0.6	cut, wrong	stopped, wrong	8,321
q_104	1.0	cut, empty	stopped, wrong (refusal)	2,018
q_211	1.0	cut, empty	stopped, wrong (refusal)	2,436

• All seven stopped on their own, and all in fewer than 20,480 tokens. The longest took 13,669.
• q_156's first attempt kept generating for over an hour and hit the HTTP read timeout; the retry stopped at 9,679 tokens. Same context and seed, different outputs. This setup does not reproduce an answer even with a fixed seed.
• So these answers cut at 20,480 look less like "too much to think about for the cap" and more like "that sample ran away". Sampled again, they finished within the original cap.
• The repetition check works on lines and is crude. At 1.0, q_104 and q_211 repeated one long sentence 28 and 23 times near the end of their thinking, but the lines were too long to trigger the check. Both finished in about 2,000 tokens when sampled again. The answers that the check did flag were not rerun, so whether they would finish on a fresh sample is unknown.
• Two of the seven reruns became correct. They are not added to the 70.6%: giving only the cut questions a second chance would favor that condition.

The 8,192 and 20,480 cuts are different cases. Of the 29 cut at 8,192 at temperature 0, 26 finished once the cap was 20,480: there, the cap was the cause. The answers cut at 20,480 finished when sampled again, so there sampling is the more likely cause.

The card recommends 262,144 max new tokens for reasoning and chat. On the RTX 3090 (24 GB), about 4 GB of the model already spilled to the CPU at a 32,768 context, and long generations ran at about 10 tokens per second. At 262,144 tokens that is more than seven hours per question, which a 3090 cannot run.

What changed in the method

• The leaderboard now has a "card sampling" column. Ranking stays on the column where every model runs under the same conditions (temperature 0); reruns at card sampling sit next to it. Raon-1 mini and LFM2.5 already have card-sampling rows, or were measured that way.
• Pick the card row that fits the task. When a card recommends different values per task, as Nanbeige's does, use the row closest to document QA and say which one.
• A single run at temperature > 0 has 1 to 2 points of noise. The same 29 questions at the same settings differed by two answers between runs, and in an earlier test the same model and settings on different hardware disagreed on 9 to 11 verdicts.

Limitations

• Each card-sampling condition was run once. Do not read 1 to 2 points as a ranking.
• For thinking on, the cap (8,192 to 20,480) and context (16,384 to 32,768) changed along with sampling. qwen3.8 never reached the cap at temperature 0, so it is unaffected; Nanbeige is separated above.
• At a 32,768 context part of Nanbeige spills to the CPU, and the median time per question rose from 41 to 99 seconds. Do not use these runs for speed.
• These reruns were judged by the primary judge qwen3.6 27B only, which is in the same Qwen family as qwen3.8. In earlier runs it agreed with the second judge, Claude Opus 5.5, on 95 to 97% of verdicts, with no sign of favoring Qwen models.
• One benchmark of Korean public documents. Temperature 0 may hurt more on math, coding or long reasoning. The conclusion is limited to answering from Korean documents.

Reproduce

Generation: card_rerun_3090.sh (both qwen3.8 conditions, Nanbeige at 1.0) and nanbeige_t06_3090.sh (Nanbeige at 0.6 and the 40,960 reruns). card_sampling_summary.py recomputes every number in this post from the per-question verdicts, which are in the first post's reproduction bundle.

Read next

• Korean document RAG leaderboard: the full table, now with the card-sampling column
• Nanbeige4.3 33B on the Korean RAG set: the first run and the output-cap problem
• Local RAG scored on 300 Korean questions: the benchmark and qwen3.8 27B