Nanbeige4.3 33B on a Korean Document-QA Benchmark: The Model That Leads qwen3.8 27B on Its Card, Compared Under the Same Conditions

Every number here comes from runs on my own machines. The benchmark and the other models' results are the same as in Local RAG scored on 300 Korean questions. (한국어판: Nanbeige4.3 33B, 한국어 문서 RAG 300문항)

Summary

• Accuracy (299 questions): thinking off 64.9%, thinking on 66.2%. Rerunning the 29 answers that hit the output cap with the card's settings gives thinking on 70.6%.
• That is below qwen3.8 27B with thinking (77.3%), the model the card compares against. Even with the rerun, Nanbeige alone got 6 questions right and qwen alone got 26 (p below 0.001).
• Its same-condition scores (66.2%, 64.9%) land next to gemma4 12B (66.6%) and qwen3.5 9B (65.6%): under the same conditions, a 33B model scoring like the 9 to 12B ones.
• A second judge (Claude Opus 5.5) gives the same conclusion. With Opus scoring, Nanbeige (rerun) gets 73.2% and qwen3.8 27B 80.6%; Nanbeige alone 3, qwen alone 25 (p below 0.001). No sign that the Qwen-family judge favoured qwen.
• In the card's table of 20 agentic coding, office automation and reasoning benchmarks, Nanbeige scores above qwen3.8 27B on 18 (qwen is higher on SWE-Bench Pro and GDPval). The card has no Korean or document-QA results, and its language tags are English and Chinese only. So this does not say the card is wrong. It says the lead does not carry over to Korean document QA.
• In the first thinking run, 29 answers hit the 8,192-token cap, and 20 of them were empty: the model was still thinking when it was cut. Half of this post is about rerunning those questions, and about the changes to my measurement kit so this gets caught next time.
• On an RTX 3090 it generates about 38 tokens/s, about 18 s per question with thinking off. qwen3.8 27B on the same card: about 85 tokens/s, 4 s.

What Nanbeige4.3 33B is

A 33-billion-parameter model posted on Hugging Face on October 24, 2026 (model card).

• The card calls it the "teacher model" of the Nanbeige4.3 series: build a strong agent model first, then distill it into smaller ones (e.g. Nanbeige4.3-3B).
• Its focus is agentic coding and office work ("cowork"). It is compared with qwen3.8 27B, Gemma4 31B and others on Terminal-Bench, SWE-Bench Pro, Automation Bench and similar, all "evaluated at their highest thinking effort".
• License: Apache 2.0.
• The card's language tags are English (en) and Chinese (zh) only. Korean is not mentioned.

So why test it on Korean? Because the card goes head to head with qwen3.8 27B, which is the top local model on this benchmark and on par with Claude Opus 5.5. I wanted to see how a model that leads qwen3.8 27B on the card's benchmarks does on Korean documents.

Setup

Item	Details
Test set	Allganize Korean RAG evaluation set, 300 questions (finance, public sector, medical, legal, commerce). 299 after dropping one with no evidence page number, plus 20 questions I wrote whose answer is not in the documents
Retrieval	Hybrid (kiwi morpheme BM25 + bge-m3), top 5 pages from the whole domain. Every model gets the same 5 pages
Prompt	"Answer only from the excerpts; if the answer is not there, say you cannot find it"
Model	bartowski's Q4_K_S GGUF (19.7GB). qwen3.8 27B is the official Ollama Q4_K_M
Run	Ollama 0.40.2, RTX 3090 (24GB, whole model on GPU), temperature 0, context 16,384
Output cap	1,024 tokens thinking off, 8,192 thinking on (same as every other model)
Judge	Primary: qwen3.6 27B (88.5% agreement with Allganize's own verdicts, checked in the first post). Second: Claude Opus 5.5 (same judging prompt)

I used Q4_K_S (19.7GB) rather than Q4_K_M (20.8GB) to fit the whole model on one 3090 with some room to spare. Q4_K_S used 21.7GB at 16,384 context; I did not try Q4_K_M.

Results

Setting	Accuracy	Second judge	Retrieval hit	Paragraph	Text	Table	Image
qwen3.8 27B, thinking	77.3%	80.6%	90.1%	87%	84%	80%	44%
Nanbeige thinking, 29 rerun	70.6%	73.2%	81.5%	82%	71%	76%	37%
Nanbeige thinking	66.2%	69.2%	76.5%	78%	64%	70%	35%
Nanbeige no thinking	64.9%	66.9%	76.1%	80%	71%	54%	30%

There are 147 paragraph, 45 text, 50 table and 57 image questions. "Retrieval hit" is the 243 questions whose evidence page made the top 5. Of the 20 questions with no answer in the documents, all three settings declined 19. The one they missed is the same in all three (the punitive-damages multiple in a smartphone throttling ruling): the model answered "1x", a number that is not in the excerpts.

Paired comparison on the same questions (exact McNemar test, 299 questions. "A only" = A right, B wrong)

Comparison	A only / B only (p)
Nanbeige thinking (rerun) vs qwen3.8 27B thinking	6 / 26 (below 0.001)
Nanbeige thinking vs qwen3.8 27B thinking	5 / 38 (below 0.001)
Nanbeige no thinking vs qwen3.8 27B no thinking (71.2%)	10 / 29 (0.003)
Nanbeige thinking (rerun) vs no thinking	31 / 14 (0.016)
Nanbeige thinking vs no thinking	26 / 22 (0.67)
Nanbeige thinking (rerun) vs gemma4 12B	26 / 14 (0.08)
Nanbeige thinking (rerun) vs Raon-1 mini thinking (74.2%)	11 / 22 (0.08)

How to read it:

• The gap to qwen3.8 27B is significant in every pairing. After fixing the cap problem it is still about 7 points.
• Thinking only shows a benefit after the fix. The first number (66.2%) was no different from thinking off, because the 29 capped answers (20 of them empty) were pulling the thinking score down. Note that 31 / 14 (p=0.016) comes from rerunning only the capped questions, which favours thinking on.
• Thinking lifts table questions from 54% to 70 to 76% (27 of 50 to 35 to 38). Text questions went from 71% to 64% in the first run, but that is 3 of 45 questions.

A second judge

The primary judge, qwen3.6 27B, is from the same Qwen family as the comparator qwen3.8 27B. To check whether it favoured qwen, I scored the answers again with Claude Opus 5.5 and the same prompt, as for the other models.

Model	Primary judge	Opus 5.5	Difference	Judges agree
qwen3.8 27B, thinking	77.3%	80.6%	+3.3	95.3%
Nanbeige thinking	66.2%	69.2%	+3.0	96.3%
qwen3.8 27B, no thinking	71.2%	72.2%	+1.0	97.0%
Nanbeige no thinking	64.9%	66.9%	+2.0	96.0%

• Opus is 1 to 3 points more lenient on both models alike. There is no sign that the qwen judge was extra lenient on qwen3.8 27B.
• With Opus scoring, Nanbeige (rerun, 73.2%) is still below qwen3.8 27B (80.6%): 3 / 25, p below 0.001.
• Two comparisons change with the judge. Nanbeige (rerun) vs gemma4 12B was not significant with the primary judge (26 / 14, p=0.08) but favours Nanbeige with Opus (23 / 8, p=0.01). Thinking off with the card's settings vs qwen3.8 27B without thinking goes the other way: significant with the primary judge, not with Opus (below).

The capped answers: what went wrong and how I re-measured

What looked off in the first results was the 22 empty answers. qwen3.8 27B never hit the 8,192-token cap with thinking on; its longest thinking was 4,558 tokens (Raon-1 mini hit it 16 times). Nanbeige hit the cap on 29 questions, and 20 of those were cut while it was still thinking, so the answer was empty. The other 2 empty answers ended before the cap with nothing in them.

The first run had two flaws.

1. Answers cut at the cap were simply scored as wrong. Only 4 of the 29 were right.
2. The thinking text was not saved, so I could not tell whether the model was thinking hard or repeating itself.

I reran just those 29 questions.

Rerun	Correct of 29	Finished on their own	Median tokens generated (incl. thinking)
First run (temperature 0, cap 8,192, context 16k)	4	0	over 8,192
Temperature 0, cap 20,480, context 32k	15	26	7,158 tokens
Card settings (temperature 0.6, top_p 0.95), cap 20,480, context 32k	17	26	5,207 tokens

Two things stood out.

• Rerunning the same prompts at temperature 0, 19 of the 29 now finished under 8,192 tokens. Some that had gone past 8,192 tokens finished in 422 or 714. But this rerun changed several things at once: context (16k to 32k), the cap, and about 4.2GB of the model spilling to the CPU. Also, 4 of the 29 had prompts over 10k tokens, so in the first run prompt plus thinking overflowed the 16k context. So I cannot point to a single cause. The first overruns were probably less "the model needed more room to think" and more instability: a small change in conditions makes the thinking much longer or shorter.
• Even with a 20,480 cap, 3 questions were cut again. In one of them the end of the thinking repeats "Final answer: not found in the document…" over and over. That one is a repetition loop, seen directly.

The card's sampling made thinking shorter, but 17 vs 15 correct is within chance (p=0.69). When I did the same check on Raon-1 mini, temperature 0 got 222 right and the card's settings 224. On this benchmark temperature 0 by itself did not move the score much. What moved it was scoring cut-off answers as wrong.

Thinking off had the same kind of problem. 92 of 320 answers were cut at the 1,024-token cap, against 0 or 1 for other models apart from gpt-oss 20B (76; its reasoning cannot be turned off). Rerunning them with a 4,096 cap, 45 were cut again and 43 were repeating sentences of the answer itself. One answer wrote "Compared with 2023, the share of mandatory spending in 2027 rose by 2.8%." six times in the first run (cap 1,024) and fifteen times in the 4,096 rerun before it was cut. Correct answers went from 70 to 72, no real change (64.9% to 65.6% overall). The judge also reads only the first 3,000 characters, so the ends of 44 answers over that length were never scored. I then reran all 320 thinking-off questions with the card's settings (temperature 0.6, top_p 0.95). Accuracy was 65.6%, no different from temperature 0 (18 / 16, p=0.86); 19 answers still had a line appearing three or more times, and answers cut at the 1,024-token cap went up from 92 to 118. With these settings it is still below qwen3.8 27B without thinking (14 / 31, p=0.016), although with the second judge it is 18 / 30 (p=0.11), so the thinking-off comparison depends on the judge. Without thinking, Nanbeige often loops on its own sentences, and neither a higher cap nor different sampling fixes it.

So I changed the measurement kit:

• The thinking text and the stop reason (cap or natural end) are always saved.
• The report now opens with warnings: answers cut at the cap, empty answers, and answers or thinking in which the same line appears three or more times in the last 4,000 characters (the cap warning fires above 5). Run on earlier measurements of qwen3.8 27B, gemma4 12B and Raon-1 mini (no thinking), none of these three warnings fired.
• The sampling a model card recommends (temperature, top_p, and top_k or repeat_penalty where a card gives them) is now an option.

The leaderboard gets 66.2% and 64.9%, the same conditions as every other model. The rerun figure of 70.6% is reported here only. It swaps in 29 questions run under different conditions (32k context, card settings), so it is not a like-for-like number.

Other differences

• Answers without thinking are long. Median 633 characters, nearly twice qwen3.8 27B with thinking (359). The sentence repetition above accounts for much of it.
• With thinking, answers get short (median 213 characters): the model settles things while thinking and then writes just the answer.
• Image questions (text that only exists inside a figure) are low, 30 to 37%, as for other models. Only 57 questions, so treat it as a hint.

Speed: RTX 3090

Same 30 questions, thinking off, context 12,288, with the whole model confirmed on the GPU. Medians. The same prompt is about 3,200 tokens with qwen3.8 27B's tokenizer and about 5,100 with Nanbeige's. 6 of Nanbeige's 30 answers ran to the 1,024-token cap.

	Nanbeige4.3 33B (Q4_K_S)	qwen3.8 27B (Q4_K_M)
Generation	37.8 tokens/s	84.5 tokens/s
Prompt processing	about 1,370 tokens/s	about 1,270 tokens/s
Median tokens generated	448	91
Per question	17.8 s	4.0 s

• Generation takes 11.8 s vs 1.0 s (median): 2.2x the speed gap times about 5x the tokens. Prompt reading (3.8 s vs 2.6 s) is added on top, which brings the per-question gap down to 4.4x.
• With thinking on, the median was 41 s per question (the 320-question main run).
• At 32k context the model no longer fits in the 3090's 24GB; 4.2GB spills to the CPU and generation drops to about 11 tokens/s.
• At 19.7GB it does not fit on one GTX 1080 Ti (11GB). I did not try two 1080 Tis.

Limitations

• One run each. As shown above, this model gives different answers on some questions even at temperature 0. Do not read a 2 to 3 point gap as a ranking.
• Different quantizations: Q4_K_S for Nanbeige, Q4_K_M for qwen3.8 27B. One step is unlikely to explain 7 points, but I did not measure it.
• The 70.6% rerun figure swaps in 29 questions under different conditions; it is not a full 320-question run with the card's settings. Only the capped questions got a second try, which favours this number. qwen3.8 27B never hit the cap, so that comparison is roughly fair, but Raon-1 mini had 16 capped questions with thinking on and was not rerun. The comparison with Raon (11 / 22) leans in Nanbeige's favour.
• The main runs used temperature 0 for every model; the card recommends 0.6. I reran all 320 thinking-off questions and the 29 capped thinking-on questions with the card's settings, but not all 320 thinking-on questions. With the other questions unchanged, even if all 29 capped questions were counted as correct, thinking on would be 74.6%, still below qwen3.8 27B (77.3%).
• The card recommends up to 262,144 output tokens. My cap was 8,192 (20,480 for the reruns).
• The card says to turn on preserve_thinking for multi-turn tool use. This benchmark is single-turn QA, so it does not apply (the card recommends leaving it off for general QA).
• Two LLM judges (qwen3.6 27B and Claude Opus 5.5), not checked by a person end to end. The two agreed on 95 to 97% of verdicts.
• One Korean public-document test set. The agentic coding and office work the card is built for were not measured. The conclusion is limited to answering from Korean documents.

Reproduce it

Generation and scoring scripts and per-question verdicts are added to the reproduction bundle from the earlier post. The main run is measure.sh; the capped-question reruns are nanbeige_long.sh and nanbeige_bc.sh; thinking off with the card's settings is nanbeige_offcard.sh; merging is merge_rerun.py; the figure is figs_nanbeige.py.

Read next

• Korean document RAG leaderboard: every model and setting scored on this test set, in one table (Korean page with an English summary)
• Local RAG scored on 300 Korean questions: the benchmark and qwen3.8 27B
• NVIDIA Nemotron 3.5 Bolt E4B on a Korean document-QA benchmark: a 4B model measured the same way
• Raon-1 mini on a Korean document-QA benchmark: KRAFTON's MoE on the same test