Local RAG Scored on 300 Questions: a 27B on One RTX 3090 Tied Claude Opus 5.5, and Most Misses Happened Before the Model

Most "I built a local RAG" posts end with "it answered my question pretty well." Plausible and correct are different things, so I wanted a test where every question has a known answer and a known source page. Then I can score the answers, and I can also see where each wrong answer went wrong: the PDF parser, the retriever, or the model.

Every number here comes from runs on my own machines. Per-question verdicts and all scripts are in the repro bundle. I wrote six predictions down before running anything; two were wrong and I've left them in. A separate reviewer re-derived the numbers from the raw files before publishing, and its corrections are folded in. (Korean version of this post: 로컬 RAG, 한국어 문서에 정말 맞게 답할까.)

TL;DR

• Same retrieved pages, different generator: qwen3.8 27B with thinking on a single RTX 3090 was statistically indistinguishable from Claude Opus 5.5. 77.3% vs 76.6% under the main judge (McNemar p=0.82), 80.6% vs 80.3% under a second judge from a different model family (p=1.0). With thinking off, the same 27B was clearly behind Opus under both judges.
• For the top models, about two thirds of the wrong answers were already lost before generation. Opus 5.5's 70 misses split into 22 where the gold page has no text layer, 23 where retrieval ranked the gold page out of the top 5, and 25 where the model had the right page and still answered wrong.
• Tokenization alone moved BM25 hit@5 from 56% to 80%. Korean attaches particles to words, so whitespace tokens are a poor fit. Swapping in a morphological analyzer was one function.
• A 2024 commercial RAG product (Allganize Alli + Claude 3.5 Sonnet, 80.6%) beat my pipeline + Opus 5.5 (76.6%) only on pages without a text layer. On those 23 questions it got 19 right and Opus got 1. On everything else, Opus was slightly ahead (31 vs 25 discordant, not significant). Two years of model progress didn't make up for missing OCR.
• A 9B model on one GTX 1080 Ti with a tuned pipeline (65.6%) beat the 2024 LangChain quickstart + GPT-4-turbo (56.2%), p=0.003.
• 20 unanswerable questions I wrote myself: every model, 7.8B to Opus, refused all 20. I predicted the small models would invent answers. They didn't, which mostly tells me my unanswerable set was too easy.

Setup

Benchmark	allganize/RAG-Evaluation-Dataset-KO (MIT). 300 questions over Korean public PDFs in finance, public sector, medical, legal, and commerce, 60 each. Every question has a gold answer, gold page, and evidence type (paragraph, text, table, image). The dataset also ships the 2024 answers of ~25 commercial RAG systems with Allganize's O/X labels
PDFs	Only 21 of 64 source links were direct PDFs (the rest are government board pages), so I used the page-level mirror datalama/RAG-Evaluation-Dataset-KO (MIT, 64 docs, 1,450 pages) and checked it against the 21 originals (identical apart from one running header)
Scored	299 questions (one has no gold page) + 20 unanswerable questions I wrote
Parsing	PyMuPDF, pypdf, pymupdf4llm compared; pymupdf4llm (tables as markdown) for generation. No OCR
Retrieval	Page-level, within each domain. BM25 (whitespace vs kiwi morphemes), bge-m3 dense, and RRF hybrid; generation gets the hybrid top 5 pages
Generation	Every model gets the same 5 pages and the same instruction ("answer only from the excerpts; if it's not there, say you can't find it")
Local	Ollama, temperature 0, 1,024-token output cap, thinking off. The "thinking on" row and gpt-oss (can't turn reasoning off) use an 8,192 cap
Claude	Opus 5.5, Sonnet 5.5, Haiku 5.5 via API, effort medium, 16,000 max tokens, default sampling, no server-side fallback. Opus 5.5 can't disable thinking, so all three ran with the default adaptive thinking
Hardware	2× GTX 1080 Ti 11GB (WSL2), 1× RTX 3090 24GB (separate box). Ollama 0.30.2 / 0.40.5 on the 1080 Tis, 0.40.2 on the 3090
Judges	Main: qwen3.6 27B. Second: Claude Opus 5.5. Same prompt, O/X against the gold answer

Validating the judge first

The dataset includes Allganize's O/X labels for the 2024 systems, so I could check my judge against them before trusting it: 88.5% agreement, Cohen's kappa 0.76 over 452 answers, with my judge about 4 points stricter. Allganize reports roughly 8% disagreement between their own auto-judge and humans (on the finance domain).

So I didn't reuse the published 2024 scores. I re-judged the published 2024 answers with my judge. All eight systems I re-judged came out 3–10 points lower than their published numbers. The six in the table below kept their order (Upstage and LangChain ended up tied); one system not shown (Alli + KONI-8B) dropped from 5th to 7th.

The main judge (qwen3.6) and the headline model (qwen3.8) are the same family, so I re-judged every answer with Opus 5.5 as well. The two judges agreed on 92–97% of verdicts per system. Opus-as-judge was more lenient overall and noticeably kinder to Claude answers (+4 to +6 points vs. −2 to +5 for local models). The conclusions below hold under both.

Predictions vs. results

#	Prediction (written before running)	Result
1	Morpheme tokenization lifts BM25 hit@5 by ≥10 pts	56.2% → 80.3% (+24.1)	✅
2	Image-evidence questions have the lowest retrieval hit rate	Lowest under all four methods (hybrid: image 54%, paragraph 88%, table 84%, text 89%)	✅
3	Hybrid beats each single method at hit@5	81.3 vs 80.3 (kiwi) vs 77.3 (bge-m3), i.e. 3 questions; kiwi alone wins at hit@3	△
4	With the gold page retrieved, 7–12B models ≥70%; a 27B within 5 pts of GPT-4o-class cloud (re-judged)	7.8B 72.8%, 9B 76.5%, 12B 77.4% / qwen3.8 27B (no thinking) 71.2% vs Alli+GPT-4o 75.3%, but gemma4 31B is 5.1 behind and gemma4 26B MoE 11.8	✅ / △
5	Best local below Alli+GPT-4o (published 0.783), above LangChain+GPT-4-turbo (0.61)	On the same judge, qwen3.8 thinking (77.3%) is above Alli+GPT-4o (75.3%), not significant (p=0.58). Only "true" against the published number, which uses a different judge	❌
6	At least half of the 7–12B models invent answers to unanswerable questions	0/20 for every model	❌

1. Retrieval: whitespace vs. morphemes is worth 24 points

Method (pymupdf4llm text)	hit@1	hit@3	hit@5	hit@10
BM25, whitespace tokens	37.1%	49.5%	56.2%	59.5%
BM25, kiwi morphemes	61.9%	78.6%	80.3%	82.6%
bge-m3 dense	54.2%	71.9%	77.3%	80.9%
Hybrid (kiwi + bge-m3, RRF)	61.9%	77.9%	81.3%	82.9%

Korean is agglutinative: "은행법" (Banking Act), "은행법에" (in the Banking Act), "은행법상" (under the Banking Act) are three different whitespace tokens. If you copy an English BM25 example, that's what you get. Keeping only noun/verb/adjective stems plus numbers and Latin tokens from the kiwi analyzer made BM25 beat bge-m3 on its own:

from kiwipiepy import Kiwi
kiwi = Kiwi()
KEEP = ('NN', 'VV', 'VA', 'SL', 'SN', 'XR', 'SH', 'MM')
def tok_kiwi(s):
    return [t.form.lower() for t in kiwi.tokenize(s) if t.tag.startswith(KEEP)]
# BM25Okapi([tok_kiwi(page) for page in pages])

The same lesson applies to Japanese, Turkish, Finnish, and other languages where words carry suffixes. The parser choice barely mattered for retrieval (hybrid hit@5: PyMuPDF 81.3%, pymupdf4llm 81.3%, pypdf 77.9%).

2. Generation: same five pages, different models

Model (where it ran)	Main judge	Second judge	Gold page retrieved	Paragraph	Text	Table	Image
qwen3.8 27B, thinking on (3090)	77.3%	80.6%	90.1%	87%	84%	80%	44%
Claude Opus 5.5	76.6%	80.3%	89.7%	85%	84%	80%	46%
Claude Sonnet 5.5	73.9%	79.3%	86.0%	83%	78%	82%	40%
Claude Haiku 5.5	72.2%	78.3%	85.6%	83%	73%	76%	40%
qwen3.8 27B (3090)	71.2%	72.2%	84.0%	85%	76%	68%	35%
gemma4 31B (3090)	70.2%	72.9%	81.5%	84%	78%	66%	33%
gpt-oss 20B (3090)**	69.6%	71.6%	79.4%	82%	80%	70%	30%
gemma4 12B (1080 Ti class)*	66.6%	68.2%	77.4%	84%	62%	58%	32%
qwen3.5 9B (1080 Ti)	65.6%	66.9%	76.5%	84%	64%	52%	32%
gemma4 26B MoE (3090)	63.5%	68.9%	74.5%	78%	67%	56%	30%
EXAONE 3.5 7.8B (1080 Ti)	63.2%	61.5%	72.8%	81%	60%	50%	32%

Per-type columns use the main judge. Question counts: paragraph 147, text 45, table 50, image 57; "gold page retrieved" covers the 243 questions where the gold page made the top 5. *gemma4 12B answers were generated on the 3090 because Ollama 0.30.2 couldn't load it (see pitfalls); accuracy doesn't depend on the GPU, and 1080 Ti speed is measured separately below. **gpt-oss can't turn reasoning off; these are from a full rerun with an 8,192-token cap.

Paired comparisons (exact McNemar, 299 questions):

Comparison	Judge	Only A right	Only B right	p
qwen3.8 thinking vs Opus 5.5	main	10	8	0.82
〃	second	5	4	1.0
qwen3.8 thinking on vs off	main	25	7	0.002
〃	second	28	3	0.001 or less
Opus 5.5 vs qwen3.8 thinking off	main	25	9	0.009
〃	second	28	4	0.001 or less
qwen3.8 thinking vs Sonnet 5.5	main	15	5	0.04
Opus 5.5 vs Sonnet 5.5	main	12	4	0.08

Thinking was the single biggest generator-side lever: +6 points on qwen3.8, largest on table questions (68% → 80%), at 2.3× the time per question on the 3090 (4.1 s → 9.7 s).

3. Where the wrong answers come from

The green segment (gold page not in the top 5) is 14–16% for every model. Swapping in Opus doesn't shrink it, because the model never sees the evidence (accuracy on those 56 questions is 14–27% regardless of model; I didn't check why the few hits happened, likely similar info on another page or prior knowledge). The orange segment (had the page, still wrong) shrinks from 22% to 8% as the model gets better.

The green segment hides two different failures. For the two top models:

Why it was wrong	Opus 5.5 (70)	qwen3.8 thinking (68)
Gold page has no text layer (parsing)	22	22
Text exists, ranked out of top 5 (retrieval)	23	22
Had the gold page, answered wrong (generation)	25	24

At the top, parsing and retrieval each cost about as much as generation. If you already run a good model, OCR and retrieval are where the remaining points are.

4. Against 2024 commercial RAG: the gap is OCR

2024 system (published answers, re-judged with my judge)	Overall	Paragraph	Table	Image
Allganize Alli + Claude 3.5 Sonnet	80.6%	90%	64%	67%
Allganize Alli + GPT-4o	75.3%	88%	52%	61%
OpenAI Assistant + GPT-4-turbo	67.6%	82%	56%	39%
LangChain quickstart (pypdf, 1,000-char chunks, OpenAI embeddings) + GPT-4-turbo	56.2%	63%	54%	35%
Upstage + Solar	56.2%	67%	44%	28%
AnythingLLM + GPT-4-turbo	20.1%	16%	8%	23%

Alli + Claude 3.5 Sonnet scored 4 points above my pipeline + Opus 5.5, but paired, that's 43 vs 31 discordant questions, p=0.20. Splitting the questions shows where the difference is:

Question subset	Alli + Claude 3.5 Sonnet	My pipeline + Opus 5.5
23 questions whose gold page has no text layer	19 right	1 right
The other 276 (discordant counts)	25	31 (p=0.50)
242 non-image questions	83.9%	83.9%

All of the gap is on pages without a text layer. I didn't run OCR, so those pages arrive empty. Allganize says it uses its own page-level parser plus hybrid search; from these numbers, that parser evidently reads scanned and image-only pages. On pages that do have text, the 2026 model was slightly ahead. A model generation's worth of progress didn't buy back those 23 questions.

Tables went the other way (Opus 80% vs 64%, 11 vs 3 discordant, p=0.057). pymupdf4llm's markdown tables might be why, but the model differs too, so I can't separate the two.

The LangChain quickstart + GPT-4-turbo (56.2%) lost clearly to qwen3.5 9B on one GTX 1080 Ti with my pipeline (65.6%), 28 vs 56 discordant, p=0.003. I can't say how much of that is parsing, chunking, or retrieval. These are pipeline comparisons under different conditions; the 2024 systems' retrieval scope wasn't published either.

5. Pages with no text layer

With pymupdf4llm, 178 of 1,450 pages (about 12%) yield fewer than 50 characters: scans or pages that are one big image. Four documents have no text at all (a 32-page KPMG report, an 11-page hospital cancer guide, a 9-page court case, a 6-page journal article). 23 of the 299 questions have their gold page on such a page, including 9 labeled "paragraph". A person sees text there, but the PDF only stores a picture. A text-only RAG can't answer those, ever.

The other 46 image-evidence questions do have text on the gold page, but the answer sits in a chart. There, Alli (30) and Opus (25) were close. The obvious next experiment is running a vision model over those pages and measuring how many of the 23 come back.

6. Unanswerable questions: everyone refused, which means the test was too easy

I wrote 3–5 per domain, plausible but absent (e.g., "In 2024 Tax Changes, what withholding rate applies to ad revenue earned in Korea by YouTubers living abroad?"), and kept only questions whose key terms appear nowhere in the extracted text of that domain's documents. All local models and all three Claude models answered "can't find it" on 20 of 20.

I read that as "these questions were too easy," not "every model is careful." The information was simply absent, and the instruction told them exactly what to say. The hard case is a document with a similar number under different conditions. In an earlier post, a real number attached to the wrong question was the hardest error to catch. That's the next question set.

What I got wrong first: my first tally said EXAONE invented 8 answers. Reading them, all 8 were refusals worded "the document does not specify…", a phrasing my refusal regex missed. Rule-based scoring needs eyeballing too.

7. Speed: GTX 1080 Ti vs RTX 3090

The same 30 questions (5 pages of context, median prompts ~3,200–3,600 tokens), with each model fully on the GPU. Seconds per question, prompt processing through the last token, median.

Model	GTX 1080 Ti (Ollama 0.40.5)	RTX 3090 (Ollama 0.40.2)
EXAONE 3.5 7.8B	9.1 s (one card)	—
qwen3.5 9B	8.4 s (one card)	—
gemma4 12B	10.0 s (one card)	2.0 s
qwen3.8 27B	30.9 s (two cards)*	4.0 s
qwen3.8 27B, thinking on	—	9.7 s*
gemma4 26B MoE	—	1.6 s*
gpt-oss 20B	—	4.4 s*
gemma4 31B	—	63 s*

*Median over the full 320-question run (num_ctx 16384) rather than the 30-question timing pass.

• gemma4 31B was ~15× slower than qwen3.8 27B on the same 3090. Ollama reported it fully on the GPU (19.3 GB), but prompt processing ran at ~75 tok/s vs ~1,270 for qwen3.8. Check this before choosing the 31B.
• qwen3.8 27B runs split across two 1080 Tis, 7–8× slower than the 3090 (173 tok/s prompt, 12.7 tok/s generation).
• Same model, same settings, different box: qwen3.8 27B scored 72.2% on 2× 1080 Ti (Ollama 0.40.5) and 71.2% on the 3090 (0.40.2). 11 verdicts flipped (7 one way, 4 the other). Don't read 1–2 point gaps between models as a ranking.
• qwen3.5 9B was 3× faster on a newer Ollama. 0.30.2 sized it at 11.4 GB and put only 3.3 GB on the GPU (23 s/question); 0.40.5 loaded it fully at ~5.5 GB (8.4 s).

8. Pitfalls I hit

1. Ollama 0.30.2 can't load gemma4 12B at all (unknown model architecture: 'gemma4-assistant'). All 320 requests failed into empty answer slots. Count empty answers after every run.
2. gpt-oss 20B can't turn reasoning off. With a 1,024-token cap, 48 answers came back empty (it ran out of tokens while reasoning) and 28 were cut mid-sentence. I reran all 320 with an 8,192 cap; 2 still ran out and are scored wrong. Accuracy went from 57.5% to 69.6%. Ranking off the first run would have blamed the model for my config.
3. Batching 16 inputs per bge-m3 embed call crashed the Ollama runner (connection refused). One at a time with retries fixed it.
4. The Claude Batch API is half price but its status lags. Sonnet and Haiku finished within an hour; Opus showed 0 processed for over two hours, and when I cancelled, it turned out almost everything was already done.
5. Many dataset source links aren't PDFs. 43 of 64 pointed at government or court board pages.

Cost

Claude generation, 320 questions × ~1.85M input tokens per model, via the Batch API: Opus 5.5 ~$5.7, Sonnet 5.5 ~$2.4, Haiku 5.5 ~$0.15. The second judge (Opus 5.5, standard API) was ~$12.2 (2.57M input, 0.09M output tokens). About $20 total. Local runs cost electricity.

Limitations

• LLM judges. The main judge agrees with Allganize's labels 88.5% of the time, and a second judge from a different family gave the same conclusions, but individual verdicts can be wrong. The Opus judge favors Claude answers, so for Claude vs. local, lean on the main judge.
• One run per local model at temperature 0. The same model moved by 11 verdicts across hardware and Ollama versions.
• The 2024 systems used different parsers, retrievers, and (unpublished) search scopes. That section compares pipelines, not models.
• The documents are 2008–2024 Korean public PDFs; internal company documents may behave differently.

Reproduce it

ko-rag-eval.zip contains every script, the pre-registered predictions, my 20 unanswerable questions, and per-question verdicts from both judges (verdicts.csv). The README inside is in Korean; the scripts are commented in English. The benchmark and PDFs aren't redistributed; get them from the two Hugging Face links above.

Order: parse.py → retrieve.py → gen.py (local) or claude_gen.py (Claude) → score.py (main judge) / judge2.py (second judge). Re-judging the 2024 systems is rescore_cloud.py, judge validation is validate_judge.py, speed is timing.py, figures are figs.py.

Related

• I added a verify layer to my local RAG: why a real number on the wrong question is the hard hallucination
• A fully local paper RAG on a 1080 Ti and a 3090: hybrid retrieval and reranking for English papers
• Ollama num_ctx trap: what a bigger context costs in speed