LFM2.5 24B-A2B on a Korean Document-QA Benchmark: Below qwen3.8 27B (Thinking), No Detectable Gap to gemma4 12B

Every number here comes from runs I did myself (Claude Opus via the API, everything else on my own machines). The benchmark and the other models' results are the same as in Local RAG scored on 300 Korean questions. (한국어판: LFM2.5 24B-A2B, 한국어 문서 RAG 300문항)

Summary

• Accuracy (299 questions): 68.6%. The model always reasons before answering, so there is only a thinking run.
• With about 2.4B parameters active per token, no difference from gemma4 12B (66.6%) or qwen3.5 9B (65.6%) was detected. No difference was detected against gemma4 E4B with thinking (69.6%) or qwen3.8 27B without thinking (71.2%) either. Many models sit in this 65 to 71% band.
• It is below qwen3.8 27B with thinking (77.3%): LFM alone got 11 questions right, qwen alone 37 (p below 0.001).
• I could not compare it with gemma4 26B MoE, a comparator on its card, under equal conditions. It scored above gemma4 26B MoE measured with thinking off (63.5%; 29 / 14, uncorrected p=0.03), but LFM2.5 thinks and gemma did not, so this is not a ranking.
• The 14.5GB file fits entirely on two GTX 1080 Tis (11GB each). Generation runs at about 24 tokens/s, but thinking is long (median about 1,940 tokens), so a question takes 85 s.
• It is the first model measured with the kit fixed after the Nanbeige4.3 run. No answers hit the 20,480-token cap and none were empty.

What LFM2.5 24B-A2B is

Posted on Hugging Face by Liquid AI on October 12, 2026 (model card).

• 24 billion parameters in total, about 2.4 billion (2.45B) active per token. A MoE with 64 experts and top-4 routing; 30 of its 40 layers are short convolutions and 10 are attention.
• The card calls it "a pure reasoning model that always thinks before it answers". In Ollama it thought even with thinking turned off.
• Its pitch is fast agents running on-device (reasoning, tool calls). The card's benchmarks cover knowledge, instruction following, math, code, tool use and agentic work, compared against Qwen3.6 35B-A3B, Gemma 4 26B-A4B and Nemotron 3.5 Lightning. There is no Korean benchmark and no separate test of answering from documents (one agentic task uses retrieval).
• Korean is one of the 16 languages listed on the card.
• The card says: "It is not recommended for knowledge-heavy tasks on its own: pair it with retrieval or web search". The card adds that this plays to its tool-calling strength. This benchmark puts the retrieved pages into the prompt instead, so it is close to the recommended use, but not the same.
• License: Liquid AI's own (LFM 1.0).

Setup

Item	Details
Test set	Allganize Korean RAG evaluation set, 300 questions (finance, public sector, medical, legal, commerce). 299 after dropping one with no evidence page number, plus 20 questions I wrote whose answer is not in the documents
Retrieval	Hybrid (kiwi morpheme BM25 + bge-m3), top 5 pages from the whole domain. Every model gets the same 5 pages
Prompt	"Answer only from the excerpts; if the answer is not there, say you cannot find it"
Model	Liquid AI's official Q4_K_M GGUF (14.5GB)
Run	Ollama 0.40.5, two GTX 1080 Tis (whole model on GPU), context 32,768
Sampling	The card's values: temperature 0.1, top_k 50, repeat_penalty 1.05, seed 42
Output cap	20,480 tokens (thinking included)
Judge	qwen3.6 27B (88.5% agreement with Allganize's own verdicts, checked in the first post)

Two things differ from the other local models, which ran at temperature 0 with a 1,024-token cap (thinking off) or 8,192 (thinking on): this one ran at the card's sampling, and since it always thinks, with a 20,480-token cap. With Nanbeige4.3 I ran a long-thinking model at temperature 0 with an 8,192 cap and scored the cut-off answers as wrong. This time no answer hit the 20,480 cap. Seven went past 8,192 tokens and only one of those was correct, so the old cap would have changed the score by at most one question. Among the comparators, qwen3.8 27B, gemma4 E4B and Nemotron Bolt (all thinking) had no answers at the 8,192 cap, so the cap difference does not affect those comparisons. Raon-1 mini is the exception (below).

Results

Setting	Accuracy	Retrieval hit	Paragraph	Text	Table	Image
qwen3.8 27B, thinking	77.3%	90.1%	87%	84%	80%	44%
LFM2.5 24B-A2B	68.6%	79.4%	82%	64%	72%	33%
gemma4 12B	66.6%	77.4%	84%	62%	58%	32%
gemma4 26B MoE, no thinking	63.5%	74.5%	78%	67%	56%	30%

There are 147 paragraph, 45 text, 50 table and 57 image questions. "Retrieval hit" is the 243 questions whose evidence page made the top 5. Of the 20 questions with no answer in the documents, it declined 19. The miss was a question about the foreign ownership limit, where it answered with a different figure from the excerpt (the 15% limit on non-financial shareholders of regional banks).

Paired comparison on the same questions (exact McNemar test, 299 questions)

Comparison	LFM only / other only (p)
qwen3.8 27B, thinking (77.3%)	11 / 37 (below 0.001)
Claude Opus 5.5 (76.6%)	12 / 36 (below 0.001)
Raon-1 mini, thinking (74.2%)	12 / 29 (0.012)
qwen3.8 27B, no thinking (71.2%)	19 / 27 (0.30)
gemma4 E4B, thinking (69.6%)	19 / 22 (0.76)
gemma4 12B (66.6%)	20 / 14 (0.39)
Nanbeige4.3 33B, thinking, 29 capped answers rerun (70.6%)	20 / 26 (0.46)
Nemotron 3.5 Bolt E4B, thinking (67.2%)	22 / 18 (0.64)
qwen3.5 9B (65.6%)	29 / 20 (0.25)
gemma4 26B MoE, no thinking (63.5%)	29 / 14 (0.032)
EXAONE 3.5 7.8B (63.2%)	32 / 16 (0.029)

How to read it:

• It is below qwen3.8 27B (thinking) and Claude Opus 5.5. Both gaps stay significant after correcting for the 11 comparisons.
• It also scored below Raon-1 mini (thinking), p=0.012, but that does not survive correction. Raon-1 mini had 16 thinking answers cut at the 8,192 cap that were never rerun, so the real gap may be larger.
• From qwen3.8 27B without thinking down to qwen3.5 9B, no difference was detected. This 65 to 71% band cannot be ranked with a test set this size.
• It scored above gemma4 26B MoE and EXAONE 3.5 7.8B, but neither survives correction. These are p around 0.03 from 11 comparisons with no multiple-comparison correction. Also, gemma4 26B MoE was measured with thinking off, while LFM2.5 always thinks. This is not a comparison with gemma4 26B MoE thinking.
• Table questions (72%) stand out against gemma4 12B (58%) and gemma4 26B MoE (56%). Only 50 questions, so treat it as a hint.

What I saw measuring a model that always thinks

It thinks in English and answers in Korean. 318 of 320 thinking traces start with "We need…" (the other 2 with "We have a user…"), and in a random sample of 20 the reasoning prose was English in all of them (quotes from the excerpts and answer drafts are Korean). Every answer was in Korean. Twenty-seven traces contained Chinese or Japanese characters: 16 only copied characters from the documents, and 11 contained Chinese or Japanese words that are not in the prompt (现场, 実績 and so on). None of those model-made words showed up in an answer (3 answers copy Hanja from the documents).

Thinking and answer sometimes disagree. I found the two cases below while reading answers one by one; I did not count how often this happens overall. In one question the thinking settles on "thyroid cancer, 100 per 100,000" and the answer then says "not found in the document". In another, after 8,500 tokens of thinking, the trace says "the limit is 15억 per person (또는 30억?)", and the answer then states 30억, the value it had put a question mark on, as fact.

The repetition warning did not point to a problem this time. The check added after Nanbeige ("a line appears three or more times") flagged 12 answers, as written, but none were what it is meant to catch (loops that run to the cap). All 12 were in the thinking, all stopped on their own before the cap, and all produced a well-formed answer (2 of the 12 were judged wrong). The pattern was rumination: draft an answer, check it against the excerpt, paste the same draft again; four traces wrote "Thus answer accordingly." three times. There was no loop like Nanbeige's, where the same sentence repeats dozens of times until the cap cuts it. Nanbeige ran at temperature 0 with no repetition penalty, while this model ran with the card's repeat_penalty 1.05, so I cannot tell whether the difference is the model or the settings. I plan to refine the check so it flags only loops that run toward the cap.

Speed: two GTX 1080 Tis

	Value
Generation	about 24 tokens/s
Prompt processing	about 1,080 tokens/s
Median tokens generated (incl. thinking)	about 1,940
Per question (median of the 320-question main run)	85 s
GPU memory	about 16.2GB at 32k context as reported by Ollama (split over two cards, checked before the run)

• With 2.4B active, each token is cheap, but the long thinking makes each question slow. On the same two 1080 Tis, qwen3.8 27B without thinking took 31 s per question, with no detected accuracy difference (71.2%, p=0.30). For Korean document QA alone on this hardware, qwen3.8 27B without thinking gets there in a third of the time.
• The kit's 30-question speed pass (thinking-off settings, 1,024 cap) gave 45 s per question, but this model cannot turn thinking off, so 25 of the 30 were cut at 1,024 tokens. The table above uses the 320-question main run instead. Generation and prompt speeds matched between the two.
• I did not measure it on an RTX 3090.

Limitations

• One run each. Do not read 2 to 3 point gaps in the 65 to 71% band as a ranking. This model was sampled at temperature 0.1, so a rerun would flip some per-question verdicts; comparisons with p between 0.01 and 0.03 are fragile for that reason.
• The other local models ran at temperature 0 (cap 1,024 without thinking, 8,192 with); this one ran at the card's sampling with a 20,480 cap. On the same benchmark, temperature 0 vs the card's settings changed the correct count by 2 questions for both Raon-1 mini (thinking) and Nanbeige4.3 (no thinking), although 12 and 34 individual verdicts flipped.
• gemma4 26B MoE was measured with thinking off. I could not confirm what setting the card used for its comparison.
• One LLM judge (qwen3.6 27B), which saw only the final answer, not the thinking. No second judge this time, and no end-to-end check by a person. The judge is from the same family as the top comparator (qwen3.8); in the first post a judge from another family (Claude Opus) gave the same conclusions.
• Q4_K_M only.
• The card's main comparator, Qwen3.6 35B-A3B, was not measured.
• One Korean public-document test set. The tool calling and agent work the card is built for were not measured. The conclusion is limited to answering from Korean documents.

Reproduce it

Generation and scoring scripts and per-question verdicts are added to the reproduction bundle from the earlier post. The run is measure.sh (options --think-only --ctx 32768 --np-think 20480 --temp 0.1 --top-k 50 --repeat-penalty 1.05); the figure is figs_lfm.py.

Read next

• Korean document RAG leaderboard: every model and setting scored on this test set, in one table (Korean page with an English summary)
• Nanbeige4.3 33B on a Korean document-QA benchmark: rerunning the capped answers, and why the kit changed
• Local RAG scored on 300 Korean questions: the benchmark, qwen3.8 27B and the gemma4 results
• Raon-1 mini on a Korean document-QA benchmark: KRAFTON's MoE on the same test