Same Task, Five Tries: 4 Local LLMs and 2 Claude Models on 100 Multi-Turn Tool-Use Tasks

Every number here comes from runs I did myself (local models on one RTX 3090, Claude through the API). Tasks and grader are the public BFCL ones, unchanged; the scripts and all run logs are published. (한국어판: 같은 일을 5번 시켰더니)

Why I measured this

My own reasons, translated from Korean:

I'm worried about cloud API costs, and I wanted to hand tool calling to models I can run locally and try all kinds of work with them. I have one RTX 3090 installed at home and had been thinking about how to make use of it.

So the question was: give a model on one 3090 some tools, and does it finish the job every time? Doing it once and doing it every time are different, so each task ran five times.

Summary

• Models had to finish multi-step jobs ("move the file, sort it, tweet the result") by calling tools. After each job a grader checked the actual state of the simulated systems against the answer. Saying "done" does not count if the state is wrong.
• Each of the 100 tasks ran five times per model. The average success rate and the number of tasks solved in all five runs told different stories.
• gemma4 26B on a single 3090 solved 60 tasks in all five runs (71.4% average), close to Claude Sonnet 5.5 (57 tasks, 67.0%).
• qwen3.8 27B solved the most tasks at least once (83) but only 42 in every run: it can do many of them, just not every time.
• When a model ended with "shall I go ahead?", answering "yes" made scores worse (Sonnet, all five: 57 → 43), because models also did the extra work they had offered. Answering "do exactly what I asked" kept every model's average success within 4 points of the original.
• Sonnet sometimes failed by doing more than asked: told to create an empty document, it wrote a title and a template into it in all five runs.

What I measured

Tasks: the 100 even-numbered tasks of BFCL v4 multi_turn_base (the Berkeley Function Calling Leaderboard, Apache-2.0). Each task runs simulated systems (a file system, trading, travel booking, messaging, a Twitter-like feed, vehicle control) and has 1 to 6 user requests in a row.

One example:

1. "In research directory, there is a file called 'research_notes.txt'. Please make a prioritized backup by moving a copy of it to the folder named 'archives' within research, labeling the new file as '2024_research_backup.txt'."
2. "Now, take the contents of '2024_research_backup.txt' that we just created and sort the lines alphabetically."
3. "Once the list is sorted, find out how many lines are in the sorted '2024_research_backup.txt' file."

Grading: the official BFCL checker (bfcl-eval 2026.3.23) re-executes the model's calls and, after each request, compares the system state and the execution results with the ground truth. Any mismatch fails the task.

Setup

• 4 local models: one RTX 3090, Ollama 0.40.13, Ollama's native tool calling, 32k context. Sampling was each model's Ollama default (see LFM2.5 below); only the seed changed between the five runs.
• 2 Claude models: Anthropic API, default sampling (temperature 1.0), no extended thinking.
• Each tool call was executed on the BFCL simulator and the result returned to the model; when the model answered without a tool call, the next request followed (BFCL's own protocol). Up to 20 model calls per request.

Results

Model	Where	Average success	Solved at least once	Solved in all 5	Mixed
gemma4 26B	3090	71.4%	80	60	20
Claude Sonnet 5.5	API	67.0%	76	57	19
LFM2.5 24B (temperature 1.0)	3090	63.8%	78	45	33
qwen3.8 27B	3090	62.8%	83	42	41
gpt-oss 20B	3090	53.8%	76	28	48
Claude Haiku 5.5	API	52.0%	69	36	33

(100 tasks × 5 runs, BFCL's own protocol. "Average success" is over the 500 runs; the other columns count tasks.)

Average and "every time" are different numbers. Sonnet and qwen3.8 27B are 4 points apart on average (67.0% vs 62.8%) but 15 tasks apart on "all five" (57 vs 42). qwen3.8 27B solved the most tasks at least once, so it misses tasks not because it can't do them but because it doesn't take the same path every time. If you are handing work to an agent, the second number is the one that matters.

One 3090, close to a mid-tier commercial model. gemma4 26B is a MoE with a small active parameter count, and on this test it was close to Sonnet 5.5 on both counts (60 vs 57 tasks, 71.4% vs 67.0%). This is one 100-task benchmark; don't read it as "better than Sonnet".

LFM2.5 and temperature. LFM2.5's Ollama default temperature is 0.1 (its model card's recommendation); every other model ran at 1.0. Low randomness makes it easier to take the same path every time, which favours "all five". The table therefore shows LFM2.5 re-run at temperature 1.0 like the others. At its card's 0.1 it averaged 67.0% and solved 55 tasks in all five runs.

When the model asks "shall I go ahead?"

Reading Claude's failed runs, I kept seeing the model stop mid-task and ask whether to proceed. Under BFCL's protocol a reply without a tool call ends the request, so asking is a failure. My reaction, translated:

I was startled. Claude failing, what is going on? I sensed something was wrong and went looking for the error.

To keep that from penalising cautious models, I added one rule and re-ran: if a reply ends with a permission question ("Should I…", "Would you like me to…", "Shall we proceed?" and similar), the user answers once and the same request continues. Same rule for every model.

First try, answering "Yes, please go ahead.": scores went down.

Model	Original	"Yes"
gemma4 26B	71.4 / 60	70.6 / 57
Claude Sonnet 5.5	67.0 / 57	63.4 / 43
LFM2.5 24B (temperature 1.0)	63.8 / 45	47.6 / 21
Claude Haiku 5.5	52.0 / 36	50.6 / 34

(Average success / tasks solved in all five. qwen3.8 and gpt-oss were stopped once I found the cause.)

The failed runs showed the cause was my rule. It caught "may I do what you asked?" but also "shall I do something else for you?". Say "yes" to the second and the model does work nobody asked for yet. On a support-ticket task, LFM2.5 showed the ticket and asked whether to update the description, resolve it, or do anything else; after "yes" it resolved the ticket with a reason it made up. The user's very next request was to resolve it with a different reason. BFCL compares the state after every request, so that run fails.

I thought: Claude thinks ahead and reacts in advance, and it made me trust it more. It also made me wonder whether I should just use a commercial model after all.

Second try, answering "Yes, go ahead with exactly what I asked. Do not do anything beyond that.": averages almost unchanged.

Model	Original	"Exactly what I asked"	Runs that ended in a question (of 500)
gemma4 26B	71.4 / 60	70.6 / 57	12
Claude Sonnet 5.5	67.0 / 57	66.4 / 59	102
LFM2.5 24B (temperature 1.0)	63.8 / 45	60.2 / 39	256
qwen3.8 27B	62.8 / 42	59.6 / 39	322
gpt-oss 20B	53.8 / 28	54.4 / 33	54
Claude Haiku 5.5	52.0 / 36	54.4 / 39	138

Every model's average success stayed within 4 points of the original, which 100 tasks can't tell apart from noise. Tasks solved in all five moved a little more (LFM2.5 −6, gpt-oss +5). On that count Sonnet (59) edged past gemma4 26B (57); a 2-task gap, like the original 60 vs 57, is too small to rank them. The idea that asking costs a model points does not hold in this data. For the two Claude models, the runs that ended in a question were concentrated on hard tasks: those tasks' original average success was 53% for Sonnet (71% for the rest) and 32% for Haiku (60%), and an answer often didn't save them.

Claude was not the model that asked most. qwen3.8 27B (322 runs) and LFM2.5 (256) asked far more; gemma4 26B asked 12 times. These counts include offers like "shall I do anything else?".

Where gemma4 pulled ahead

Split by the main system each task uses, the models have different strengths (average success, BFCL's own protocol):

Main system	Tasks	gemma4 26B	Sonnet 5.5	LFM2.5	qwen3.8	gpt-oss	Haiku 5.5
Trading	22	75%	84%	89%	86%	82%	87%
Twitter-like feed	21	67%	60%	41%	49%	36%	26%
Messaging	13	83%	62%	65%	68%	60%	55%
Support tickets	13	49%	48%	45%	54%	45%	40%
Vehicle control	13	80%	72%	77%	49%	48%	42%
File system	12	67%	65%	53%	55%	25%	48%
Travel booking	6	87%	77%	83%	80%	90%	63%

(A task using several systems is grouped by the first. Cells hold few tasks, so a few points mean little.)

gemma4 26B was lowest on trading but far ahead on messaging and the feed, which put it first overall. All 21 feed tasks are mixed with vehicle control, the file system or travel booking, so they often mean carrying an earlier result into a post.

Where Sonnet lost these tasks, it sometimes did more than asked. Told "create a document titled 'TeamNotes.txt' for keeping track of all the fresh ideas", the expected action is an empty file. In all five runs Sonnet wrote a TeamNotes - Fresh Ideas Tracker title and a "Date | Idea | Submitted By | Status" table header into it (once a "Date: / Contributor: / Idea:" form instead). A person might be grateful; the grader saw a file that no longer matched right after the first request. gemma4 26B created the empty file in four of five runs. One case does not explain the whole gap, and I did not count how common this kind of failure is.

"Done", but it wasn't

Some failed runs ended with the model telling the user everything was done, with no mention of a problem. On a vehicle task, gpt-oss 20B worked out itself that ~980 miles needs about 39 gallons, filled 26, and replied "Fuel level: 26 gal (≈ 98 L) in the tank – plenty for the ~980-mile trip".

I had first picked four examples like this. Before publishing, a review checked them against the ground truth, and three were my misreading: in one the user had asked for the average of a price and a volume, in another the model's "tank filled" was true and the run failed on ordering. Only the confirmed one is left. A failure that looks obvious still needs checking against the answer key.

I tried to count these "silent failures" per model but don't report the numbers. A rule-based classifier agreed with my hand labels only 68% of the time (82 of 120 runs). The original-protocol logs also kept only the first 500 characters of each reply, so later warnings were invisible. The re-runs saved full replies.

So which model would I use?

My take after seeing the results, translated:

I'm still puzzling over the results and trying to understand them, and it isn't easy. Claude not getting the top score surprised me. I need to find a hybrid way of working, using several models for different jobs. And I still don't understand why gemma4 26B came first, haha.

The per-system table looks like a starting point for that hybrid: give jobs most models handle well, such as trading, to a small model on the 3090, and pick the model with the best "all five" for jobs that chain several steps. That is a direction from one benchmark, to be checked on real work.

Limitations

• One task family, in English: 100 BFCL multi-turn tasks. Other agent tasks, or Korean instructions, may differ.
• Five runs is not many. "All five" only goes down with more repeats.
• Errors count as failures. That includes Ollama failing to process a model reply (original protocol: gpt-oss 7 runs, gemma4 4, qwen3.8 3; "exactly what I asked": gpt-oss 8, gemma4 4) and Claude API connection errors (original protocol: Sonnet 2; "yes": Sonnet 2, Haiku 2).
• Question detection is a rule. Only replies ending in listed phrasings counted as questions; other phrasings are missed, and some non-questions may be caught.
• Defaults only. Local models ran with Ollama's default sampling and thinking settings; Claude without extended thinking. Tuning per model could change the results.
• No Claude Opus. The same run would have cost about $50 (it solved 6 of 10 in a pilot).
• Local models are 4-bit builds (Q4_K_M; gpt-oss MXFP4).

Reproduce

The runners (run_bfcl.py, run_bfcl_claude.py for Claude), the summary and chart scripts and every run log are in the reproduction bundle: 3,500 original-protocol runs (LFM2.5 at both temperatures), 3,000 "exactly what I asked" runs and the 2,295 "yes" runs before I stopped them. Tasks and grader come from pip install bfcl-eval==2026.3.23. The permission answers are the --confirm-once ("yes") and --confirm-only ("exactly what I asked") options.

Next

When a new model ships claiming agent performance, I'll run this test alongside my Korean document-RAG set and post the results.

• Korean document RAG leaderboard
• Local RAG scored on 300 Korean questions