Can a Local LLM Redact the Same Document Five Times? 100 Korean Documents, Graded

Every number here comes from runs I did myself (local models on one RTX 3090 and a GTX 1080 Ti, Claude through the API). All documents and personal data are fake; the document generator, the grader and every run log are published. (한국어판: 회사 문서를 클라우드에 못 보낼 때)

Why I measured this

My own reasons, translated from Korean:

I want to stop personal and sensitive information from leaving because of the bioinformatics analysis reports I work on. If this works, I want to know whether I can write result reports without the API, using that information without handing it to the cloud.

"I can't send the data out" is the most common reason to run an LLM locally. So the first thing to check is whether a local model masks personal data without missing any. As in my tool-use measurement, every document ran five times. Personal data that leaks once has leaked, so "passed all five tries" matters more than the average.

Summary

I planted fake names, resident registration numbers (Korea's national ID), phone numbers, bank accounts, addresses and emails in 744 places across 100 Korean documents (consultation notes, resumes, contracts, civil complaints) and asked each model to redact each document five times. One item left in the output fails the document.
qwen3.8 27B on one RTX 3090 passed 96 documents in all five tries, close to Claude Sonnet 5.5 (99), at about 9 seconds per document.
Running a regex pass first and letting the model finish raised qwen3.8 27B to 99. The price: the regex wiped out every receipt number that looked like a phone number.
One try flatters. gpt-oss 20B passed 87.6% of single tries but only 61 documents in all five.
Small models that fit a 1080 Ti fell short. The best, gemma4 E4B, passed 49 documents in all five tries; qwen3.5 9B (thinking off) masked the "Name:" field, left "lent money to an acquaintance, Lee Jeong-su" in the body, and passed 5.
Numbers (ID numbers, phone numbers) were handled well by most models; the leaks were mostly names and addresses inside sentences.

Redacting personal data, each document five times

What I measured

Documents: 25 each of consultation notes, resumes and cover letters, contracts (lease, service, used car) and civil complaints, 100 in all. Each type has three templates filled with fake data, so the answer key is exact. A document is about 360 Korean characters.

To be masked (744 occurrences): 218 full names, 100 given names used alone (as in "Jeong-hun, you…"), 126 phone numbers, 108 addresses (81 road-name, 27 lot-number), 76 ID numbers, 66 emails, 50 account numbers. A value that appears twice in one document counts twice (735 distinct values). Some came in awkward formats on purpose:

ID numbers with no hyphen (6609111588681) or with spaces
phone numbers with no hyphens (01010594410), or with dots or parentheses
emails written as "abc12 (at) naver.com"
names of the people who wrote the document: the counselor, the lawyer, the civil servant

To be kept (285 traps): receipt and waybill numbers shaped like phone numbers (8720-7113-6228), a card company hotline (1588-2184), shops named after a person ("Ha-eun Pharmacy", "Geon-u Car Center"), court case numbers, court names, company names and law articles.

Why everything is fake: names come from the Faker Korean generator; ID numbers have a deliberately wrong check digit so no real person can have them; mobile numbers use the 010-1xxx range, which is reported not to be assigned to phones. With no real personal data in them, the documents can go to the Claude API.

Prompt (same for every model; translated here, the original is Korean)

Redact the personal data in the document below.

- Redact: person names, resident registration numbers, personal phone numbers, account numbers, personal addresses, emails
- Replace them with [이름], [주민번호], [전화번호], [계좌번호], [주소], [이메일] (name, ID number, phone, account, address, email).
- Names and hotlines of companies, shops and institutions are not personal data; leave them.
- Leave every other character unchanged and print the whole document. No explanations.

Grading: a value from the answer key still in the output is a leak. Partial masks such as "010-**-5678" or "900101-1****" count as masked. A phone number leaks if its last 8 digits remain, an ID number if its last 7 digits remain, an account if all its digits remain, an address if the road name and building number (or district and lot number) remain, an email if the part before @ remains. Changing spaces or hyphens does not hide a value. An empty answer would leak nothing, so an output that keeps less than 80% of the document's non-personal text also fails. Masked traps are counted separately. "Pass" in the tables means nothing leaked and the document was not mangled.

I tested the grader first: the original text graded as every item leaked, and a correctly redacted version passed all 100 documents. After the runs I pulled random leaks for every model and checked them against the original; none was a grading error.

Runs

Four local models on the 3090: Ollama 0.40.13, 4-bit (Q4_K_M; gpt-oss is MXFP4). Sampling and thinking are each model's Ollama defaults; only the seed changes between tries.
Two small models on one GTX 1080 Ti, Ollama 0.40.5. In a one-document test on the laptop, qwen3.5 9B at its default wrote over 20,000 characters of thinking and took 176 seconds, so it ran with thinking off. gemma4 E4B ran at its defaults.
Two Claude models: Anthropic API defaults. At default settings the model thinks when it decides to; Haiku thought in all 500 calls, Sonnet in 309. The 1,000 calls cost $3.20.

Results

Model	Where	Pass, one try	Passed all 5	Never passed	Traps kept	Time per doc
Claude Sonnet 5.5	API	99.8%	99	0	100%	3.5 s
qwen3.8 27B	3090	99.2%	96	0	99.9%	9.3 s
Claude Haiku 5.5	API	95.8%	92	1	100%	3.1 s
gemma4 26B	3090	94.2%	89	0	96.2%	19.7 s
gpt-oss 20B	3090	87.6%	61	0	97.0%	8.1 s
gemma4 E4B	1080 Ti	76.2%	49	5	99.9%	11.9 s
LFM2.5 24B	3090	24.0%	6	50	93.3%	29.3 s
qwen3.5 9B (thinking off)	1080 Ti	13.4%	5	72	91.9%	6.1 s
Regex only	CPU	0%	0	100	79.3%

(100 documents × 5 tries. "Pass, one try" is over all 500 tries; the other count columns are documents. Time is the median; for Claude it is API response time, not comparable with local. The "shop address" items below are left out of the pass score.)

Near Sonnet on one 3090. qwen3.8 27B passed 496 of 500 tries. All four misses were in the same try (the third of five): three left the address of the leased property in a lease contract, one left the given name in "Jun-hyeok, the manager, can be reached at…" in a service contract. The other four tries passed all 100 documents.

One try flatters. gpt-oss 20B looks fine at 87.6% per try, but over five tries 39 of 100 documents leak at least once, because it misses different things each time (no document failed all five). The same document gives a different result on each run, so one or two trial runs prove little.

Masking the form fields only. qwen3.5 9B (thinking off) and LFM2.5 masked "Name:" and "Phone:" fields but left names inside sentences. gemma4 E4B on the same 1080 Ti did much better (49) but still passed fewer than half. Actual qwen3.5 9B output:

Summary / In 2024 the applicant lent 14 million won to an acquaintance, Lee Jeong-su, and was not repaid.

Agent: [이름] / Customer: Hello, this is Seo Ji-won.

What leaked

Share of planted items each model left in place, over 500 tries:

Model	Given name alone	Full name	Address	Other
Claude Sonnet 5.5	0.2%	0	0	0
qwen3.8 27B	0.2%	0	0.6%	0
Claude Haiku 5.5	0.2%	0	4.0%	0
gemma4 26B	0	0.8%	4.2%	0
gpt-oss 20B	9.2%	0.7%	3.0%	0
gemma4 E4B	14.5%	3.3%	7.6%	account 0.8%
LFM2.5 24B	58.7%	29.5%	18.4%	account 7.6%, phone 4.8%, email 1.5%, ID 1.3%
qwen3.5 9B	85.3%	32.7%	44.6%	email 11.5%, account 3.6%, phone 2.4%, ID 0.5%
Regex only	100%	100%	0	email 12.1%, account 12.0%

The top six models in the table (Sonnet, qwen3.8, Haiku, gemma4 26B, gpt-oss, gemma4 E4B) missed no ID or phone number, whatever the spacing. The weak spot is a given name in the middle of a sentence, without the family name. Two misses by gpt-oss 20B (translated):

After joining, I will work to be remembered as someone you can trust with the name Su-bin.

The crosswalk light in front of the elementary school is too short. Ji-hu has twice been caught crossing when the light changed.

Besides the leased-property address that other models also left (21 times), gemma4 26B left the representative's name at the end of a lease 9 times: "OO Real Estate Office, representative Lee Seong-hyeon". It seems to have read the person's name next to the business name as business information.

Leaking through its own explanation. LFM2.5 once (1 of 500) ignored "No explanations" and appended a change log after the redacted document, with the original values in it (translated):

Lawyer's name: a person's name, masked as [이름] (original name: Woo Do-hyeon)

ID number: … masked as [주민번호] (original number: 870108 - 1851554)

Once it also wrote sentences that were not in the document ("the case number was filed on March 15, 2024, … This is confidential."). That is the risk of using a redacted result without reading it.

A regex pass first

A regex is the opposite of a model. ID numbers, phone numbers and addresses follow fixed formats, so it missed none of them; it cannot know that "Kim Jun-yeong" is a person, so every name leaked. So I masked the numbers with a regex first and gave the rest to the model.

Model	Passed all 5: model alone → regex first	Traps kept: model alone → regex first
qwen3.8 27B	96 → 99	99.9% → 79.3%
gemma4 26B	89 → 93	96.2% → 77.5%
gpt-oss 20B	61 → 68	97.0% → 78.7%
LFM2.5 24B	6 → 11	93.3% → 74.5%

Leaks went down, at a cost. My account-number pattern took every "digits-digits-digits" shape for an account, so it wiped out all 59 receipt, waybill, registered-mail and contract numbers that looked like phone numbers: 58 became [계좌번호], and part of one became [전화번호] (0% kept with the regex). qwen3.8 27B alone kept 99.7% of them. A usable regex has to look at the surrounding words, such as "send to the account". The regex also caught every address only because the addresses in these documents follow a clean "city, district, road, number" or "city, district, neighborhood, lot" format. Real addresses are much messier.

Where judgment split: the shop address

In the first grading Sonnet failed 9 documents, and 8 of them were the same spot:

I run Min-jae Butcher Shop, I am [이름]. … The shop's address is Seoul, Songpa-gu, Olympic-ro 269-21.

My prompt said shop names and hotlines are not personal data, and then I graded the shop's address as a personal address. That was my mistake. Sonnet treated the address as shop information and left it 90% of the time; Haiku left it 45%, qwen3.8 27B 20%. These 8 addresses are now left out of the pass score and reported separately. Before that change Sonnet passed 91 documents in all five tries, qwen3.8 27B 91 and Haiku 86.

The leased-property address in lease contracts, on the other hand, was graded as personal. That is where three of qwen3.8 27B's four misses and 20 of Haiku's 21 were. What counts as personal data has to be decided by a person before any model runs, and that decision moves the score by several documents.

Trying a laptop

I also tried a company laptop with an 8 GB GPU (RTX 4060). qwen3.5 9B fit on the GPU at about 5 seconds per document, but I stopped a little past two of the five tries:

The CPU sits at 100 °C for far too long while it runs. It could do serious damage, so I need another way.

So the same model with the same settings ran again on the 1080 Ti. Over the same two tries the pass rate was 18.5% on the laptop and 15.0% on the 1080 Ti. Same model, similar score; what changes is speed and heat. Pushing hundreds of documents through a laptop is a separate question.

So what would I use

One 3090 with qwen3.8 27B gets close to Sonnet. Still, 4 of 500 tries leaked. Almost all is not all; check redacted output once more before it leaves.
Small models that fit a 1080 Ti or a laptop fell short. The best, gemma4 E4B, passed 49 documents in all five tries; qwen3.5 9B produced output that looks fine in the form fields while leaving the names in the sentences.
Handle names you already know with a list. In my case most of what must stay hidden is known in advance: clients, institutions, contact people, sample IDs. Replacing those from a list is safer than asking a model, with the model as a net for what the list missed. I am testing a report workflow built that way.
Results themselves cannot be masked. Unpublished gene lists and values cannot be interpreted once hidden, so I plan to measure doing the interpretation locally as well.

Limitations

Generated documents. 100 documents from three templates per type repeat their sentence structure and are cleaner than real documents; the uniform address format is one example.
The same person (me) wrote the generator and the regex. I knew which formats would appear, so the regex score may be better than it would be in practice.
One prompt. A different prompt (for example "also mask names inside sentences") might lift the small models. Not measured.
Five tries is not many. More tries would leave "passed all" the same or lower.
Borderline items. The shop address is excluded; the leased-property address and the real-estate office representative's name were graded as personal.
Default settings. Local models used Ollama's default sampling and thinking (qwen3.5 9B with thinking off), Claude the API defaults. LFM2.5's Ollama default temperature is 0.1 and it always thinks at length (about 20,000 characters per document), which made it the slowest.
Errors and mangled outputs count as failures. That includes for gemma4 26B 2 Ollama server errors and 1 empty answer (1 server error in the regex-first run), for LFM2.5 23 mangled outputs including 11 empty ones (22 in the regex-first run), for qwen3.5 9B 4 outputs that dropped a sentence, and for gpt-oss in the regex-first run 3 including 2 empty. Counting leaks alone would raise a few numbers slightly (gemma4 26B 89→91, LFM2.5 6→10).
The laptop and 1080 Ti comparison covers one model, qwen3.5 9B, over two tries.

Reproduce

The document generator (gen_docs.py, seed 20261110), the regex (regex_redact.py), the runner (run_redact.py, for Ollama and Claude), the grader (score.py, with --selftest), the figure script, the 100 documents with their answer key, and every run log including model outputs are in the reproduction files.