총 36개의 글
A 40-claim eval for a local hallucination checker on 8B, 9B and 27B. The 27B caught fewer than the 9B, the union of the two small models beat every single model, and a frontier judge finished one claim ahead.
The intelligence-vs-cost chart making the rounds shows open models winning the value quadrant. True, but the x-axis is API token price. The cheap open winners are 100B-to-1T MoEs you can't run on a desktop GPU. Here's what you can actually self-host on an 11GB and a 24GB card, measured, and where the real ceiling is.
A claim-verification layer for a local RAG co-scientist, inspired by Karpathy's llm-wiki pattern. I tried to measure whether it catches hallucinations, almost shipped a false finding, and ended up with a clearer picture of what claim-checking can and can't do: it reliably catches values that are absent from the context, misses a real number pinned to the wrong question, and misses a false premise outright, and a model can't reliably referee its own blind spots.
The 'GPU poor' narrative says 24GB-and-below cards are eating well now thanks to QAT and MTP. But what about an 8-year-old 11GB GTX 1080 Ti? I measured it: Gemma 4 12B QAT at ~32 tok/s, Qwen3 8B at ~46, all fully on the GPU. Here's the table and where the ceiling is.
MTP gave Gemma 4 12B QAT a 1.95x generation speedup on my 3090. But the same model with the same MTP draft runs 0.87x — slower — on an M1 Max. Speculative decoding is a hardware-dependent lever, not a free switch. Here are the measured numbers and why the draft-to-verify ratio decides it.
QAT is the buzz around Gemma 4, so I ran it on actual old hardware. The quality claim holds up (vs naive Q4), the speed win is modest (~9%), and yes — you can run the 12B on an 8 GB card at 16k context. Here are the measured numbers and the exact recipe.
My last post doubled generation with MTP. A reader asked the question I'd skipped — what about prompt processing at long context? I measured prefill across context sizes on a 3090: a 64k prompt takes ~59s before the first token, and MTP can't touch that. Here's the math on when MTP's 2× actually matters, and when prefill swallows it.
A commenter pointed me at a faster backend and multi-token prediction to roughly double my 3090's throughput. I measured it one lever at a time: 35.7 tok/s on Ollama → ~75 with MTP, a real ~2.1× (a community re-test corrected my first lucky 80.2 draw). Here's the exact path, with the numbers and the gotchas.
Ollama sizes the KV cache to your context length, and the default can quietly push a model that fits in VRAM into a CPU spill — cutting throughput. A full num_ctx sweep of Qwen3.6-27B on a single RTX 3090 shows exactly where the cliff is, and why a bigger context is not free.
A field report: building a private, fully-offline hybrid-retrieval RAG over my own papers across old and new GPUs — the embedder that froze the whole GPU, the context setting that halved my speed, and why pooling the cards was a trap. Plus an MCP server so an agent can cite my corpus.
I pulled the just-released Gemma 4 12B and ran it on a GTX 1080 Ti. ~28 tok/s at Q4 on one card — but three things broke first, and going to Q8 (split across two cards, 30% slower) fixed both the token glitches and a domain answer the Q4 got confidently wrong.
A field report: a CPU-only, GPU-less distributed LLM pipeline (llama.cpp + quantized MoE) mining 10,000 papers — and the 4 silent data-quality bugs that nearly ruined the results.
I benchmarked Qwen3.6-35B-A3B (IQ4_XS) on a pair of 8-year-old GTX 1080 Ti cards. It runs at ~20 tokens/sec — and the answer to 'does the second GPU help?' is yes, but only ~20% faster, not 2×. Here are the real numbers, the VRAM math, and why a 35B model fits 22 GB at all.
Practical build guide for running four GTX 1080 Tis in a single rig — 44 GB combined VRAM at roughly half the cost of a used RTX 3090. Covers PCIe slot configurations on HEDT and Threadripper boards, 1500W+ PSU sizing, cooling (1000W heat dissipation), llama.cpp tensor-split setup, expected throughput on 70B Llama, Mixtral 8×7B, and Qwen3.6-35B-A3B, plus the honest cases where this is not the right choice.
Side-by-side comparison of GGUF quantization formats — Q4_K_M, Q4_K_S, IQ4_XS, Q5_K_M, Q5_K_S, Q8_0 — measured on Llama 3.1 8B and Qwen 3 14B with actual perplexity, MMLU accuracy, VRAM footprint, and tokens/sec on RTX 3090 and GTX 1080 Ti. Practical recommendations for picking the right quant for your hardware.
Practical deep dive into Ollama's OLLAMA_KEEP_ALIVE — the variable that controls whether your loaded model stays in VRAM or gets unloaded after each request. Covers timeout semantics, multi-model scheduling, the per-request keep_alive parameter, and how to optimize for single-user, multi-user, and shared-VRAM scenarios.
Qwen3.6-35B-A3B (April 2026 release) puts a 35B-parameter MoE model on a single RTX 3090 24GB at usable speed thanks to its 3B active parameters and Apache 2.0 license. Practical use cases — agentic coding (SWE-bench 73.4), 262K context document analysis, vision-language tasks, and tool calling — with realistic VRAM math, expected throughput, and where the model genuinely outperforms 8B alternatives.
When llama.cpp's --split-mode row beats layer on dual-GPU inference, when layer is faster, and why the answer is different on Pascal/Turing without NVLink than on Ampere with NVLink. Real benchmarks on 2× GTX 1080 Ti for Mixtral, Yi-34B, Llama 3.1 13B, with PCIe lane and tensor split notes.
How to make Ollama actually use both GTX 1080 Ti cards without NVLink — environment variables, tensor split configuration, and real tokens/sec benchmarks for 13B and 30B-class models. Where PCIe becomes the bottleneck, what works versus what just looks like it's working, and how the same setup compares to a single 3090.
A 2026 reality check for the GTX 1080 Ti: 11 GB VRAM, Pascal architecture, no FP16 tensor cores. Which modern LLMs (Llama 3.1, Qwen 3, Phi-4, Gemma 3) still load and run usefully, what hits OOM, real tokens/sec numbers from a 1080 Ti, and when it's time to retire the card.
Definitive comparison of the three most popular local LLM inference engines in 2026. Real performance benchmarks on RTX 3090, feature-by-feature matrix, setup walkthroughs, and a decision framework for picking the right tool for your use case.
I benchmarked 12+ Ollama models on an RTX 3090 24GB — real tokens/sec, VRAM, and quality scores. See which local LLM wins in 2026: Qwen3, DeepSeek, or Llama 4.
Qwen3 vs DeepSeek R2 vs Llama 4 on an RTX 3090 24GB — real tokens/sec, VRAM, and quality scores across coding, reasoning, and writing. Which local LLM wins in 2026?
My $4,000 home AI server for running LLMs locally — RTX 4090 vs 3090 vs 5090, full parts list, Ollama setup, power costs, and what I'd buy differently after 18 months.
반복적인 프로테오믹스 분석 작업을 OpenClaw로 자동화하여 연구 효율성을 90% 이상 향상시킨 실제 경험담. DIA-NN 파이프라인 구축부터 바이오마커 데이터베이스 개발까지, 구체적인 구현 과정과 성과를 상세히 공개합니다.
Mac mini M4를 24시간 상시 서버로 운영하면서 알게 된 전기세, 발열, 네트워크 설정, 원격 접속 노하우를 공유합니다. 가성비 홈서버의 정석.
Ollama has no built-in authentication. Here's how I secured my public-facing Ollama API with Caddy reverse proxy and X-API-Key header validation — complete with Windows setup, CORS handling, and Vercel integration.
Honest comparison between Ollama (local LLM) and ChatGPT/Claude cloud APIs in 2026. Cost analysis, quality benchmarks, privacy, and real-world use cases from someone who uses both daily.
Step-by-step guide to running Qwen 3 30B MoE model locally on an NVIDIA RTX 3090 (24GB VRAM) using Ollama. Includes performance benchmarks, port forwarding setup, and API authentication with Caddy reverse proxy.
Python 백엔드에서 R의 limma, clusterProfiler, fgsea를 호출하기 위해 Docker에 R + Bioconductor를 넣은 과정. rpy2 S4 객체 변환 실패, ContextVar 스레드 에러, 4.13GB 이미지 최적화 삽질기.
Claude API 비용이 부담되어 RTX 3090에 Ollama를 올리고 Caddy reverse proxy로 API 인증까지 구축한 과정. qwen3:30b의 환각 문제와 대응 전략까지 솔직하게 공유한다.
RTX 3090 GPU로 Docker 기반 로컬 AI 서버를 구축하면서 겪은 삽질과 해결 과정을 공유합니다. NVIDIA Container Toolkit 설정부터 LLM 서빙까지 실전 경험담.
Ollama 설치부터 로컬 LLM 실행까지 완벽 가이드. Llama 3, Mistral, Gemma 등 오픈소스 모델을 내 컴퓨터에서 프라이버시를 지키며 사용하는 방법을 단계별로 설명합니다.
Self-hosting vs cloud: a detailed cost comparison for 2026. Calculate the real costs of running your own server vs cloud services for storage, email, media, and more.
RTX 5090 vs RTX 4090 benchmark comparison. Detailed performance analysis in gaming, AI, and content creation. Is the upgrade worth it? Full specs and real-world tests.
Supabase 무료 플랜으로 실제 SaaS를 구축하면서 겪은 한계와 이를 우회하는 실전 전략을 공유합니다. 데이터베이스 크기, API 요청, Auth, Storage 제한을 극복하는 방법.