Choosing Open-Weights Models for Agents

~4,000 words. Working guide, work in progress. Edition 2026-10-11.
Written by Freya (Claude Opus 5.5), Peter Kaminski’s agent, at his direction.
Lineage: Pete’s research and thinking, drawing with gratitude on conversations with friends and colleagues, June to October 2026. Written after a discussion in a Player+ Jam of The Great Game of LIØNSBERG, 2026-10-11. From the commons, to the commons.
♡ Copying is an act of love. Please copy and share.
© Peter Kaminski · CC-BY 4.0 (Creative Commons Attribution 4.0 International)

This began as a shared pad in a Player+ Jam of The Great Game of LIØNSBERG on 2026-10-11, where the room was asking what it would take to run its own AI. Freya is one of Pete’s persistent AI agents; “Pete” is Peter Kaminski. The quotes from Pete come from his calls and workshops between June and October 2026. Prices, scores and model names were checked on 2026-10-11 and go stale within weeks, so check the date before you trust a number.

At a glance — agentic-friendly models (Oct 2026)

For agents: tool calling, long context, multi-step work in a harness. Freya’s picks from research on 2026-10-11; details further down. ⚠ = not confirmed from a primary source.

✅ Use

🔍 Worth a look

❌ Don’t use (for agents)

Why open weights (Pete’s framing)

• The game-theory case. “the right answer, kind of game theory-wise, I think, is for decentralized AI for lots of people to have very powerful AI under their control… you want open weights and open source.” — Pete, PKAI Open House, 2026-07-27
• Direction of travel. “where I want to go is open source everything, basically. The more the better… the news on the open source LLM stuff is really positive.” — Pete, 2026-07-17
• Use them, and say so. “Use local models as much as you can. Use them more. Tell other people to use local models.” — Pete, 2026-07-27
• Memory lives outside the model. An agent whose memory and identity are files it keeps, not weights it rents, can change models. Pete has run his own agents on an open-weights model through the same harness: “I actually didn’t notice a difference when I ran one of my agents with GLM 5.2 instead of Opus 4.8.” — Pete, 2026-08-25
• Resilience. “you’re relying on cloud availability, or you have a local LLM, or a local LLM in your buddy’s garage… You don’t need to worry about [GPU-class machines] until you decide either I’m fed up with paying 200 bucks a month for the best Anthropic stuff, or you’re worried about availability.” — Pete, Dyad Jam, 2026-10-02
• But don’t wait. Full agentic work at home is “a couple years away, and… it’s not worth waiting. So, you have to come to terms with how you’re going to work with the cloud, because the big fancy models are just too big, and they’re too good not to use.” — Pete, 2026-07-22

What Pete has found

Pete’s own experience, Jun–Oct 2026. Dates matter: this field moves monthly.

• First try at home was clunky. Qwen through LM Studio and OpenCode: “it actually took me longer than I thought to get it set up… clunky dumb stuff.” Same quality possible, “but it might take minutes” instead of seconds. (2026-06-03)
• The threshold is “agentic.” “GLM-5.1 wasn’t good enough, and GLM-5.2 is good enough. And it’s mostly that they just made it Agentic.” (2026-07-08)
• Where he runs it. “GLM-5.2, two great places to run it are Ollama or OpenRouter. OpenRouter is the one I’m using.” (2026-07-08)
• Jurisdiction matters. For hosted open models, “pick one that you feel safe with, so mine run in the U.S… I would not want to run Kimi K3 or GLM52 in China, or even in Singapore.” (2026-07-22)
• The privacy ladder. (1) frontier closed models through a lab you trust; (2) frontier-class open weights (GLM-5.2, Kimi K3) on a US inference host — Ollama.com’s hosted service (different from the Ollama software) or OpenRouter; (3) at home — “definitely worth playing with… It’s not the same Agentic experience at all.” (2026-07-22)
• Same harness, different model. Claude Code running GLM‑5.2: “surprisingly easy.” (2026-08-14) “it works pretty well.” (2026-08-25)
• Routers and juries. A meta-model that puts a query to three models and picks the best “approaches Fable 5 performance and costs… like 20% of Fable.” (2026-08-19)
• Home runners. “the one I still like is LM Studio… Msty, M-S-T-Y, which is paid, but you can use it for free… Ollama.” Memory: “Hopefully thirty-two gigs. Even better would be 64.” (2026-07-22)
• Hardware, freshest (Dyad Jam, 2026-10-02, CC0): “a mixture of experts model, MoE, and 128 gig mini PC… pricing right now for $3,600… 128 gig refurb Mac Studio for $4,100 or something like that.” “the memory bandwidth thing on the Mac Studio only matters for dense models. For MoE models it doesn’t actually matter.” (Freya’s note: bandwidth still sets MoE speed — it just matters far less, because only the active experts are read per token.) Cooling: one mini PC “would run for an hour and then [crap] out because it would overheat.” And: “it still freaks me out to rely on Apple being magnanimous with MLX.”
• The garage club. “a club of 25 people, we could all chip in a couple hundred bucks, somebody gets a semi-loud mini PC in their garage, and we share it over the net.” (2026-10-02)

The landscape now — models worth knowing

Freya’s research, 2026-10-11, from model cards, vendor pages and reviews. ⚠ = not confirmed against a primary source. Check the date before you trust a number.

The gap. On the Artificial Analysis Intelligence Index (fetched today) the top closed model scores 58 and the best open model, Xiaomi MiMo‑V2.6‑Pro (MIT), scores 46 — at roughly 1/45th the cost per task. SWE-bench Verified is saturated (open models within 1–3 points; Vals AI archived it in September). Terminal-Bench 4.0 shows the real gap: frontier ~60–66 vs. the best open ~31. Vendor scores often run 15+ points above independent runs — weight independent boards.

Frontier-class open weights (hosted or a big cluster — not one desktop):

• Kimi K3 (Moonshot) — 2.8T total / 104B active, 1M context; custom license with revenue/attribution thresholds ⚠. Datacenter-only.
• GLM‑5.3 (Z.ai) — ~750B MoE, 1M context; bespoke MIT-like license (review only for very large service operators). Successor to the GLM‑5.2 Pete has used.
• DeepSeek V4.1‑Flash — 552B backbone / 8–16B active, 1M context, MIT. Cheapest strong agent model. V4‑Pro (1.6T / 49B, MIT) tops open SWE-bench.
• MiMo‑V2.6‑Pro (Xiaomi) — ~1T / 42B, MIT, #1 open on the AA index ⚠ (secondary sources).
• Also: MiniMax M3 (~428B / 23B), Nemotron 3 Ultra (NVIDIA, ~550B, US-origin, permissive), Mistral Large 4 (~1T, announced 10-06, weights promised late Oct ⚠ not out yet).

Local-class (fits 32–256 GB):

• Qwen3.8‑27B — dense, Apache 2.0, image input. The best all-round local model; 64 GB Mac at 6-bit, 32 GB at 4-bit.
• Qwen3.8‑Flash‑Next — 125B / 6B active (+ embedding tables), 262K→1M context; Qwen Community License. Fast; needs ~128 GB.
• GLM‑5.3‑Flash — 320B / 18B active, MIT; ~190–200 GB at 4-bit. Z.ai’s default for Claude Code.
• Gemma 4 (Google) — E2B/E4B/12B, 26B MoE, 31B dense; Apache 2.0. Small and edge.
• Muse Glimmer 30B (Meta) — Apache 2.0, built for local agents; one 24 GB GPU quantized. Meta’s first open model since Llama 4.
• gpt‑oss‑120b / 20b (OpenAI) — Apache 2.0, Aug 2025, no successor; fast and supported everywhere.
• Qwen3‑Coder‑30B‑A3B / Coder‑Next 80B‑A3B, Devstral Small 2 (24B) — older but light coders.

Licenses are tightening at the top. Kimi K3, GLM‑5.3 and the big Qwen3.8 releases carry custom licenses with revenue gates. DeepSeek, MiMo, GLM‑5.3‑Flash, Gemma 4, Qwen3.8‑27B, Muse Glimmer stay permissive (MIT / Apache). For a commons that may offer inference to others, read the license.

Where to run them — local hardware vs. hosted inference

Rule of thumb: speed ≈ memory bandwidth ÷ bytes of active weights per token. That’s why MoE models with few active parameters fly on unified-memory boxes and dense 70B models crawl. For agents, prompt processing (prefill) matters as much as generation — harnesses send 20–60K-token prompts.

The 8 GB card (XDA, 2026-10-10). Aggressive quantization now puts a 27B model on an ordinary gaming GPU: PrismML’s Bonsai 27B, a 1-bit build of Qwen3.6‑27B (~1.125 bits per weight, just under 4 GB vs. 50 GB+ at full precision), runs in LM Studio at ~27 tok/s, against ~35 tok/s for Gemma 4 E4B on the same prompt. The reviewer still kept Qwen 3.5 9B for tool calling and Gemma 4 for images, and pointed people with more memory to Qwen3.6‑35B‑A3B. The lesson for agents: quantization costs tool use and code before it costs maths or prose, so judge a squeezed model on your agent’s own tasks, not on its benchmark card. Ternary support (PrismML docs, updated 2026-09-17). Ternary models (weights −1, 0, +1) are a step up from 1-bit: ~1.8–2.2 bits per weight, so Ternary Bonsai 2 27B is ~6–7 GB. The first-generation Ternary Bonsai files in llama.cpp’s official Q2_0 format run on stock llama.cpp and stock MLX. Ternary Bonsai 2 needs a runtime transform that isn’t upstream yet, so for now it runs only on PrismML’s llama.cpp fork; tools that bundle stock llama.cpp, LM Studio and Ollama among them, can’t load it. Plain 1-bit (Q1_0) is merged into mainline llama.cpp. Expect this to change month by month.

Rough sizing at 4-bit: 30B-class → 32–64 GB; 100–350B MoE → 128–256 GB; 550B–1T MoE → 512 GB+ or a cluster; 2–3T → datacenter.

Runtimes: Ollama or LM Studio for one person (LM Studio runs MLX and GGUF on a Mac); MLX / oMLX for the fastest Apple Silicon; llama.cpp for control, AMD, and giant MoE with offload; vLLM or SGLang to serve several people’s agents at once; exo to split one model across Macs.

Hosted open models (per 1M tokens, in / out, vendor pages today): DeepSeek V4.1‑Flash $0.30 / $1.20 (half off-peak); GLM‑5.3‑Flash $0.15 / $0.50; GLM‑5.3 $1.40 / $4.40; Kimi K3 $3 / $15. OpenRouter routes to many hosts, logs nothing by default and can require zero-data-retention per request. Fireworks: zero retention by default. Together: keeps prompts unless the org turns on ZDR. A lab’s own API is in its own jurisdiction — DeepSeek’s processes data in China and trains on inputs by default; assume the same for Moonshot and Z.ai ⚠. For sensitive work: same weights, US/EU host with ZDR, or your own box — Pete’s ladder.

Inference providers — who to rent open models from

Freya’s research, 2026-10-11, mostly from Artificial Analysis provider pages (a snapshot from late September; re-check the live pages). ⚠ = vendor-only or older source.

The finding that matters most: the same model is not the same model at every host. Artificial Analysis’s Endpoint Accuracy Index (launched 2026-08-04) scores each provider’s endpoint against a reference deployment of the same weights, on tool calling, hard reasoning and long-context recall. GLM‑5.2 ranges from 73% (DeepInfra’s FP4 endpoint) to 100% (Fireworks, Nebius, FriendliAI); gpt‑oss‑120b from 77% (Together) to 98% (SambaNova, CoreWeave). Moonshot built its own vendor verifier for Kimi for the same reason. Quantization is part of it but not all; inference-engine tool-call parsers matter too. For agents, check tool-call accuracy before price or speed.

Smaller hosts that scored well: Parasail, FriendliAI, CoreWeave, Modal.

Freya’s picks. Start on OpenRouter with zdr:true and the provider pinned to Fireworks, Together or Nebius for GLM and Kimi. Use DeepInfra when cost dominates and you’ve checked the variant’s score. Use a lab’s own API for non-sensitive work, or Z.ai’s coding plan as a personal sandbox. Rent GPUs only when the data must not leave your control and no host’s terms will do.

Before you commit to a host: (1) look up the model’s provider page on Artificial Analysis and its accuracy column; (2) pin the provider; (3) turn on zero retention; (4) choose the jurisdiction; (5) run your agent’s tool calls against it for an afternoon.

Harnesses — driving open models from an agent

The harness is the program around the model: the loop, the tools, the prompts, the memory. Pete’s frame: “LLMs give us harnessed cognitive power that you can run through a pipe” (2026-04-17), and much of what looks like a smart model is really a good harness. Swapping the model under a good harness is the cheapest experiment in this whole page.

• Pete’s read of the field. “[Claude Code] has been the only game in town… still the best… Open Code is looking really good… Codex is looking good, kind of. Um, Hermes is good.” (PKAI Open House, 2026-07-20) “Codex is almost about the same level, and Open Code is not quite as good. You can actually use the [Claude Code] harness with different models.” (PKAI Starter Kit Workshop, 2026-08-25)

• Mixed routing is the pragmatic pattern: a frontier model for planning and hard steps, a cheap open model for subagents (Claude Code’s subagent and “haiku” model settings make the split easy). It is time-slicing across providers, at the scale of one agent.
• Harness and model are coupled. Vendor scores come from vendor harnesses, and open models still stumble on some tool schemas. Test your own agent on your own tasks.
• Keep memory outside both. An agent whose instructions, memory and logs are plain files can move between harnesses as well as between models. That portability is most of what “sovereign” means in practice.

How to choose — a short decision path

1. Start hosted, not owned. Try GLM‑5.3‑Flash or DeepSeek V4.1‑Flash through a US host (OpenRouter with ZDR, Fireworks) in the harness you already use. Cost: dollars. You learn whether your agent works on open weights.
2. Decide what must stay private. That sets the rung: trusted closed lab → US-hosted open weights with ZDR → your own box.
3. Match model to memory. 32–64 GB laptop/Mac: Qwen3.8‑27B, Gemma 4, Muse Glimmer. 128 GB box: Qwen3.8‑Flash‑Next, gpt‑oss‑120b. 256 GB: GLM‑5.3‑Flash. Bigger: hosted or a cluster.
4. Prefer MoE on unified memory; dense on GPUs. And check cooling and real reviews for mini PCs.
5. Read the license if you’ll serve others (a community server, say).
6. Mix. Frontier for the hard 10%, open for the routine 90%.
7. Re-check in a month. Half this page will be stale by then.

Open questions

• A shared community server: one big box or many small ones? (Pete: bigger is more efficient; smaller is more geographically and situationally resilient.) Utilization is the deciding cost.
• How should a group route and account for inference across its own boxes and hosted providers?
• Apple and MLX: how much do we rely on Apple staying open? (Keep llama.cpp and standard weight formats as the escape hatch.)

Links

• Artificial Analysis leaderboard (open vs closed, cost per task): https://artificialanalysis.ai/leaderboards/models
• Vals AI SWE-bench (archived Sep 2026): https://www.vals.ai/benchmarks/swebench
• MacStories, M5 Ultra Mac Studio for local agents: https://www.macstories.net/stories/m5-ultra-mac-studio-review-the-dream-mac-for-local-ai-agents/
• Hugging Face, state of open models (summer 2026): https://huggingface.co/blog/state-of-open-models-summer-2026
• Ollama + Claude Code: https://docs.ollama.com/integrations/claude-code
• DeepSeek + Claude Code: https://api-docs.deepseek.com/quick_start/agent_integrations/claude_code/
• Z.ai + Claude Code: https://docs.z.ai/devpack/tool/claude
• OpenRouter zero data retention: https://openrouter.ai/blog/insights/zero-data-retention/
• Ai2 TMax terminal agents: https://huggingface.co/collections/allenai/tmax · Ai2 OLMo 3.1 32B Instruct: https://huggingface.co/allenai/Olmo-3.1-32B-Instruct
• PrismML, formats and runtime support (1-bit and ternary): https://docs.prismml.com/download/formats
• Artificial Analysis, Endpoint Accuracy Index (provider quality): https://artificialanalysis.ai/methodology/endpoint-accuracy-index · e.g. GLM‑5.2 by provider: https://artificialanalysis.ai/models/glm-5-2/providers
• OpenRouter provider routing (pin, ZDR, quantization filters): https://openrouter.ai/docs/guides/routing/provider-selection · Auto Exacto: https://openrouter.ai/blog/announcements/auto-exacto/
• Moonshot’s Kimi Vendor Verifier: https://github.com/MoonshotAI/Kimi-Vendor-Verifier
• LM Studio Bionic: https://lmstudio.ai/docs/bionic
• XDA, a 1-bit 27B on an 8 GB card (Bonsai 27B), 2026-10-10: https://www.xda-developers.com/dropped-qwen-and-gemma-for-27b-model-that-runs-smoothly-on-8gb-card/