sipratt-p

Model Comparison — AI skill for Claude Code

AI community

Local open-weight models (Qwen3.5-397B, MiniMax M2.5) vs cloud coding agents (Codex 5.3, Claude Code) on six work prompts with a fixed rubric, Feb 2026: reports, per-model results, raw outputs, script.

How to install Model Comparison

This entry records only its repository, not the path inside it, so there is no exact command to give. Open sipratt-p/model-comparison and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Model Comparison does

Alternatives in AI

Qwen Code — A command-line AI workflow tool adapted from Gemini CLI, optimized for Qwen3-Coder models with enhanced parser 20.8k ★
Pi Agent VS OpenCode — Customization & Control Comparison — Pi v0.52+ vs OpenCode v1.1+ (Feb 2026) Thesis: Pi and OpenCode are both MIT-licensed, open-source, model-agnos 526 ★
Slotstream — Run a 105 GB AI model on a Mac that can't hold it 396 ★

README

model-comparison

Local open-weight models against cloud coding agents on six work prompts, scored with a fixed rubric, on a Mac Studio M4 Ultra (512 GB). February 2026. Three rounds, all artifacts kept: the reports, the per-model JSON results, the raw outputs, and the scripts that produced them.

What was compared

Model	How it ran
Qwen3.5-397B-A17B, 4-bit MLX	local, LM Studio server, 208 GB RAM
MiniMax M2.5 (456B, ~46B active), 6.5-bit MLX	local, LM Studio server, 186 GB RAM
Codex 5.3	cloud, through the `codex` CLI
Claude Code (Sonnet)	cloud, through the `claude` CLI

Six prompts, one per category: a Next.js / Prisma race condition to diagnose and fix, an agentic deployment plan with a fixed tool set, a PRD for an audit-scheduling feature, a LinkedIn post under a word limit, a fencing optimisation problem, and a REST-vs-GraphQL list in a strict format. Each answer is scored 1 to 5 on five criteria (accuracy, depth, structure, usefulness, instruction following) and averaged per category.

Rounds

Round 1 (COMPARISON_REPORT.md): Qwen 3.5 vs MiniMax M2.5 at max_tokens=1000. Qwen 3.6 vs MiniMax 2.5 on average, but the token limit invalidated two of the six tests: both are reasoning models and spent the whole budget thinking, so MiniMax returned empty content on the coding task and both scored 0 on instruction following. The finding that survived is the operational one: give reasoning models 2,000 to 4,000+ tokens or they return nothing.
Round 2 (COMPARISON_V2.md): the same pair at max_tokens=4000. Coding a tie at 5.0, Qwen ahead on the agentic plan, MiniMax ahead on the PRD, the LinkedIn post (3.6× fewer tokens) and the maths proof.
Round 3 (COMPARISON_4WAY.md, v3/): the two local models plus Codex 5.3 and Claude Code. Averages 4.6 / 4.6 / 4.8 / 4.8. Cloud agents were about twice as fast (20 to 22 s per answer against 40 s). Each model had a category it owned: Code