OlivierLAVAUD

Q Agent — Development skill for Claude Code

Development community

Self-hosted llama.cpp stack for Qwen + MTP (multi-token prediction): a 125B MoE read from ≥ 80 GB of RAM, or a 4.33B Qwen3.5 Q4_K_M entirely on a 6 GB GPU.

How to install Q Agent

This entry records only its repository, not the path inside it, so there is no exact command to give. Open OlivierLAVAUD/Q-Agent and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Q Agent does

Self-hosted llama.cpp stack for Qwen + MTP (multi-token prediction): a 125B MoE read from ≥ 80 GB of RAM, or a 4.33B Qwen3.5 Q4_K_M entirely on a 6 GB GPU. OpenAI- and Anthropic-compatible APIs (Cline, Claude Code, Continue, Aider), automatic GPU detection, RAM/disk preflights before any download, resumable downloads, localhost-only by default.

Alternatives in Development

Llamacoder — Open source Claude Artifacts – built with Llama 3.1 405B 7.1k ★
Ralphy — My Ralph Wiggum setup, an autonomous bash script that runs Claude Code, Codex, OpenCode, Cursor agent, Qwen & 2.8k ★
Cline — Overture Integration for Cline 612 ★

README

🤖 Run a 125B-parameter AI on your own machine

A fully local, self-hosted chatbot served by **llama.cpp** behind an OpenAI-compatible API, with **two presets** for two very different machines: [**Qwen3.8-Flash-Next**](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) — a 125 B MoE frontier-class model read from system RAM — or [**Qwen3.5-4B**](https://huggingface.co/unsloth/Qwen3.5-4B-MTP-GGUF) — 4.33 B parameters, 2.83 GB, entirely on a 6 GB GPU. Both are wired for **MTP** (multi-token prediction, the newest speculative-decoding technique) in a single `docker compose up`; what that means *in practice* today is spelled out in the [reality check](#-reality-check-what-actually-runs) below.

No cloud, no API bills, no data leaving your machine. Just an OpenAI-compatible endpoint at `http://localhost:8080` that anything can talk to (curl, SillyTavern, Open WebUI, your own scripts…), plus a built-in web chat.

Sounds like every other local-LLM demo? This one is a bit different. It runs on **hardware that shouldn't be able to run it** — and that was the whole point.

⚡ Reality check: what actually runs

The honest answer to *"which model am I really running?"* depends on the preset selected in `.env` — and on which weights were actually downloaded. Nothing is fetched until the preflight has cleared the machine, so **the files on disk are the truth, not the headline**:

	Path A — `PRESET=flash-next`	Path B — `PRESET=qwen35-4b`
Weights	Qwen3.8-Flash-Next UD-IQ1_M (~74.5 GB) + shared MTP head (~2 GB)	Qwen3.5-4B Q4_K_M imatrix, 2.83 GB, MTP head inside the GGUF
Parameters	125 B stored, ~6 B active per token (MoE)	4.33 B
Runs in	SYSTEM RAM (`LLM_NGL=0`), GPU optional	100 % in VRAM (`LLM_NGL=99`)
Spec-MTP	enabled and supervised (`MTP_FALLBACK=1`)	off — head present, unused (see below)
On-disk status	~77 GB to download, machine needs ≥ 80 GB of RAM	the configuration validate