Gemma 4 31B MTP MLX

Gemma 4 31B MTP MLX

Local MLX gateway for running Gemma 4 31B with the Gemma 4 MTP assistant drafter.

Current status: this is a text-only local gateway alpha. It supports Claude Code and Kilo Code protocol smoke testing, but it is not a full agent-ready gateway until tool/function calling, true token streaming, and stronger runtime concurrency are implemented.

Architecture Overview

flowchart LR
    clients["OpenAI / Anthropic compatible clients"]
    gateway["FastAPI local gateway"]
    guardrails["Auth, limits, rate limit, CORS"]
    selector["Server engine selector"]
    serialized["serialized engine
mlx_vlm.generate"]
    batched["batched-mtp engine
experimental ResponseGenerator"]
    upstream["mlx-vlm runtime"]
    target["Gemma 4 31B target model"]
    drafter["Gemma 4 MTP assistant drafter"]
    diagnostics["/health, /readyz, /metrics"]

    clients --> gateway
    gateway --> guardrails
    guardrails --> selector
    selector --> serialized
    selector --> batched
    serialized --> upstream
    batched --> upstream
    upstream --> target
    upstream --> drafter
    gateway --> diagnostics

The default profile is built for 64 GB Apple Silicon machines:

• Target: mlx-community/gemma-4-31b-it-8bit
• Draft: mlx-community/gemma-4-31B-it-assistant-bf16
• Draft kind: mtp
• Server draft block size: 4
• CLI draft block size: 6

`mlx-vlm` is pinned to a post-`v0.5.0` upstream `main` snapshot:

d85ca4d0f06abc55be5b359f2bf1d33c53be2608

`v0.5.0` includes the initial Gemma 4 MTP drafter and server support. This newer SHA additionally includes later upstream fixes for Gemma 4 MTP server batching and speculative-decoding utility refactors. It is a reproducibility pin for the tested upstream snapshot, not the original MTP feature merge commit.

Install

This project targets macOS on Apple Silicon with enough unified memory for the selected profile. The default `safe64` profile is intended for 64 GB machines.