Vllm Mlx

vllm-mlx

**Continuous batching + OpenAI + Anthropic APIs in one server. Native Apple Silicon inference.**

**Read this in other languages:** [English](README.md) · [Español](README.es.md) · [Français](README.fr.md) · [中文](README.zh.md)

[](https://pypi.org/project/vllm-mlx/) [](https://pepy.tech/projects/vllm-mlx) [](https://www.python.org/downloads/) [](LICENSE) [](https://support.apple.com/en-us/HT211814) [](https://github.com/waybarrios/vllm-mlx)

---

What is vllm-mlx?

A vLLM-style inference server for Apple Silicon Macs. Unlike `Ollama` or `mlx-lm` used directly, it ships **continuous batching, paged KV cache, prefix caching, and SSD-tiered cache**, and exposes **both OpenAI `/v1/*` and Anthropic `/v1/messages`** from a single process. Run LLMs, vision models, audio, and embeddings on Metal with unified memory, no conversion step.

Quick start (30 seconds)

pip install vllm-mlx
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching

**OpenAI SDK:**

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
r = client.chat.completions.create(model="default", messages=[{"role": "user", "content": "Hi!"}])
print(r.choices[0].message.content)

**Anthropic SDK / Claude Code:**

export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=not-needed
claude

Features

APIs

• OpenAI-compatible: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/rerank, /v1/responses
• Anthropic-compatible: `/v1/mes