Part 10: State-of-the-Art Embeddings (Pick the Right Local Model)

Part 10: State-of-the-Art Embeddings (Pick the Right Local Model)

*By Terp - [Terp AI Labs](https://x.com/OnlyTerp)*

---

The main guide's one-shot prompt installs a local embedding model via Ollama. Pick the right tier for your hardware:

Embedding Model Tiers

Tier	Model	Dims	RAM	Speed	Quality	Best For
Budget	nomic-embed-text	768	~300MB	Fast	Good	Minimal hardware, getting started
Recommended	qwen3-embedding:0.6b	1024	~500MB	Fast	Great	Most setups (Mac Mini, laptops)
Power	qwen3-embedding:4b	2048	~3GB	Medium	Excellent	32GB+ RAM, want best local quality
GPU Beast	Qwen3-Embedding-8B	4096	~8GB VRAM (INT8)	Fast	SOTA	Dedicated GPU (RTX 3090+, 5080+)
Cloud ⚠️	Gemini, OpenAI, Voyage	varies	0	SLOW (2-5s)	Excellent	NOT recommended — latency kills UX

**⚠️ Do not use cloud embeddings as your primary provider.** Every memory search round-trips to an API server, adding 2-5 seconds of latency PER QUERY. This defeats the entire purpose of fast memory search. Local embeddings respond in <100ms. Use cloud only as a fallback if you have no local option at all.

The `qwen3-embedding:0.6b` model is the sweet spot for most users — it's from the same Qwen3 family that holds #1 on MTEB, runs on anything, and blows away nomic on quality. Install via `ollama pull qwen3-embedding:0.6b`.

If you have a dedicated GPU with 16GB+ VRAM, read on for the power user setup with Qwen3-VL-Embedding-8B.

---

Here's everything we learned building a production embedding system on a Windows RTX 5090.

---

Why nomic-embed-text Hits a Ceiling

	nomic-embed-text	Qwen3-Embedding-8B
Dimensions	768	4096
MTEB Rank	~Top 20	#1 family
Multimodal	No	No (text-only, which is what you need for memory)
Context	512 tokens	8192 tokens
Serving	Ollama (GGUF)	FastAPI custom server (INT8 quantized)
VRAM	~300MB	~8GB (INT8)

768-dim vectors mean less expressiveness. You get fuzzy matches where you wanted exact hits, and the model misses subtle distinctions between concepts.

Qwen3-Embedding-8B is from the Qwen3 family that holds #1 on MTEB. 4096 dimensions, 8K context, text-focused. If you have a GPU with 8GB+ VRAM, this is the upgrade.

**Note:** We originally used Qwen3-**VL**-Embedding-8B (the vision-language variant) but switched to the non-VL text-only version. Same 4096 dims, same cache compatibility, but more stable and we don't need multimodal embeddings for markdown files. The migration from VL → non-VL was seamless — no re-indexing needed.

---

The Architecture: Stark Edition Server

Don't just point Ollama at Qwen3-VL. Ollama uses GGUF format and can't serve this model efficiently. Instead, build a proper production embedding server — we called ours the "Stark Edition" because GPT-5.4 (Codex) bu