Moe Swarm — Development skill for Claude Code

MoE-Swarm

Claude Code agents that optimize CUDA/Triton kernels. DeepSeek-V3 FP8 MoE: 3.17x faster than FlashInfer's CUTLASS.

Seven agents per round compete on the FP8 fused-MoE layer of DeepSeek-V3, on one NVIDIA H200. The speedup over `cutlass_fused_moe` is averaged over the 19 workloads of the [MLSys 2026 FlashInfer contest](https://github.com/flashinfer-ai/flashinfer-bench) (Track A), all 19 correct.

The repo holds the agent harness and the kernel it produced. The harness is built around this one kernel's spec; pointing it at another kernel means writing a new spec and roofline model for it.

Speedup is the mean of `baseline_latency / our_latency` over the 19 workloads, measured on one H200 with the SM clock locked at 1400 MHz. The contest's own baseline (TRT-LLM) runs only on Blackwell, so these numbers are not comparable with the contest leaderboard. On the largest prefill workload the call drops from 2383 to 1580 µs, with both GEMMs above 1 PFLOP/s.

Where the speed came from:

step	score	what changed
PyTorch port of the reference	0.13×	dequantizes each expert's weights to bf16 before every matmul
Triton kernel, first agent rounds	2.84×	FP8 weights read from HBM once, no dequantized copy, fused SwiGLU and combine, no host syncs, CUDA graph
three more agent rounds	3.01×	launch overhead, tile shapes, the combine epilogue
hand-written CUDA prefill path	3.17×	warp-specialized TMA + wgmma for T ≥ 512

Most of the gain is the first jump, and it is about memory traffic. Before every matmul, v1 converted each active expert's FP8 weights into a bf16 copy in HBM: 1 byte read, 2 bytes writte