Agent Observability Stack

agent-observability-stack

A self-hostable observability stack for **LLM agents + a Linux host (Intel iGPU/NPU + NVIDIA GPU aware)**, built as a single LAN-accessible pane of glass. Two pipelines, one Grafana:

• Infra metrics (pull): CPU, RAM, disk, network + latency, containers, and accelerator telemetry — Intel iGPU/NPU and NVIDIA GPU (nvidia-smi: util, VRAM incl. per-process, temp, power, clocks) → Prometheus.
• Agent telemetry (push): OpenClaw diagnostics → OTLP → OpenTelemetry Collector → Grafana Tempo (traces) + Prometheus (metrics); LLM tokens, cost, latency, cache from a LiteLLM proxy; and local llama.cpp inference (/metrics: decode/prefill tok/s, queue depth, and an MTP speculative-decoding acceptance signal).

New agents, containers, and accelerators **onboard themselves** — no per-target config edits.

Demo

Full walkthrough (~72s) — the dashboards end to end: infra (host, accelerators, containers), the agents pipeline (cost, tokens, latency, call outcomes, traces, live agent inventory), the **Claude Code & RAG** dashboard, and the **QMD Vector Space** (RAG memory corpus + benchmark queries projected to 2D/3D).

https://github.com/user-attachments/assets/ef08dcad-d20c-43eb-aa14-42b8cd2c014c

Architecture

INFRA METRICS (pull)                       AGENT / AI TELEMETRY (push)
 node_exporter ────┐                        OpenClaw gateway          Claude Code (OTel)
 cadvisor         ─┤                          ├─ diag-prometheus ─(loopback textfile)─┐   │
 blackbox         ─┤                          │                                       │   │ OTLP
 intel igpu/npu   ─┼─► Prometheus ◄───────────┤                                       │   ▼
 nvidia gpu       ─┤        ▲                  └─ diag-otel ──OTLP─┐                   └► OTel Collector
 llama.cpp /metrics┤        │                                     ▼                        │
 litellm /metrics