SkillEvaluator

SkillEvaluator


[](LICENSE) [](https://www.python.org/) [](https://docs.nvidia.com/skills/skillevaluator/)

SkillEvaluator is an open-source, multi-tier framework for evaluating AI agent artifacts, starting with agent skills: deterministic quality gates, semantic overlap detection, synthetic eval dataset generation, and live agent evaluation.

Agent skills are folders of instructions and supporting files that extend AI agents, as defined by the [Agent Skills specification](https://agentskills.io/). SkillEvaluator is part of the [NVIDIA Verified Skills pipeline](https://github.com/NVIDIA/skills).

Three-tier overview


Tiers are independent entry points; nothing requires running earlier ones first.

Tier	Purpose	Representative commands	Requires
Tier 1: Validation	Safe & well-formed?	validate, quality-check, security-scan, pii-scan, lint-scripts, rubric-eval	No API key for deterministic checks; the security extra plus external Semgrep, SkillSpector, and Gitleaks for full scanner coverage; a provider key for LLM checks
Tier 2: Deduplication	Overlap with what exists?	context-optimization-check, similarity-check	An embeddings provider; intra-skill analysis also needs a chat LLM — local OpenAI-compatible endpoints work
Tier 3: Live Evaluation	Does it help the agent?	create-eval-dataset, tier3 evaluate, compare	No credential for keyless templates and report inspection; a provider key for LLM generation and grading; live evaluation a