Dotnet Skills Evals

dotnet-skills-evals

Evaluation harness for the [dotnet-skills](https://github.com/Aaronontheweb/dotnet-skills) Claude Code plugin. Tests whether AI coding assistants correctly activate and benefit from .NET development skills.

Goals

1. Skill Activation Accuracy - Given a user's .NET development task, does the model correctly identify which skill(s) to use from the catalog of 31 skills?
2. Skill Effectiveness - Once activated, does a skill actually improve the quality of generated code compared to a baseline (no skill)?
3. Size Impact - Several skills exceed Claude's recommended 500-line limit. Does this hurt effectiveness, or are larger skills fine?

Platform Parity

The dotnet-skills plugin is consumed by multiple AI coding assistants:

• Claude Code (primary) - reads SKILL.md on activation, can read additional files via progressive disclosure
• GitHub Copilot - supports agent skills with multi-file progressive disclosure (since Dec 2025)
• OpenCode - reads SKILL.md from ~/.config/opencode/skills/

All three platforms support the same skill directory structure with progressive disclosure: SKILL.md serves as the entry point (<500 lines recommended), with sibling reference files loaded on demand. See [Claude skill best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices#progressive-disclosure-patterns).

Current Focus: Akka.NET Skills

The initial evaluation targets the 5 Akka.NET skills, which are the most critical and also the most likely to have overlap/confusion issues:

Skill	Lines	Over 500-line limit?
akka-net-best-practices	1123	2.2x
akka-net-testing-patterns	1280	2.6x
akka-hosting-actor-patterns	614	1.2x
akka-net-management	685	1.4x
akka-net-aspire-configuration	719	1.4x

Architecture

This harness connects to Claude mod