claude plugin eval: test skills and plugins like you test code

You write a skill, try it a few times, it works, and you ship it to the team. A week later a new model comes out, or someone edits the description, and the skill silently stops being invoked. Nobody notices until someone complains. Claude Code now has a claude plugin eval command to test plugins and skills against real test cases, score them, and compare with a no-plugin run so you know what the plugin actually contributes.

How an eval run works

• The suite lives in your plugin's evals/ directory; each case is a subdirectory with a prompt.md (what a user would type) and one or more graders.
• Each run opens an isolated non-interactive session with only your plugin loaded, sends the prompt and lets Claude work until it finishes.
• Each case runs 3 times by default because agents are non-deterministic; the case score is the mean across runs.
• By default a no-plugin arm runs too: a positive Δ means the plugin improved the result.

Getting started

claude plugin eval init            # Claude interviews you, proposes cases and graders, trials them, writes files
claude plugin eval .               # run every case
claude plugin eval . --case <name> --runs 1 --ablation none   # iterate fast on one case

The output shows WITH, W/OUT, Δ and COST columns and produces an HTML report with every grader's verdict for every run.

Six grader types

There are no custom-code graders. Each grader can have a weight; a case passes when its score is ≥ --threshold (default 1.0).

Tips for a stable signal

• Write prompts like real users: don't name the skill, so you test whether Claude picks it from its description.
• Two graders per case: one on the result (reply, file) and one on the process (tool_used, tool_order).
• Use regex for long output: keep llm graders for short outputs, with rubrics written as concrete PASS/FAIL conditions.
• Include cases that shouldn't trigger: a "must not invoke the skill" grader with arm: both catches false triggering.
• The most common finding: Δ near zero with tool_used: Skill failing means Claude doesn't pick the skill on natural phrasing; fix the description and rerun.

Risks and caveats

• It costs real money: every run and every llm grader is a model call against your plan or bill. Calls are roughly cases × runs × 2 arms, plus judge calls. Use --ablation none while you're only tuning graders.
• Small judges can misgrade: if the skill fires correctly but Δ is negative, suspect the judge first; try --judge-model sonnet.
• Early access: the feature may not be enabled on every account yet.
• Different from skill-creator: the skill-creator plugin has its own evals/evals.json format; the two tools don't read each other's files.

How to apply it

• Dev: Run claude plugin eval init on the skill you use most; tune its description until the tool_used: Skill grader passes consistently.
• Tech Lead: Put evals into CI for your internal plugin repo; rerun whenever a new model ships.
• PM: Use the Δ column to decide which skills are worth maintaining; skills that don't improve results should go, to keep context lean.