Claude Code's New Plugin Eval Command: How to Test and Score Your Plugins

On September 14, 2026, Anthropic shipped Claude Code v2.1.271 — and buried in the release notes next to output-style switching and artifact improvements is a feature that quietly changes how the Claude Code plugin ecosystem works: claude plugin eval. For the first time, plugin authors can run a scored, reproducible eval suite against their own plugin and get back a JSON and HTML report instead of just hoping it works.

If you've published a Claude Code plugin — or you're thinking about publishing one to a marketplace — this is the tool that turns "I tested it manually a few times" into something you can actually show a team lead, a marketplace reviewer, or your own CI pipeline. Here's what shipped, how to use it, and why it matters more than it looks at first glance.

What Shipped in v2.1.271

The headline feature is the plugin eval suite itself. Running claude plugin eval against a plugin executes its defined eval cases and produces:

A JSON report — machine-readable, suitable for CI gates or dashboards
An HTML report — human-readable, suitable for sharing with reviewers or teammates

This landed alongside a few related changes worth knowing about if you maintain plugins:

/output-style [name] — lists and switches output styles, including over Remote Control and in cloud/headless sessions (added in v2.1.269, extended in v2.1.271)
Plugin display metadata now prefers marketplace entries over the plugin's own plugin.json (v2.1.265), so what a marketplace shows about your plugin can diverge from what's in your repo unless you keep both in sync
Plugin installs and updates now support an --accept-command flag, letting you pin acceptance of a specific command hash rather than trusting whatever ships next
claude plugin validate --json (from v2.1.259) gives machine-readable validation reports, which pairs naturally with eval scoring in a CI pipeline

Put together, these are the pieces of a real plugin quality pipeline: validate structure, score behavior, pin trust to a specific hash, and ship. That's a meaningfully more mature workflow than what existed even a month ago, when "does this plugin work" was answered by running it a few times in a terminal and eyeballing the output.

Why Plugin Evals Didn't Exist Until Now

Claude Code plugins bundle skills, hooks, agents, and commands into an installable package that can be distributed through marketplaces — Anthropic's official one or an organization's private one. That distribution model has been live for a while, and plugin authorship has grown fast enough that unofficial marketplaces (curated collections of community plugins) have started showing up on GitHub.

The gap was quality control. A skill or hook can look correct on its SKILL.md or plugin.json structure and still fail in practice — a hook that fires on the wrong event, a skill that pulls the wrong file paths, a command that behaves differently across CLI, VS Code, and headless sessions. Manual testing catches the obvious breaks. It doesn't catch regressions when you tweak a prompt six months later, and it gives a marketplace maintainer nothing concrete to check before approving a submission.

claude plugin eval closes that gap the same way unit tests close it for application code: you write eval cases once, and every future change gets checked against them automatically.

How to Set Up and Run Your First Plugin Eval

Here's the practical path if you're maintaining a plugin today.

1. Confirm your Claude Code version. Plugin evals require v2.1.271 or later.

bashclaude --version

If you're behind, update through your normal install path (npm, the standalone installer, or your package manager) before continuing.

2. Validate plugin structure first. Before scoring behavior, make sure the plugin's structure is sound:

bashclaude plugin validate --json

This catches malformed plugin.json, missing required fields, or broken references to skills and hooks — the same category of error a linter catches before you run tests.

3. Define eval cases for your plugin. An eval case is a scenario: a prompt or trigger condition, and an expected outcome. If your plugin is a code-review skill, an eval case might assert that reviewing a file with an obvious SQL injection flags it. If it's a deploy-checklist command, an eval case might assert that running it in a repo without a Dockerfile produces a specific warning rather than a silent no-op. Model your eval cases the way you'd model integration tests: cover the golden path, then cover the failure modes you've actually seen in practice. 4. Run the eval suite:

bashclaude plugin eval

This executes your defined cases against the plugin and produces both a JSON report (for scripts and CI) and an HTML report (for humans).

5. Wire it into CI. Because the JSON output is structured and reproducible, you can gate merges on a minimum score the same way you'd gate on test coverage:

bashclaude plugin eval --json > eval-report.json
# parse eval-report.json in your CI step and fail the build below threshold

Reading the Scored Report

The value of the eval command isn't just "pass or fail" — it's reproducibility. Two different contributors running the same eval suite against the same plugin version should get the same score, which means:

Regressions become visible. If a prompt tweak drops your score from 95 to 80, you know immediately, not three user reports later.
Marketplace trust becomes verifiable. A plugin submission that includes an eval report gives a reviewer something concrete to check instead of a README claim.
Version comparisons become fair. You can score v1.2 and v1.3 of the same plugin against the identical eval suite and compare like-for-like.

This is the same shift the software industry went through when CI-gated test suites replaced "it worked on my machine." Claude Code plugins are following the same trajectory, just a few years compressed into a few release cycles.

Why This Matters for the Plugin Ecosystem — and for You

If you're a developer using Claude Code day to day, you're increasingly likely to install plugins from a marketplace rather than write everything from scratch — output-style packs, review workflows, GitHub release automations, and more are already circulating. Plugin evals give you a signal to actually check before installing: does this plugin's author publish eval results, and do they look rigorous? That's a much better trust signal than star counts or download numbers alone.

If you're building plugins yourself — for your own team's internal marketplace or for public distribution — this is the moment to add an eval suite before your plugin list grows large enough that manual testing stops scaling. The combination of claude plugin validate --json for structure and claude plugin eval for behavior gives you a two-stage quality gate that costs very little to set up now and saves real debugging time later.

And if you're studying for the Claude Certified Architect (CCA) exam, this is exactly the kind of platform-level detail that shows up in scenario questions about production Claude Code workflows — not just "what is a skill" but "how do you validate and score a plugin before shipping it to a team." Anthropic's certification content moves fast to reflect what actually shipped, and plugin quality tooling is a natural next topic once the basics of skills and hooks are covered.

Key Takeaways

claude plugin eval shipped in Claude Code v2.1.271 (September 14, 2026) and produces scored, reproducible JSON + HTML reports for a plugin's eval suite.
It pairs with claude plugin validate --json (structure) and the --accept-command install flag (trust pinning) to form a real CI-ready plugin quality pipeline.
Marketplace metadata now prefers marketplace entries over a plugin's own plugin.json, so keep both in sync.
Write eval cases the way you'd write integration tests: golden path plus known failure modes.
This is a good signal to check before installing a community plugin, and a good habit to build before maintaining more than one or two of your own.

Next Steps

Want to make sure you actually understand Claude Code's plugin, skill, and hook architecture at the depth the Claude Certified Architect (CCA-F) exam expects — not just the happy path? AI for Anything's CCA-F study guides and practice tests are updated as Anthropic ships changes like this one, so you're studying the platform as it exists today, not six months ago. Start with our free AI Work Readiness diagnostic to see where your Claude Code fluency actually stands before you commit study time.

Sources: