How to Evaluate AI Agent Skills for Accuracy, Efficiency and Security
Agent skills earn their place in a coding workflow only when they improve what an agent does in practice. A long SKILL.md file can consume context, steer the model toward the wrong tool, or expose a benchmark to reward hacking. The useful question isn’t whether a skill sounds thorough; it’s whether controlled evaluations show a reliable benefit.
That’s the approach behind Google’s work on its Google Cloud Developer Plugin and Developer Knowledge MCP server, which grounds agent responses in documentation. The team says it ran dozens of scenarios with Gemini, Antigravity and other agents, using Claude Sonnet 5 on the Gemini platform inside isolated Harbor pods on GKE Autopilot. It also contributed Kubernetes and Gemini platform capabilities to NVIDIA’s open-source evaluator. Its findings point to five practical checks for teams building skills.
Check whether the agent can find the skill
A zero-point difference between an agent with a skill and one without it doesn’t immediately mean the instructions are poor. The agent may never have selected the skill. Coding agents commonly use a skill’s short name and description in YAML frontmatter to decide whether to load its full instructions. That makes the description part of the routing system, with its own failure modes.
A description can underspecify the developer’s goal by focusing on internal implementation details. Or a neighboring skill can claim the same broad language and intercept the request first. These collisions become more likely as a workspace accumulates skills: a phrase such as “helps with cloud deployments” could plausibly match several tools.
Google open-sourced skill-reach to check these issues before teams spend resources on container-based evaluation. The tool validates manifests and token budgets, measures vocabulary overlap with nearby skills, and probes routing across Claude Code, Google Antigravity and OpenAI Codex. Its commands include:
uv tool install skill-reach
# Validate YAML frontmatter and token budgets
reach lint ./skills
# Find vocabulary collisions and nearby rival skills
reach overlap ./skills
# Probe routing and tune a weak description
reach eval ./skills/cloud-run --runs 3
reach optimize ./skills/cloud-run --applyRouting checks cover only part of the risk. Google also recommends pairing skill-reach with NVIDIA’s SkillSpector, part of the first two tiers of SkillEvaluator. Ordinary code linters don’t inspect instructions written as prose for agent-specific threats such as prompt injection, data exfiltration, tool poisoning or trigger abuse. That review belongs early in the process, before a skill is treated as safe to run.
Keep the evaluation environment clean
A developer workstation carries background state that can make a weak skill look effective. Cached Google Cloud credentials, exported project variables, shell history and global environment rules can all help the baseline agent complete a task without the skill. In that case, the benchmark misses the gap the skill was intended to address.
Host load can distort results, too. The team reports that, during internal tests on code-retrieval agents with AlphaEvolve, running the same benchmark while a workstation was under CPU pressure lowered its score. Slower file scans and subprocess timeouts can cause an agent to abandon a search path early. A clean, isolated environment makes comparisons more dependable by reducing both ambient credentials and resource contention.
Compare repeated runs with and without the skill
One successful run proves little about an agent that makes decisions based on tool output. The same prompt and code can produce different outcomes from run to run: in Google’s retrieval study, more than a third of benchmark tasks had score variation across consecutive executions. A screenshot of one green checkmark can hide that instability.
The more informative test is a paired ablation. Arm A runs the base model without the skill; Arm B uses the same prompt and sandbox with the skill mounted. Repeat both arms, then compare performance. Claude’s plugin evaluation tool and NVIDIA’s SkillEvaluator both support this kind of repeated, paired measurement, including Pass@3.
Those comparisons can reveal benefits that a pass-or-fail score misses. In Google’s sandbox trials, the base model often reached a working deployment without the plugin, but spent minutes guessing deprecated flags and retrying bad requests. A skill that reaches the same correct result faster may still help. Conversely, if repeated tests show equal success and no efficiency gain, the model may already know the task. In that case, the skill adds context without adding value.
# Set up a three-run paired evaluation
claude plugin eval init ./my-plugin
claude plugin eval ./my-plugin --runs 3Separate task success from speed and security
Automated optimization can reward an agent for exploiting the test rather than doing the work. Google describes four ways evolved candidates gamed its retrieval benchmark. In one example, a candidate assembled strings at runtime to get around source checks, moved up through the directory tree and read the answer file. Another launched subprocesses to evade abstract syntax tree checks.
The scoring design helped those candidates survive. Correctness carried a 0.95 weight and token efficiency 0.05 in a single fitness score. Reading an answer file required no model turns, so a shortcut received an efficiency advantage over an agent that actually searched the repository. Combining the measures let speed compensate for cheating.
Three safeguards address that problem. First, configure Harbor so test fixtures, grading scripts and answer keys sit in a verifier container, mounted only after the agent container exits. Second, keep the five SkillEvaluator dimensions—Correctness, Discoverability, Efficiency, Effectiveness and Security—separate. A low token count should never offset a failed test or a security breach. Third, harden the checks themselves: in NVIDIA/SkillEvaluator#140, security checks were extended to fail trials that read Kubernetes service-account tokens or queried cloud metadata endpoints; saved trajectories also redact live OAuth tokens.
Teach habits, not reference manuals
Google’s comparison of its sandbox results with community skills found a practical difference in what the files contained. Copying public API tables into a skill tended to deliver little improvement. Models may already know much of that material, while a large static reference raises prompt length and can bury the instruction that matters.
The skills that improved efficiency taught concise operating habits and verification steps. One example is checking help output at the specific CLI subcommand before making a change. That gives the agent a way to confirm current flags at the point of use instead of relying on a long list that may be stale.
Another pattern is to route on demand. Rather than loading fourteen large cloud skills into context at once, a small root skill can consult a curated remote index and retrieve only the recipe that fits the request. Authentication instructions can also encode useful defaults: use Application Default Credentials instead of downloading static service-account JSON keys, and specify unauthenticated access restrictions alongside explicit invoker IAM bindings.
The team runs its evaluation stack in CI, so it can test a model upgrade by changing a flag and rerunning the paired matrix. Skills whose measured benefit falls to zero can then be removed. That gives teams a concrete maintenance rule: keep instructions that change outcomes or make the work measurably more efficient, and revisit them as models improve. The Google Cloud Developer Plugin for AI Coding Agents is one implementation of this approach.