Google Agent Skills Adds Automated CI Gates and AI Quality Reviews
Google’s Agent Skills repository has grown from 15,000 to more than 21,000 GitHub stars since the last update. It now contains over 170 skills built by teams across Google and distributed across multiple AI platforms. At that scale, quality depends on more than careful authorship: every change needs checks that can catch problems before they reach users.
That’s the focus here: the automated gates each change must pass, the AI agent that reviews skill quality, and the lessons behind the system. The same checks can be adapted for other skill repositories. For background on how the project began and how Google approaches quality and governance, see the earlier account of how Google Agent Skills is built, tested, and scaled.
Explore the Google Agent Skills repository.
Treat each skill change like a product release
A developer can write a SKILL.md, add a few instructions, and try them by hand. That works for a small project. It breaks down when many people contribute: an unchecked edit can introduce a subtle failure, and manual review alone makes it harder to catch problems before users encounter them.
The important shift is to regard a skill as a product that keeps changing, rather than a document that gets written once. Documentation helps a person understand a subject. A skill shapes an AI agent’s reasoning, choice of tools, and execution. A poorly chosen instruction can send an agent toward an invalid command flag, trap it in a loop, or burn tokens without making progress.
So an instruction edit changes behavior. It deserves the same care as a product change. That means explicit semantic versioning for behavioral updates, automated checks before human or AI review, and evaluation evidence attached to every change. Evaluations compare a revised skill with its previous version; for a new skill, they compare performance with the skill against an agent without it.
Checks before a change ships
The repository’s internal CI/CD pipeline runs as soon as an author submits a change. Before that skill is exported to GitHub, it must pass seven gates and receive approval from a human subject matter expert. The gates cover directory structure and taxonomy; skill frontmatter, folder names, and Markdown; version changes; evaluation suites and fresh reports; link validity; and AI review of quality and design.
Structure and formatting
Automated validation checks each pull request for structural and syntactical errors. Among other things, the folder name must match the metadata name, required frontmatter fields must be present, and limits on SKILL.md and description length must be met. Skill directories can contain only approved files and folders, Markdown files must follow formatting rules, and both the main README.md and machine-readable index.json registry must be current.
Version numbers with a purpose
Each skill declares a semantic version in the frontmatter of its SKILL.md. The version needs to be a quoted string: YAML can parse an unquoted value such as 2.0 as a number, which can break tools that expect a string.
---
name: spanner-basics
metadata:
version: "1.1.1"
category: Databases
description: ...
---New skills start at 1.0.0. Changes to instructions, scripts, or references require a version bump; otherwise, the pull request is blocked. The validator also checks the transition: moving from 1.4.2 to 2.0.0 resets the minor and patch numbers, while moving from 1.4.2 to 1.5.0 resets the patch number.
Why keep a version number if the model only reads the Markdown? It’s there for developers and package tools, not for the model at runtime. Skills don’t update themselves in the background, so invoking one adds no network check or latency overhead. A version lets developers compare an installed skill with the upstream catalog and choose when to update it, for example with npx skills update. Teams can also pin a critical skill to a known-good release and keep agent behavior reproducible across projects.
Evaluation suites need their own checks
Continuous evaluations compare an agent’s performance with and without a skill. Those results are only useful if the evaluation suite is sound. Broken file references or expectations that combine multiple requirements can make results noisy and hard to trust.
The validator reads the evaluation suite and the pull request description, then checks several conditions:
- For file-based prompts such as
@file:test_prompt.txt, the referenced file must exist and contain something. - Each expectation must test one criterion. Compound expectations are blocked until they’re split into separate checks; a later lesson in the article explains why.
- Duplicate prompts and conflicting expectations in the same suite are flagged.
- Every pull request that changes a skill must link to an evaluation report in its description.
- The report must cover the latest commit. Because authors often update a pull request after reviewer feedback, an earlier report can become stale.
Catch broken links before users do
More than 170 skills point to documentation for Google products that change over time. A link can break after a skill has passed review, so checks run at two points. Before a change is merged, pull request checks validate relative links, including links to files in references/. Then, every Monday, a GitHub Actions workflow runs lychee against external links in the public repository and reports failures to the skill owners.
An AI reviewer looks beyond syntax
Linters can catch malformed metadata and Markdown. They can’t tell whether an instruction is confusing, a reference guide is too long, or a task would be better handled by a bash script than by a large language model (LLM) instruction.
Google added an AI review agent to the pipeline to assess those questions. When a change affects a skill, the agent reviews it using a dedicated review skill: a skill reviewing other skills. Its first test is whether the task belongs in an instruction at all.
1. Keep deterministic work out of instructions
One recurring mistake is the “deterministic trap”: authors put rigid, numbered sequences of command-line interface (CLI) commands directly into Markdown instructions. LLMs are probabilistic. When a task requires commands to run in a strict sequence, even a small variation can cause an error—and writing the sequence as prose invites the model to improvise.
The reviewer draws a line between deterministic work and tasks that call for judgment. Strict CLI sequences, JSON parsing, and deployment steps belong in executable scripts under scripts/. Instructions are better suited to work such as choosing an architecture, diagnosing error logs, or synthesizing code.
This fits the Model Context Protocol (MCP) guidance from the earlier post: use remote MCP tools when possible. When a CLI or API call is necessary, put it in a script instead of spelling out the procedure in the instructions.
2. Organize information in three tiers
Context is limited. Loading a 2,000-line skill into an agent consumes context and can weaken its reasoning. The AI reviewer therefore checks for a three-tier information hierarchy based on the progressive disclosure model of Agent Skills.
Keep the instructions close to the work
A skill’s documentation structure affects whether an agent can find and use it. The repository divides that structure into three levels: YAML frontmatter with a name and description helps with discovery and routing; the main SKILL.md contains compact core instructions and stays under 500 lines; and detailed API references, schemas, and tables go in references/.
The reviewer enforces two rules for those reference files. The directory must stay flat, which makes files easier to find and discourages documentation sprawl. References must also stop after one hop: if SKILL.md points to a reference that points somewhere else, agents need another tool call and may never follow it. That extra step adds latency and cost; evaluations found agents often stopped before reaching a second level of references.
Make helper scripts safe to run unattended
Skills may include helper scripts in scripts/, which agents are expected to run in terminal sandboxes. That makes interactive prompts a practical failure point: the reviewer flags scripts that wait for a person to respond.
Scripts should take input through command-line flags or environment variables, explain their options with --help, and offer --dry-run or a confirmation flag before destructive operations. They should also keep machine-readable output—usually JSON—on standard output and send useful error messages to standard error.
Turn review rules into specific checks
The AI review skill checks frontmatter against the specification and translates the L1/L2/L3 design into concrete repository rules. For example, it requires YAML frontmatter to follow the Agent Skills Reference Specification, including folded strip-chomping format (>-) for multi-line descriptions. It also checks the description against the Description Guide.
The same criteria keep the main SKILL.md below 500 lines and direct deeper material to references/, as described in Skill Structure. That directory must remain one level deep. Files longer than 100 lines need a table of contents with line hints. The reviewer also rejects chains such as SKILL.md → references/guides.md → assets/output.tmpl; agents may not load the final target.
Example AI review checklist
### Front Matter & Discovery
- **Spec Compliance**: YAML front matter MUST strictly follow the Agent Skills
Reference Specification (https://agentskills.io/specification). Use `>-`
(folded strip-chomping) format for multi-line descriptions.
- **Description Quality**: The `description` field must follow the
[Description Guide](references/description_guide.md).
### Content Structure & Authoring Quality
- **L1/L2/L3 Architecture**: Enforce modular structure. The main `SKILL.md`
body must stay under 500 lines. Move deep domain reference material to
`references/` (see [Skill Structure](references/skill_structure.md)).
- **Flat References Directory**: `references/` must be a flat directory (one
level deep, no nested subdirectories). References > 100 lines must include a
Table of Contents with line hints (see
[Skill Structure](references/skill_structure.md)).
- **Avoid Deeply-Nested References**: Avoid deeply-nested references (e.g.,
`SKILL.md` -> `references/guides.md` -> `assets/output.tmpl`). Keep
reference structures shallow (one level of indirection from `SKILL.md`)
because agents might not load such link targets fully.Keep a human in the review loop
Automated checks are only one part of approval. Every pull request also needs a human reviewer, usually someone with subject matter expertise. For a new skill, that reviewer is a member of the Central Skills team.
What repository history taught us
Six months of commits and pull request discussions exposed a few ways well-intended process can go wrong. The lessons are useful well beyond skills repositories.
Scope CI rules to the change
Early pull request checks used directory patterns that were too broad. One rule demanded an approved bug-tracking ticket for every change, so even a one-line typo fix could be blocked. At the same time, newly added skills in deeper directories slipped past inspection.
CI rules need to distinguish the operation being performed—adding a file versus modifying one—and match the intended path depth. A small edit to an existing skill shouldn’t automatically face the same checks as a new skill.
Score one expectation at a time
Compound evaluation criteria made results hard to trust. Consider asking whether an agent created a Cloud Storage bucket in us-central1, enabled uniform bucket-level access, and added lifecycle rules. If it gets two of the three right, an LLM judge can rate the result inconsistently.
Splitting that request into three checks makes the result easier to interpret:
# Bad: Compound expectation
- "The agent must create a Cloud Storage bucket in us-central1, enable
uniform bucket-level access, and apply lifecycle rules."
# Good: Atomic expectations
- "The agent specifies us-central1 for the bucket location."
- "The agent enables uniform bucket-level access on the created bucket."
- "The agent defines a lifecycle configuration rule deleting objects
older than 30 days."Each expectation should test one criterion. That makes scores clearer, more consistent, and easier to debug.
Generate catalogs instead of maintaining them by hand
In a repository with many contributors, manually maintained documentation tables can drift within days. If you publish an index or catalog, generate it with a script and enforce the output with a CI unit test. That’s why the linters check README.md and index.json; the JSON file begins with the warning, “This file is generated. Do not edit it by hand.”
Run cheap checks before expensive ones
Full evaluations and AI review can consume significant tokens and CI time, especially when authors push many small commits. The pipeline described here runs quick, deterministic checks first: structural conformance, skill linters, and semantic versioning. Any failure stops the run before AI review or evaluation begins.
If you’re building a similar pipeline, consider additional controls for costly checks. Heavy evaluations could require a /run-evals pull request comment, or automatic reruns could depend on the size of the diff.
Treat skills as maintained software
The larger point is that good prompts don’t make a scalable skills repository on their own. Skills need the same working infrastructure as other software: versioning, validation, evaluation, and clear review ownership. Semantic versioning and atomic expectations belong in automated CI gates. An AI reviewer can catch design problems such as the deterministic trap, while link checks should run both before merge and on a schedule.
The Google Agent Skills repository and related reading are available here:
- Google Agent Skills repo: github.com/google/skills
- Previous post: Behind the scenes: How we build, test, and scale Google Agent Skills
- Launch announcement: Level Up Your Agents: Announcing Google's Official Skills Repository
- Skills tutorial: Introduction: What Are Google Cloud Agent Skills?
If this was useful, share it with your peers, or connect with the author on LinkedIn, X, or Bluesky.
Thanks for reading.