DOCSEARCHbyAlgolia
Submit Docs

skill-validator

repository·main·Indexed 41 days ago

https://github.com/agent-ecosystem/skill-validator

A tool for validating and scoring Agent Skill packages. It provides structural validation, link resolution checks, content quality analysis, and LLM-as-judge scoring for dimensions like clarity and actionability. It includes a CLI, a Go library, and integrations for GitHub Actions and pre-commit hooks.

Tokens
17.5K
Snippets
66
Records
81
Agent score
50%

What's inside skill-validator

  1. Overview of skill-validator commands

    main

    The skill-validator CLI provides commands mapped to different stages of the skill development lifecycle:

    StageCommandPurpose
    Scaffoldingvalidate structureChecks spec compliance (frontmatter, tokens, code fences, internal links, orphan files).
    Writing contentanalyze contentMeasures instruction quality (density, specificity, imperative ratio).
    Adding examplesanalyze contaminationDetects cross-language contamination in code examples.
    Reviewvalidate linksVerifies external (HTTP/HTTPS) links resolve.
    Quality scoringscore evaluateUses an LLM to judge quality (clarity, actionability, novelty, etc.).
    Comparing modelsscore reportCompares cached LLM scores across different providers/models.
    Pre-publishcheckRuns all checks except LLM scoring.

  2. Validate multi-skill directories

    main

    If you point skill-validator at a directory that does not contain a SKILL.md but contains subdirectories that do, the validator automatically detects and validates each subdirectory as an independent skill. Symlinks are followed during detection.

    Exit Codes:

    • If no SKILL.md is found at the root or in any immediate subdirectory, the validator exits with code 3 (CLI error).
    # Validates all skills inside the 'skills/' directory
skill-validator check skills/

  3. Perform Cross-Model Comparison

    main

    If CROSS_MODEL=true is set, you can compare scores from two different model families (Anthropic and OpenAI) to validate novelty and spot scoring bias. This is highly recommended for assessing if a skill provides genuine new information or just restates common knowledge.

    1. Run Primary Scoring: (e.g., using Anthropic).
    2. Run Secondary Scoring: Use the other provider (e.g., OpenAI).
    3. Generate Comparison Report: Use the --compare flag.

    Interpreting Results:

    • Novelty Validation: If both models rate novelty below 3, the skill likely restates common knowledge.
    • Bias Detection: If scores diverge by more than 1 point on any dimension, the content structure may favor one model family.
    • Confidence: Agreement between models increases assessment reliability.
    # Generate the side-by-side comparison report
skill-validator score report <path> --compare -o json

  4. Interpret LLM Scoring Results

    main

    LLM scores are used to assess the quality of a skill. While there are no hard pass/fail gates, use these thresholds for guidance:

    • Overall >= 3.5: Good quality across most dimensions.
    • Overall 2.5-3.5: Needs work in specific areas.
    • Overall < 2.5: Requires significant revision.
    • Any dimension <= 2: Critical area needing attention.

    Novelty and novel_info

    Novelty is the most critical metric. A low novelty score suggests the skill restates information the model already knows.

    Each scored file contains a novel_info field. This field lists the specific content the LLM identified as genuinely novel. Authors should use this to:

    1. Verify: Ensure the LLM correctly identified the high-value content.
    2. Trim: Content NOT mentioned in novel_info is likely common knowledge and should be compressed or removed to save context window costs.

  5. Interpret skill validator scoring results

    main

    The skill-validator uses a 1-5 scale to score various dimensions of a skill. The overall score is calculated as the mean of all individual dimension scores.

    Score Scale

    ScoreMeaningDescription
    5ExcellentGenuinely outstanding on this dimension
    4GoodMinor improvements possible but solid
    3AdequateFunctional but has clear room for improvement
    2Needs workNotable issues that should be addressed
    1PoorFundamental problems on this dimension

  6. Understand the stability and versioning of skill-validator packages

    main

    The project follows semantic versioning. Most core packages are considered Stable, meaning breaking changes only occur in major releases. The judge package is currently Experimental; its API may change in minor releases without a major version bump.

    Stable packages:

    • orchestrate
    • evaluate
    • structure
    • content
    • contamination
    • links
    • skill
    • skillcheck
    • report
    • types

    Experimental packages:

    • judge (API subject to change in minor releases)

    Deprecation Policy: Deprecated symbols are marked with // Deprecated: comments. They are maintained for at least one minor release after a replacement is introduced to allow for migration.

  7. Content review criteria for Agent Skills

    main

    Beyond structural validity, an Agent Skill should be evaluated against the following content quality dimensions:

    • Examples: Does the skill provide concrete examples of expected inputs and outputs?
    • Edge cases: Does the skill document common edge cases or failure modes?
    • Scope-gating: Does the skill clearly define when to stop/continue, prerequisites, and conditions for branching paths?

  8. Improve skill quality based on dimension scores

    main

    When a dimension receives a low score, use the following guidance to advise the skill author on specific fixes:

    Clarity (Applies to SKILL.md and references)

    Issue: Ambiguous or confusing instructions. Fix: Rewrite ambiguous sections to have exactly one interpretation. Declare all dependencies (tools, runtimes, permissions) explicitly. Use consistent formatting and logical ordering.

    Actionability (Applies to SKILL.md only)

    Issue: Instructions cannot be followed step-by-step. Fix: Convert abstract guidance into numbered steps. Add concrete examples and fill in gaps where an agent might need to guess the next step.

    Token Efficiency (Applies to SKILL.md and references)

    Issue: Content is bloated relative to instructional value. Fix: Cut redundant sections and boilerplate. Replace verbose phrasing with concise directives. Ensure every sentence provides necessary information.

    Scope Discipline (Applies to SKILL.md only)

    Issue: The skill covers too many topics or languages. Fix: Split broad skills into focused ones. Remove content that does not directly serve the stated purpose.

    Directive Precision (Applies to SKILL.md only)

    Issue: The skill uses hedging language instead of direct commands. Fix: Replace words like consider, may, or could with precise directives like must, always, never, or ensure. Add explicit conditional gates for branching paths.

    Novelty (Applies to SKILL.md and references)

    Issue: Content mostly restates information the model already knows (e.g., official docs, standard patterns). Warning: A score below 3 is a critical sign that the skill may not justify its context window cost. Fix: Focus on proprietary or non-obvious information: internal API conventions, organization-specific workflows, undocumented gotchas, or domain knowledge not available in public docs.

    Instructional Value (Applies to references only)

    Issue: References are abstract rather than practical. Fix: Add concrete, copy-pasteable code examples and actual API signatures.

    Skill Relevance (Applies to references only)

    Issue: Reference content is unrelated to the parent skill. Fix: Curate reference files tightly. Remove sections an agent would never need for the specific task.

  10. Run LLM Scoring with skill-validator

    main

    Use the score evaluate command to perform LLM-based scoring. If you have already run scoring and want to refresh the results, use the --rescore flag. Use --full-content to ensure the model sees the entire file and --display files to see per-file breakdowns.

    Anthropic (Default)

    skill-validator score evaluate <path> --full-content --display files -o json

    OpenAI

    skill-validator score evaluate <path> --provider openai --full-content --display files -o json

    Claude CLI

    skill-validator score evaluate <path> --provider claude-cli --full-content --display files -o json

    OpenAI-compatible

    skill-validator score evaluate <path> --provider openai --base-url <url> --model <model> --full-content --display files -o json
    # Example: Rescoring with OpenAI
skill-validator score evaluate <path> --provider openai --full-content --display files -o json --rescore

  11. Emit GitHub Actions annotations

    main

    The --emit-annotations flag allows skill-validator to surface errors and warnings as inline annotations in the GitHub PR diff view.

    • It works with all output formats (text, json, markdown).
    • It works for both single-skill and multi-skill directories.
    • Annotations are appended after the normal output.
    • Only error and warning levels are emitted as annotations; pass and info levels are skipped.
    • File paths in annotations are relative to the working directory (the repository root in CI).
    - name: Validate skills
  run: |
    skill-validator check --emit-annotations my-skill/

  12. Validate skill structure with `validate structure`

    main

    The validate structure command ensures your skill conforms to the Agent Skills specification. It performs several critical checks:

    • Directory & File Structure: Verifies SKILL.md exists and that only recognized directories (scripts/, references/, assets/) are used. It flags extraneous files (like README.md or LICENSE) that might unnecessarily consume an agent's context window.
    • Frontmatter: Validates required fields (name, description) and checks that optional fields (license, compatibility, metadata, allowed-tools) conform to expected types.
    • Token Limits: Reports token counts (using o200k_base). It warns if SKILL.md exceeds 5,000 tokens or 500 lines, and enforces limits on reference and asset files.
    • Markdown & Links: Checks for unclosed code fences in SKILL.md and reference files, and validates that internal relative links are resolvable.
    • Orphan File Detection: Uses a reachability graph to ensure files in scripts/, references/, and assets/ are actually mentioned (directly or transitively) in SKILL.md. Unreferenced files are flagged as potential orphans.
    • Keyword Stuffing: Detects descriptions that rely on excessive quoted strings or long comma-separated lists instead of concise prose.
    skill-validator validate structure my-skill/