SkillOpt: Agentic Skill Optimization via Reflective Training Loops

SkillOpt-Sleep is a deployment-time companion (currently in preview) designed to give local coding agents a nightly 'sleep cycle'. It automates the process of reviewing past agent sessions, replaying recurring tasks using your API budget, and consolidating learnings into validated long-term memory and skills.

Key characteristics:

• No weight training required: It does not require model fine-tuning.
• Zero dependency: The skillopt_sleep/ engine is independent of the core skillopt/ package.
• Review-driven: Improvements are staged for your review behind a validation gate before being adopted.
• Workflow: It harvests transcripts (e.g., from Claude Code, Cursor, VS Code Copilot) $\rightarrow$ mines recurring tasks $\rightarrow$ replays tasks via a configured backend $\rightarrow$ consolidates via reflection and bounded edits $\rightarrow$ gates on held-out tasks $\rightarrow$ stages a proposal for user adoption.

SkillOpt-Sleep is a deployment-time companion that allows a Codex agent to undergo a 'sleep cycle' to self-improve. It reviews past local sessions, mines recurring tasks, re-runs them through a selected backend, and proposes improvements to skills (SKILL.md) and project memory (CLAUDE.md).

Key Features:

• Validation Gating: By default, changes are only accepted if they improve a held-out validation score.
• Staging Workflow: Proposed changes are written to <project>/.skillopt-sleep/staging/<date>/ and do not affect live files until explicitly adopted.
• No Model Training: It optimizes via skill and memory edits, not model-weight training.

Important Note on Targets:

• The engine does not write AGENTS.md.
• To target a specific Codex skill, you must use the --target-skill-path flag (e.g., .agents/skills/<name>/SKILL.md).
• To prevent updates to CLAUDE.md, set "evolve_memory": false in ~/.skillopt-sleep/config.json.

bash
# Example: Running a dry-run to see proposed improvements
export SKILLOPT_SLEEP_REPO=/path/to/SkillOpt
TARGET_SKILL=.agents/skills/example/SKILL.md
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" dry-run --project "$(pwd)" \
  --source codex --target-skill-path "$TARGET_SKILL" --backend mock

What is SkillOpt Sleep?

skillopt-sleep is a Claude Code plugin designed to provide a nightly sleep cycle for local Claude agents. It enables an agent to evolve its capabilities without model-weight training by consolidating short-term experiences into long-term memory and skills.

Core Workflow

1. Harvest: Extracts the day's Claude Code session transcripts from local storage (~/.claude).
2. Mine: Converts transcripts into discrete task records with verifiable outcomes.
3. Replay / "Dream": Re-runs those tasks offline using the user's existing API budget to simulate performance.
4. Optimize: Runs the SkillOpt optimizer loop (reflect $\rightarrow$ bounded edit $\rightarrow$ held-out gate) to update text-based knowledge.
5. Validate & Stage: Only changes that pass a validation gate are moved to a review staging area. The user must manually approve changes before they are adopted into live configuration files.

Key Benefits

• Weight-free evolution: Improves performance via text-based updates to CLAUDE.md and SKILL.md rather than fine-tuning.
• Regression protection: Uses a validation gate to ensure nightly updates actually improve performance on held-out tasks.
• Safety: Follows a "stage $\rightarrow$ approve $\rightarrow$ adopt" model, ensuring user configuration is never silently overwritten.

In the SkillOpt system, the meta_skill.md prompt defines the role of an optimizer-coach. The coach's purpose is to generate high-signal, optimizer-side memory that guides future optimizer calls in producing better skill edits.

Input Data

The coach receives four specific pieces of context:

1. The previous epoch's last-step skill.
2. The current epoch's last-step skill.
3. A longitudinal comparison of the same sampled tasks performed under both skills.
4. The previous optimizer meta skill (if available).

Objective

The goal is to produce a concise meta-skill that improves future optimizer behavior in specific stages: failure analysis, success analysis, patch merging, and edit ranking. The meta-skill should capture environment-specific patterns such as:

• Effective vs. ineffective edit types.
• Optimal levels of abstraction for rules.
• Prioritized failure-repair patterns.
• Regression risks to guard against.

Output Format

The coach must respond ONLY with a valid JSON object containing two keys:

• reasoning: A brief reflection on which editing directions helped or hurt.
• meta_skill_content: Compact, optimizer-side guidance for future edit generation and selection.

Critical Constraints

• Target Audience: Address the FUTURE OPTIMIZER, not the target agent.
• Content Focus: Focus on editing strategies and skill updates, not task instructions.
• Evidence-Based: Use evidence from the epoch comparison rather than generic advice.
• Compactness: Prefer a few durable principles over verbose text.

{
  "reasoning": "<brief reflection on what editing directions helped or hurt>",
  "meta_skill_content": "<compact optimizer-side guidance for future edit generation and selection>"
}

When using SkillOpt-Sleep, it is critical to understand the data boundaries, especially when using real backends instead of the mock backend.

General Security Warning

Harvesting is local and read-only. However, a real backend sends truncated excerpts from harvested sessions and derived tasks to your selected model provider for mining, replay, judging, and reflection. Outbound prompts are not currently guaranteed to be secret-free.

Recommended Workflow for Sensitive Projects:

1. Harvest to a task file.
2. Inspect and redact the file manually.
3. Mark the file with "reviewed": true.
4. Replay that file using a real backend.

Source-Specific Data Handling

• Cursor: Reads local user/assistant message text, explicit turn errors, and tool names. It excludes tool arguments, tool outputs, and non-message records. It uses cursor-agent for replays in an empty temporary workspace with project files denied.
• VS Code Copilot: Reads local user-entered prompts, visible assistant Markdown, and tool names. It excludes system notifications, reasoning, tool inputs/outputs, rendered context, and account/model metadata.
• Pi: Reads sessions from ~/.pi/agent/sessions. It retains user/assistant text, tool names, and lexical feedback. It excludes thinking, tool arguments, tool outputs, and images. The absolute project cwd from the session header is retained and may be sent to providers.

Evidence Logging

By default, each stateful night writes a local evidence.jsonl file under the project staging tree (or the Sleep state directory during dry-runs). This log contains best-effort-redacted, per-field-truncated copies of prompts and replies from the miner, replay, judge, and reflection stages.

Configuration Options:

• To disable evidence logging: Set "evidence_log": false.
• To disable secret redaction (not recommended): Set "redact_secrets": false.

SkillOpt is architected to mirror the structure of training neural networks, where natural-language prompts are optimized instead of model weights. Understanding this mapping allows machine learning practitioners to apply familiar tuning intuitions to SkillOpt's hyperparameters and lifecycle.

Core Mapping

The ALFWorld skill supports several distinct task patterns. Agents should follow these specific sequences:

Pick & Place

Goal: Put object X in/on receptacle Y. Workflow: Find X $\rightarrow$ take X $\rightarrow$ go to Y $\rightarrow$ put X in/on Y.

Pick Two & Place

Goal: Put two instances of X in/on Y. Workflow: Find X1 $\rightarrow$ take $\rightarrow$ place $\rightarrow$ find X2 $\rightarrow$ take $\rightarrow$ place. Note: Choose one destination receptacle instance once it is opened/usable and remember it as the target. Both objects must go into that same instance. If objects are split across different receptacles, consolidate them into the chosen target.

Examine in Light

Goal: Examine object X under a desklamp. Workflow: Find X $\rightarrow$ take X $\rightarrow$ find desklamp $\rightarrow$ use desklamp. Constraint: While holding X where a desklamp is visible, use the desklamp directly; do not attempt to place X on the lamp first.

Transformation Tasks

• Clean & Place: Find X $\rightarrow$ take X $\rightarrow$ go to sink $\rightarrow$ clean X $\rightarrow$ go to Y $\rightarrow$ put X.
• Heat & Place: Find X $\rightarrow$ take X $\rightarrow$ go to microwave $\rightarrow$ heat X $\rightarrow$ go to Y $\rightarrow$ put X.
• Cool & Place: Find X $\rightarrow$ take X $\rightarrow$ go to fridge $\rightarrow$ cool X $\rightarrow$ go to Y $\rightarrow$ put X.