Noodle Agent Orchestration Framework

repository·main·Indexed 42 days ago

https://github.com/poteto/noodle

An agent orchestration framework that manages backlogs and tasks through a continuous schedule-execute-commit loop. Noodle uses a file-based API for state persistence in a `.noodle/` directory and extends functionality via Markdown-based 'Skills'. It ensures code isolation by running agents in dedicated Git worktrees and supports multiple execution modes (auto, supervised, manual) to control human oversight.

Tokens
79.8K
Snippets
209
Records
431
Agent score
59%

What's inside Noodle

  1. How events work in Noodle

    main

    Events are the mechanism Noodle uses to track history and ingest information from external systems. The scheduler uses these events as context (not as direct commands) to decide how to react to the state of the world.

    Noodle emits internal events automatically (e.g., stage completion, merge conflicts) and allows you to inject custom events from external tools like CI pipelines, deployment scripts, or monitoring systems.

    When you emit an event, it is appended to .noodle/loop-events.ndjson. During the next build, these events appear in the recent_events array of the mise, which the scheduler reads to inform its decision-making process.

  2. How reflect processes structured findings

    main

    The reflect component uses structured findings to automate the lifecycle of learnings in the brain. When a tagged envelope is detected, it follows these routing logic patterns:

    1. Principle Violations: If a finding includes a principle reference, reflect checks brain/ for existing notes.
      • If a match is found (via check name, file path, or principle ref), it strengthens the existing note.
      • If no match is found, it creates a new targeted note using the provided evidence.
    2. Structural Enforcement: If a finding includes a file_path and a check, reflect checks if structural enforcement (like a lint rule or test) already exists. If not, it proposes adding one.
    3. Escalation: If a finding has severity: high and matches an existing brain note, it is escalated as a principle candidate for meditate.

    If no tagged structured findings are present, reflect falls back to its existing narrative scanning behavior.

  3. How `noodle worktree prune` determines safety

    main

    The noodle worktree prune command performs auto-cleaning of worktrees using a safety rule based on patch equivalence rather than just commit ancestry. This ensures that worktrees are only removed if their content is already safely represented on the integration branch.

    Safety is determined using git cherry <base> <branch> logic:

    • Safe to prune: If there are no + lines, every commit on the worktree branch is already represented on the integration branch (defaults to main).
    • Safe to prune: If there are only - lines, the commits are patch-equivalent (e.g., they were landed via cherry-pick or rebase with different hashes).
    • Unsafe to prune: If + lines exist, the worktree contains unique changes not yet on the integration branch and will be skipped.

    Additional Requirements:

    • Clean Worktree: The worktree must be clean (git status --porcelain must be empty). Dirty worktrees are skipped even if they are patch-equivalent.
    • Filesystem Cleanup: The command also removes stale directories in .worktrees/ that lack git metadata as part of general filesystem cleanup.
  4. Manage ingestion lifecycles with IngestionHandle and IngestionRegistry

    main

    The ingestion system uses two primary data structures to manage concurrency and state:

    • IngestionHandle: Used to control a specific ingestion session. It contains:

      • SessionID: Unique identifier for the session.
      • Cancel: A function to trigger session cancellation.
      • Done: A signal indicating the session has completed.
      • Sources: The list of active sources in the session.
    • IngestionRegistry: A central registry keyed by the parent session ID. It tracks active workers, checkpoints, and poller metrics to ensure reliability across restarts.

  5. How the Skill Refinement loop works

    main

    Skill refinement is a hill-climbing process used to improve skills through measured experimentation. It is triggered when a reviewer flags a skill for improvement and evals.json criteria exist.

    The Refinement Process:

    1. Isolate: Copy the target skill directory to a working location. Record a manifest_hash of all files. Only SKILL.md is mutable; references/ and scripts/ are read-only.
    2. Baseline: Run the skill against test prompts 5 times. Record the baseline pass rate.
    3. Analyze: Identify the most common failure pattern from execution traces.
    4. Hypothesize + Mutate: Propose and apply a single targeted change to SKILL.md (preferring subtraction over addition).
    5. Test + Score: Re-run tests 5 times. Accept the mutation only if the pass rate improves by $\ge 5%$.
    6. Log: Record the experiment in <skill>-refinement-changelog.md.
    7. Repeat: Continue until a budget cap (default: 5 experiments) is reached, a 95%+ pass rate is achieved for 2 consecutive runs, or no improvement occurs after 2 runs.
    8. Promote: The user reviews the changelog and diff. The system verifies the manifest_hash matches the original directory before promotion.
  6. Design idempotent operations for state convergence

    main

    When designing state-mutating operations (such as CLI commands, lifecycle operations, or scheduling loops), ensure they are idempotent. An idempotent operation is designed so that it converges to the correct state regardless of how many times it runs or the state it starts from.

    The Idempotency Test

    Before shipping an operation, verify it against these three questions:

    1. What happens if this runs twice in a row?
    2. What happens if the previous run crashed at every possible point?
    3. Does re-execution converge to the same end state?

    If the outcome depends on the leftover state from a previous run, the operation requires a reconciliation step to ensure convergence.

    Implementation Patterns

    • Convergent Startup: Scan existing environments (e.g., tmux sessions), clean stale state, and adopt live sessions to reach the desired state regardless of previous runs.
    • Content-based Cleanup: Use content/patch equivalence (e.g., git cherry) rather than ancestry to determine safety, ensuring re-runs converge to the same state.
    • Self-healing Locks: Implement lockfile merging with PID-based stale lock detection to automatically recover from orphaned locks caused by crashed processes.
    • Idempotent Scheduling: Ensure failed tasks respawn with recovery identifiers and regenerate inputs (e.g., mise regeneration) after each run to prevent stale state accumulation.
  7. Define Schedulable Skills using the schedule field

    main

    Skills can be registered as part of a pipeline by including a schedule field in their YAML metadata. This tells the scheduler when the skill is eligible to run within an order's lifecycle.

    Common scheduling patterns:

    • schedule: "After execute stages complete, to verify changes": Used for testing skills to run after implementation.
    • schedule: "After test stages pass, to ship verified changes": Used for deployment skills to run after verification.

    Example test skill metadata (.agents/skills/test/SKILL.md):

    ---
    name: test
    description: Runs the project test suite and reports results.
    schedule: "After execute stages complete, to verify changes"
    ---
  8. Advanced Skill Patterns: reflect and meditate

    main

    To build a self-improving system, you can implement specialized skills that interact with the 'brain' (the markdown-based knowledge base):

    • reflect: An agent writes what it learned during a session to the brain. This allows subsequent agents to read these notes and avoid repeating mistakes.
    • meditate: An agent analyzes the accumulated learnings in the brain to extract high-level principles. These principles can then be encoded back into your skills to ground future agent decisions.
  9. Behavior of active_order_ids in snapshots

    main

    The active_order_ids field in snapshots no longer relies on inference from live cooks. Instead, it follows the state defined by the projection:

    1. Inclusion: Review-blocked or non-running orders will remain in active_order_ids if the projection marks them as active.
    2. Runtime Enrichment: Snapshot enrichment is restricted to session-level fields (e.g., active/recent sessions, current action, cost, and context usage).
  10. How the self-learning loop works with Reflect, Meditate, and Ruminate

    main

    The brainmaxxing skill pack introduces three core skills that create a continuous learning loop for agents:

    1. Reflect: Runs after every session. It reviews the session and encodes learnings into the brain.
    2. Meditate: Runs periodically. It distills accumulated reflections into high-level project principles.
    3. Ruminate: Run ad-hoc. It mines past conversations to extract and capture knowledge that was never explicitly written down.

    Agents use these to treat the brain as shared memory: reading context before work, writing notes during work, and using reflect to encode learnings after work.

  11. Codex Child Path Resolution Rules

    main

    When ingesting Codex child sessions, the system resolves file paths using the following rules:

    1. Primary Lookup: Files are searched by thread ID filename under a date-partitioned tree: ~/.codex/sessions/YYYY/MM/DD/{thread_id}.jsonl.
    2. Search Window: The system checks today, yesterday, and tomorrow partitions to handle midnight rollovers.
    3. Retry Logic: If a file is unresolved, the system retries with exponential backoff (from 250ms to 5s) for a maximum of 90s before emitting a timeout warning.
    4. Discovery Order: Exact partition match $\rightarrow$ adjacent partitions $\rightarrow$ bounded glob fallback within the 3-day window.
  12. Understand the Projection-Backed Snapshot Architecture

    main

    In the canonical state convergence model, the system moves away from rebuilding topology from LoopState. Instead, internal/projection/projection.go becomes the sole owner of structural snapshot topology (including projected review state and projected order/state output).

    Key architectural shifts:

    • projection.SnapshotView: This is the sole structural snapshot model consumed by the server. It is enriched at runtime.
    • internal/snapshot/snapshot_builder.go: This component is reduced to performing runtime enrichment over the projection.SnapshotView rather than rebuilding the topology from scratch.
    • Data Authority: Files like orders.json and state.json transition from being mutated by legacy workflows to being pure projection outputs.