DOCSEARCHbyAlgolia
Submit Docs

aislop

repository·main·Indexed 17 days ago

https://github.com/scanaislop/aislop

A deterministic code scanner designed to detect 'slop'—low-quality patterns left by AI coding agents such as narrative comments, swallowed exceptions, and dead code. It supports 50+ rules across TypeScript, JavaScript, Expo/React Native, Python, Go, Rust, Ruby, and PHP. The tool provides a codebase score from 0–100 and includes a CLI, a VS Code extension, CI integration, and an MCP server for use with AI agents.

Tokens
55.3K
Snippets
185
Records
245
Agent score
66%

What's inside aislop

  1. Safety rails and sandbox constraints for automated fixes

    main

    To ensure security and prevent regressions, the automated fix flow operates under strict constraints:

    • Ephemeral Sandbox: Jobs run in network-restricted containers. Only the cloned repo is writable.
    • Command Allowlist: The loop is restricted to running git, aislop, and the project's package-manager commands (install/build/test) resolved from the repo manifest. LLM output is applied as patches, not executed as arbitrary shell commands.
    • Budget Caps: Hard limits are enforced on wall-clock time, iteration count $N$, and total LLM token spend.
    • Score-Gated Writes: No code is committed unless the deterministic score improves. This prevents the LLM from bypassing rules by deleting code or modifying configuration.
    • Read-Only Config: .aislop/config.yaml and .aislop/rules.yaml are treated as read-only inputs to prevent the job from gaming the score.

  2. Research Track: Agent Output Benchmark

    main

    Running the same tasks across various AI coding agents and scoring the produced repositories with aislop. This answers which agents produce the least maintainability debt.

    Minimum required output:

    • Task prompt
    • Agent and version
    • Clean-room run notes
    • aislop score and rule distribution
    • Qualitative code review notes (only after the deterministic score is provided)

  4. Understand how the aislop score is calculated

    main

    aislop produces a single score from 0 to 100 for every scan. The score is calculated by aggregating weighted penalties from every diagnostic, which are then normalized using logarithmic scaling relative to the number of production and test files (density normalization). This ensures the score remains meaningful regardless of project size.

    Each diagnostic contributes a penalty based on its Severity:

    • Error: 3.0 base penalty
    • Warning: 1.0 base penalty
    • Info: 0.25 base penalty

    Total penalty for a diagnostic = (Base Penalty) * (Engine Weight) * (Rule Impact Tier Multiplier).

  5. Understand the six aislop detection engines

    main

    aislop uses six deterministic engines to score code quality without using LLMs:

    1. Formatting: Checks code style consistency using tools like Biome, ruff, gofmt, cargo fmt, rubocop, and php-cs-fixer.
    2. Linting: Language-specific issue detection using oxlint, ruff, golangci-lint, clippy, and expo-doctor.
    3. Code Quality: Detects complexity and dead code (function/file size limits, deep nesting, unused files/deps via knip, and AST-based unused-declaration removal).
    4. AI Slop: Specifically targets AI-authored patterns like narrative/trivial comments, dead patterns, unused imports, as any, console.log leftovers, TODO stubs, and generic names.
    5. Security: Identifies vulnerabilities and risky code (eval, innerHTML, SQL/shell injection, and dependency audits via npm/pip/cargo/govulncheck).
    6. Architecture: (Opt-in) Enforces structural rules like custom import bans and layering rules.

  6. Research Track: Benchmark-to-Rule Translation

    main

    Translating academic or industry benchmarks into scanner-shaped rules. The model for this is the SlopCodeBench-derived Python rules.

    Workflow:

    1. Identify a deterministic pattern.
    2. Avoid judge-only scoring.
    3. Write positive and negative fixtures.
    4. Document rule provenance.

  7. Research Track: GitHub Trending Quality Sweep

    main

    A monthly scan of trending open-source repositories by language. The goal is to identify noisy rules in real ecosystems before they reach users in CI gates.

    Minimum required output:

    • Cohort list and commit SHAs
    • Top finding classes
    • False-positive fixes
    • Regression tests added

  8. How the `aislop agent` repair loop works

    main

    The aislop agent is a local-first repair loop. By default, it creates an isolated git worktree to prevent polluting your main branch. It runs safe deterministic fixes, then streams a headless provider session (like Claude or Codex) to attempt repairs. It verifies the results with aislop scan --json and writes a session transcript for audit. You can review the worktree and manually apply the diff or push it as a PR.

    # Preview the plan without running it
aislop agent plan

# Run the agent in the current worktree instead of an isolated one
aislop agent --in-place

# Apply a reviewed worktree session later
aislop agent apply <session_id>

  9. Understand the six engine architecture of aislop

    main

    The aislop engine groups all checks into six distinct engines that run in parallel to ensure high performance. When you run a scan, these engines execute concurrently to identify issues across different categories:

    1. Formatting: Enforces consistent code style using language-specific tools (e.g., Biome for TS/JS, ruff for Python).
    2. Linting: Catches bugs and bad practices using standard linters (e.g., oxlint, ruff, clippy).
    3. Code Quality: Measures structural complexity, detects dead code, and finds unused dependencies (e.g., knip, complexity rules).
    4. AI Slop: A unique engine that detects patterns typically left behind by AI assistants (e.g., trivial comments, swallowed exceptions, generic naming).
    5. Security: Identifies secrets, risky constructs (like eval()), and vulnerable dependencies.
    6. Architecture (opt-in): Allows for custom import and path rules defined by the user.

  10. How the fix and scoring loop works

    main

    The automated fix process follows a deterministic "AlphaGo for code quality" loop. It uses a strong generator (LLM) paired with a cheap, deterministic evaluator (the aislop scoring engine). The engine acts as a gate: changes are only kept if they result in a higher score than the previous iteration.

    The Loop Logic:

    1. Clone the repo at the target ref.
    2. Scan to establish a baseline score ($S_0$).
    3. Iterate up to $N$ times:
      • Run aislop fix (deterministic, free mechanical fixers).
      • If findings remain, an LLM proposes edits (bounded, allowlisted patches).
      • Re-scan to get the new score ($S_i$).
      • If $S_i > S_{i-1}$: Keep the changes (commit).
      • Else: Revert the iteration's edits.
    4. Stop when findings are gone, no score gain occurs for $K$ iterations, or the budget is hit.
    5. Final Gate: If the final score $S_f > S_0$, open a PR. Otherwise, hand off with a summary but no PR.

  11. Integrate aislop with pre-commit

    main

    You can use aislop to scan only staged files during a pre-commit hook.

    To run it directly via CLI:

    aislop scan --staged

    To integrate it into the pre-commit framework, add the following to your .pre-commit-config.yaml:

    repos:
  - repo: https://github.com/scanaislop/aislop
    rev: v1
    hooks:
      - id: aislop

  12. Initialize a new .aislop configuration

    main
    To create a new configuration file for your project, run the aislop init command. This will generate a .aislop/config.yml file populated with the default settings for engines, quality thresholds, scoring weights, and CI integration.
    aislop init