Osmedeus Documentation

repository·main·Indexed 27 days ago

https://github.com/j3ssie/osmedeus

A declarative orchestration engine for security automation that allows users to define complex workflows in YAML. Osmedeus supports execution locally, via Docker, SSH, or distributed cloud infrastructure (AWS, GCP, DigitalOcean, Linode, Azure). It includes a Go library SDK (v5) for programmatic workflow execution, JavaScript expression evaluation, and workflow validation.

Tokens
77.8K
Snippets
192
Records
413
Agent score
91%

What's inside Osmedeus

  1. Understand the Osmedeus Project Structure

    main

    The Osmedeus repository is organized into several key directories that define its functionality:

    • cmd/osmedeus/: The main application entry point.
    • internal/: Contains private packages including core logic (core), execution engine (executor), database management (database), cloud provisioning (cloud), and workflow linting (linter).
    • pkg/: Contains public packages for the cli (Cobra-based) and the server (Fiber-based REST API).
    • public/: Holds public assets such as examples and presets.
    • test/: Contains E2E, integration, and test data.
    • lib/: Shared library utilities.
    • docs/: API documentation.
  2. Understand the Osmedeus Layered Architecture

    main

    Osmedeus is built using a layered architecture consisting of:

    1. CLI / API Layer: The interface via pkg/cli and pkg/server.
    2. Executor Layer: Manages the Executor, Dispatcher, and various Step Executors (e.g., bash, function, foreach).
    3. Runner Layer: Provides execution environments including Host Runner, Docker Runner, and SSH Runner.
    4. Support Systems: Includes the Template Engine, Functions Registry, and Scheduler (triggers).
    5. Data Layer: Handles the Parser/Loader, Database (SQLite/PG), and Workspace Manager.
  3. Access Osmedeus Documentation

    main

    Comprehensive documentation for Osmedeus is available at the following locations:

  4. Understand the Osmedeus Execution Pipeline

    main

    Osmedeus uses a multi-stage pipeline to execute workflows (Modules or Flows):

    1. CLI/API: The entry point for requests.
    2. Executor: Initializes the execution context, sets up the runner, and iterates through steps. It handles pre-conditions, exports, decision routing, and on_success/on_error actions.
    3. Dispatcher: Uses a plugin registry to route steps to the correct handler based on stepType.
    4. Runner: The low-level component that actually executes commands on a host, in Docker, or via SSH.

    Built-in Step Executors:

    • bash: Executes bash commands.
    • function: Executes internal functions.
    • foreach: Iterates over inputs.
    • parallel-steps: Runs steps in parallel.
    • remote-bash: Executes bash on remote targets.
    • http: Performs HTTP requests.
    • llm: Uses LLM capabilities.
    • agent: Agentic LLM loop with tool calling.
    • agent-acp: ACP subprocess agents.
  5. Use Distributed Mode

    main

    Submit scans to a distributed worker pool. This requires the Osmedeus server to be started with the --master flag. Each target in a targets array is submitted as a separate task to the worker queue.

    # Multiple targets distributed across workers
    curl -X POST http://localhost:8002/osm/api/runs \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "flow": "subdomain-enum",
        "targets": ["example.com", "test.com", "demo.com"],
        "run_mode": "distributed",
        "priority": "high"
      }'
  6. Add New Functions to the Goja JavaScript Runtime

    main

    To extend the Osmedeus JavaScript runtime with new functions, follow these three steps:

    1. Implement the function in Go: Add the logic in the appropriate file (e.g., internal/functions/file_functions.go) using the goja.FunctionCall signature.
    2. Register the function: In goja_runtime.go, map the function name to the implementation using vm.Set.
    3. Define the constant: Add the function name as a constant in constants.go to ensure consistency.

    Example implementation:

    // 1. Implementation
    func (vf *vmFunc) myNewFunction(call goja.FunctionCall) goja.Value {
        arg := call.Argument(0).String()
        // ... implementation
        return vf.vm.ToValue(output)
    }
    
    // 2. Registration in goja_runtime.go
    _ = vm.Set("my_new_function", vf.myNewFunction)
    
    // 3. Constant in constants.go
    const FnMyNewFunction = "my_new_function"
    // 1. Implementation
    func (vf *vmFunc) myNewFunction(call goja.FunctionCall) goja.Value {
        arg := call.Argument(0).String()
        // ... implementation
        return vf.vm.ToValue(output)
    }
    
    // 2. Registration in goja_runtime.go
    _ = vm.Set("my_new_function", vf.myNewFunction)
    
    // 3. Constant in constants.go
    const FnMyNewFunction = "my_new_function"
  7. Best practices for Osmedeus Cloud usage

    main

    To manage costs and efficiency when using Osmedeus Cloud, follow these best practices:

    • Cost Management: Always set cost limits before large-scale scans. Use --auto-destroy to prevent forgotten instances from accruing charges, and use spot instances for non-critical scans to achieve 70-80% savings.
    • Efficiency: Use --reuse to avoid re-provisioning infrastructure during iterative work. To reduce setup time from ~5 minutes to ~30 seconds, use custom snapshots with your required tools pre-installed.
    • Scaling & Monitoring: Start small by testing with 1 instance before scaling up. Regularly run cloud list to verify there is no orphaned infrastructure.
  8. Add a new custom lint rule

    main

    To extend the linter, implement the LinterRule interface in internal/linter/rules.go and register it in GetDefaultRules().

    1. Define your struct and implement Name(), Description(), Severity(), and Check().
    2. Add an instance of your rule to the GetDefaultRules() slice.
    // 1. Create the rule
    type MyNewRule struct{}
    
    func (r *MyNewRule) Name() string        { return "my-new-rule" }
    func (r *MyNewRule) Description() string { return "Detects my issue" }
    func (r *MyNewRule) Severity() Severity  { return SeverityWarning }
    
    func (r *MyNewRule) Check(wast *WorkflowAST) []LintIssue {
        var issues []LintIssue
        // ... implementation
        return issues
    }
    
    // 2. Register in GetDefaultRules()
    func GetDefaultRules() []LinterRule {
        return []LinterRule{
            &MyNewRule{},
        }
    }
  9. Configure Cloud Settings and Environment Variables

    main

    Cloud configurations are managed via the cloud config command. The system supports environment variable resolution within configuration values using the ${VAR_NAME} syntax. This allows you to securely inject credentials like API tokens.

    To set a provider and an environment variable for a token:

    osmedeus cloud config set defaults.provider digitalocean
    osmedeus cloud config set providers.digitalocean.token ${DIGITALOCEAN_TOKEN}
    osmedeus cloud config show
    osmedeus cloud config set defaults.provider digitalocean
    osmedeus cloud config set providers.digitalocean.token ${DIGITALOCEAN_TOKEN}
  10. Manage and Destroy GCP Cloud Infrastructure

    main

    Manage existing cloud instances or perform a full cleanup of your GCP infrastructure.

    List Infrastructure

    osmedeus cloud list

    Destroy Instances

    # Destroy a specific instance by ID
    osmedeus cloud destroy <infra-id>
    
    # Destroy all instances (Nuclear option)
    osmedeus cloud destroy all --force

    Persistent Recon (Reuse Instances)

    To save setup time, create instances once and reuse them for multiple scans:

    # Create instances once
    osmedeus cloud create --provider gcp -n 3
    
    # Run scans using --reuse
    osmedeus cloud run -f fast -t target1.com --reuse
    osmedeus cloud run -f fast -t target2.com --reuse
    
    # Destroy at end of day
    osmedeus cloud destroy all --force
  11. Run Cloud Workflows (Single and Multiple Targets)

    main

    Execute scans using predefined flows (-f), specific modules (-m), or custom target lists (-T).

    Single Target Options:

    • -f <flow>: Specify a flow (e.g., fast, general).
    • -m <module>: Run a specific module (e.g., enum-subdomain).
    • -t <target>: Single target string.
    • --timeout <duration>: Set a timeout (e.g., 2h).
    • --provider <name>: Override the default provider.

    Multiple Targets (Scaling):

    • -T <file>: Path to a file containing targets.
    • --instances <count>: Number of workers to distribute targets across.
    • --chunk-size <count>: Number of targets per worker.
    • --chunk-count <count>: Split targets into exactly N chunks.
    # Run a flow
    osmedeus cloud run -f fast -t example.com
    
    # Run a specific module
    osmedeus cloud run -m enum-subdomain -t example.com
    
    # Distribute targets across 5 workers
    osmedeus cloud run -f fast -T targets.txt --instances 5
    
    # 10 targets per worker
    osmedeus cloud run -f fast -T targets.txt --instances 10 --chunk-size 10