DOCSEARCHbyAlgolia
Submit Docs

CrowdSec

repository·master·Indexed 12 days ago

https://github.com/crowdsecurity/crowdsec

An open-source security engine providing crowdsourced threat detection and protection. It analyzes logs to identify malicious behavior and uses a distributed model to share threat intelligence. Key features include a WAF challenge runtime with JavaScript obfuscation, a Parser for transforming raw logs into structured Event objects, and Leakybuckets for detecting behavioral patterns like SSH bruteforce using various types including leaky, trigger, uniq, counter, and bayesian.

Tokens
63.8K
Snippets
220
Records
293
Agent score
95%

What's inside CrowdSec

  1. Overview of CrowdSec Security Engine

    master

    CrowdSec is an open-source, participative security solution designed for modern infrastructures. It uses a "Detect Here, Remedy There" approach, allowing you to analyze logs from multiple sources in a centralized location and remediate threats across different levels of your stack (application, system, or infrastructure).

    Key components include:

    • Security Engine: Acts as an IDS/IPS and WAF by analyzing log sources and HTTP requests to detect malicious behavior.
    • Remediation Components (Bouncers): Used to actively block or mitigate threats identified by the engine.
    • CrowdSec Hub: A repository where you can find detection scenarios (e.g., brute force, port scans, web scans) and remediation components to extend your protection.

  2. How the JS Obfuscation Pipeline works

    master

    To prevent attackers from easily extracting epoch keys, the challenge runtime uses a WASM-hosted JavaScript obfuscator (wazero) to generate dynamic code for the browser.

    The Four JS Components

    1. Obfuscated Challenge Code: A small, static, build-time obfuscated bundle containing the core crypto/glue logic. Injected inline.
    2. Dynamic Key Module: A small script (~30 lines) that embeds the current K_epoch as a hex literal. This is obfuscated at runtime and cached in a CryptoObfuscationPoolSize pool. Injected inline.
    3. fpscanner Bundle: The public device-fingerprint scanner. It is unobfuscated and served as a static file at /crowdsec-internal/challenge/fpscanner.js.
    4. PoW Worker: A separate worker script served at /crowdsec-internal/challenge/pow-worker.js.

    Performance Optimizations

    • Pre-warming: A background goroutine generates the next epoch's obfuscated modules before the rotation boundary to prevent latency spikes.
    • Singleflight: Concurrent requests arriving during a rotation boundary will block on a single obfuscation task rather than triggering multiple redundant WASM executions.

  3. Configure a Parser Node

    master

    A parser configuration is defined as a Node object in YAML. Nodes can contain Grok patterns, enrichment instructions, and hierarchical sub-nodes (trees).

    Key Configuration Fields

    • name (optional): If provided, statistics will be generated for this node (requires Prometheus or profiling to be active).
    • filter (optional): An expr expression evaluated against the Event. If it returns false, the node is skipped.
    • debug (optional): A boolean that enables debug mode for the node at runtime and during configuration parsing.
    • onsuccess (mandatory): Defines the behavior after a successful node execution:
      • next_stage: Move the line to the next parsing stage.
      • continue: Continue processing within the current stage.
    • pattern_syntax: A list of subgroks (custom patterns) declared at the Event level for use within Grok patterns.
    • nodes: A list of child Node entries used to build hierarchical parsing trees.
    filter: "evt.Line.Labels.type == 'testlog'"
debug: true
onsuccess: next_stage
name: tests/base-grok
pattern_syntax:
  MYCAP: ".*"
nodes:
  - grok:
      pattern: ^xxheader %{MYCAP:extracted_value} trailing stuff$
      apply_on: Line.Raw
statics:
  - meta: log_type
    value: parsed_testlog

  4. Understand the WAF challenge JS pipeline architecture

    master

    The WAF challenge runtime uses two distinct JavaScript bundles served to clients. This separation optimizes for both performance and security.

    1. Fingerprint Scanner (fpscanner/fpscanner.js)

    • Purpose: The public fingerprint scanner.
    • Content: Located in fpscanner/src/* and entered via fpscanner/global.js.
    • Format: A minified IIFE that assigns globalThis.CrowdsecFingerprintScanner.
    • Delivery: Served unobfuscated at ChallengeFPScannerPath via a standard <script> tag. It is designed to be cacheable across different challenge pages.

    2. Challenge Code (initial_bundle.js.gz)

    • Purpose: Contains sensitive crypto and glue logic (SHA-256, HMAC, PoW, fingerprint obfuscation, submission, and hook registration).
    • Content: Derived from ../challenge.js.
    • Format: Obfuscated and compressed. The runtime injects this inline on the challenge page.
    • Interaction: It reads the scanner from globalThis.CrowdsecFingerprintScanner at runtime.

    Pipeline Summary

    StepToolOutputRuntime Usage
    cmd/bundleesbuildfpscanner/fpscanner.js & challenge_code.jsfpscanner.js is served as-is; challenge_code.js is the obfuscator input
    cmd/obfuscateesbuild + javyobfuscate/index.wasm.gzUsed by wazero to obfuscate the dynamic key module per epoch
    cmd/initialbundlewazero (running obfuscator WASM)../initial_bundle.js.gzSeeded into the challenge cache at startup

  5. Use Statics to modify Events

    master

    Statics are used to alter the Event structure when a node is considered successful (e.g., an empty node, a successful Grok match, or a successful enrichment).

    Statics can target meta or parsed dictionaries, or specific fields using a target key. Data can be assigned via a static value or the result of an expression (expr_result).

    Supported targets:

    • meta: Add or alter an entry in the Meta dictionary.
    • parsed: Add or alter an entry in the Parsed dictionary.
    • target: Specify a destination field by name (e.g., Meta.my_key).
    statics:
    - meta: service
      value: tcp
    - meta: source_ip
      expression: "Event['source_ip']"
    - parsed: "new_connection"
      expression: "Event['tcpflags'] contains 'S' ? 'true' : 'false'"
    - target: Parsed.this_is_a_test
      value: foobar

  6. Security Invariants for WAF Bot Detection Challenges

    master

    When implementing or modifying the WAF Bot Detection/Challenge subsystem, adhere to these security invariants:

    • Stateless Crypto vs. Stateful Single-use: Cryptographic material (PoW-MAC, cookies) must be derivable from the master_secret, epoch, and request data to allow any instance to verify signatures. However, replay protection is stateful: the challenge nonce r must be burned in an in-memory spent_set. Note that single-use is enforced per-instance; a multi-instance fleet without a shared spent_set cannot enforce fleet-wide single-use.
    • master_secret Policy: A misconfigured master_secret must cause a configuration load failure. It must never silently fall back to a random secret if an operator intended to set one.
    • Secret Logging: Never log the master_secret. Keys like k_epoch or s may only be logged at the debug level behind an IsLevelEnabled(debug) guard.
    • Cookie/Keyring Decoupling: Cookie validity (TTL) is enforced via the not_after field inside the sealed envelope, not by the lifecycle of the keyring. Do not tie cookie TTL to live-epoch membership.
    • User-Agent Binding: The User-Agent (UA) is used as Additional Authenticated Data (AAD) in sealCookieV0 and openCookie. Do not remove this, as it breaks the binding between the cookie and the browser.
    • Challenge Bypass: Any helper producing a challenge response (e.g., SendChallenge) must early-return if state.ChallengeBypassed is set to avoid incoherent bouncer protocol responses (like sending two Set-Cookie headers and a redirect simultaneously).
    • Submission Phase Constraints: The on_challenge_submit environment intentionally omits certain state-changing functions like SendChallenge, SetRemediation, SetReturnCode, SetChallengeDifficulty, and DropRequest. Do not add these to the submission phase, as it will break the client JS state machine. Instead, set state and let the next request's pre_eval act on it.

  7. Build hierarchical Parser Trees

    master

    You can build complex parsing logic by nesting nodes within a parent node. The evaluation follows this logic:

    1. Evaluate the parent node's filter. If it fails, exit.
    2. Iterate over the child nodes and apply each.
    3. If a child has a grok entry, process it. If the Grok succeeds, apply that child's local statics.
    4. If any child node or the Grok was successful, apply the parent node's statics.
    filter: "Event['program'] == 'nginx'"
nodes:
  - grok:
      name: "NGINXACCESS"
      statics:
        - meta: log_type
          value: "http_access-log"
  - grok:
      name: "NGINXERROR"
      statics:
        - meta: log_type
          value: "http_error-log"
statics:
  - meta: service
    value: http

  8. How the CrowdSec Community Blocklist works

    master
    The Community Blocklist is a curated list of IP addresses identified as malicious by the CrowdSec network. The Security Engine can proactively block these IP addresses, preventing known malevolent actors from reaching your systems before they even attempt an attack.

  9. How Leakybuckets work for decision-making

    master

    Leakybuckets are a mechanism used to detect patterns in enriched events. When events matching a specific filter are "poured" into a bucket, the bucket tracks them based on a stackkey (e.g., source_ip). Once a bucket reaches its defined capacity, it emits an overflow event and is destroyed. This is primarily used to detect clients exceeding specific rates, such as SSH login failures or HTTP authentication errors.

    Key concepts:

    • Stackkey: Determines the bucket instance. Events with the same stackkey value go into the same bucket instance. A single configuration creates many instances distinguished by their unique stackkey values.
    • Overflow: The event emitted when a bucket's threshold is met.
    • Leaking: The process where events are removed from the bucket over time based on the leakspeed configuration.

  10. Compare Leakybucket types

    master

    CrowdSec provides several bucket types for different detection logic:

    TypeBehavior
    leakyStandard bucket. Overflows when capacity is reached. Events leak out over leakspeed intervals.
    triggerA specialized bucket with capacity: 0. Any event poured into it causes an immediate overflow.
    uniqBehaves like a standard bucket but enforces uniqueness. It uses a uniq_filter to extract a property; if that property value has already been seen in the current bucket instance, the event is ignored.
    counterHas infinite capacity and leakspeed. It does not overflow based on event count, but instead emits an event after a fixed duration.
    bayesianUses Bayesian inference. It evaluates conditions with specific likelihoods (prob_given_benign and prob_given_evil) and overflows when the posterior exceeds bayesian_threshold or leakspeed expires.

  11. Maintain the Sentinel Hook string for challenge bundles

    master

    The WAF challenge uses a sentinel string, __CSEC_CHALLENGE_HOOK_v1__, to bridge the static obfuscated challenge code and the dynamic key modules generated at runtime. This ensures both bundles meet at the same globalThis key after independent obfuscation passes.

    If you need to change the sentinel string, you must update it in all three of the following locations to prevent a mismatch:

    1. Obfuscator Configuration: obfuscate/obfuscate.js (inside the reservedStrings array).
    2. Challenge Source: pkg/appsec/challenge/challenge.js (the CSEC_HOOK_NAME constant).
    3. Dynamic Module Template: pkg/appsec/challenge/dynamic_module.js.tmpl (the hookName variable).

    Failure to synchronize these will cause the Go-side regression test TestSplitBundle_HookSentinelInBakedBundle to fail.

  12. How the Parser works in CrowdSec

    master

    The Parser is responsible for transforming raw log lines into structured Event objects that heuristics can process. Parsing occurs in sequential stages (defined in config/stage/), where the alphabetical order of directories determines the processing sequence.

    An Event is the runtime representation of a log line and contains the following key fields:

    • Parsed: A string dictionary containing the outputs from successful parsers.
    • Meta: A string dictionary containing metadata about the event.
    • Line: The raw representation of the log line.
    • Overflow: A representation of the overflow, if applicable.