Anubis Web AI Firewall

Anubis is a lightweight Web AI Firewall Utility designed to protect upstream resources from scraper bots, specifically those used by AI companies. It works by issuing one or more challenges to a connection to "weigh its soul" before allowing access.

Key Considerations:

• Impact: Using Anubis is a significant measure that may block smaller scrapers and potentially inhibit "good bots" like the Internet Archive.
• Mitigation: You can mitigate accidental blocking of legitimate bots by configuring bot policy definitions to explicitly allowlist them.
• Use Case: While Cloudflare is often sufficient for protection, Anubis is intended for scenarios where Cloudflare cannot or will not be used.

Anubis v1.20.0 (Thancred Waters) introduces several new capabilities for website protection:

• WEIGH action: A new action for manipulating request weights.
• Custom weight thresholds: Allows for more granular control over when challenges are issued.
• Imprint/impressum support: New support for legal/identity metadata.
• No-JS challenge: A new challenge type that does not require JavaScript execution.

In Anubis, a smoke test is defined as a directory containing an executable shell script named test.sh.

Execution Logic:

• The test passes if test.sh returns exit code 0.
• If the script returns any other exit code, the test is considered failed and is automatically retried up to 10 times by the CI system.

Smoke tests are used to validate server infrastructure and configurations that are commonly used with Anubis, ensuring they remain supported through continuous CI execution.

Thoth is a reputation database and web service that provides advanced intelligence to Anubis instances. While Anubis instances are typically isolated and lack long-term memory, Thoth provides external context to help Anubis distinguish between innocuous and suspicious traffic.

Key Characteristics:

• Informative, not Authoritative: Thoth does not block traffic itself; it provides information that Anubis uses to influence the weight of requests.
• gRPC-based: Thoth communicates via a gRPC web service. Protocol buffer definitions are located in the TecharoHQ/thoth-proto repository.
• Aggressive Caching: Anubis caches data from Thoth to ensure repeat visitors are processed quickly and to reduce the load on the Thoth service.
• Paid Service: Thoth is a hosted, paid service provided by Techaro. Access requires subscribing via GitHub Sponsors and contacting Xe via email.

Anubis operates in two distinct modes depending on your deployment architecture:

1. Reverse Proxy (Default): Anubis acts as a middleware, sitting in the middle of all traffic and reverse proxying requests to a destination.
2. Subrequest Authentication Mode: Anubis acts as a sidecar service. It listens for authentication requests; if a request fails authentication, it is forwarded to Anubis for challenge processing.

Important: When using Subrequest Authentication mode, you must modify the default policy because Nginx interprets a 200 status code as successful authentication. You must configure the status codes as follows:

status_codes:
  CHALLENGE: 200
  DENY: 403

When Anubis is configured to use either the fast or slow challenge methods, it issues a Proof of Work (PoW) challenge to clients. To obtain a token required to access the upstream resource, the client must solve a complex mathematical puzzle using JavaScript.

There are two modes available:

• fast: Uses a heavily optimized, multithreaded implementation for efficient solving.
• slow: Uses a simplistic, single-threaded implementation. This mode is maintained primarily for legacy compatibility.

Anubis flattens the X-Forwarded-For (XFF) header by selecting the first IP address from right to left that does not fall into specific categories. This prevents internal network IPs from being passed to the upstream.

Anubis automatically strips the following IP categories:

• Private IPs: Controlled by the XFF_STRIP_PRIVATE setting (enabled by default).
• CGNAT: Always stripped.
• Link-local Unicast: Always stripped.

Example Behavior: If the incoming header is X-Forwarded-For: 1.2.3.4, 5.6.7.8, 10.0.0.1, Anubis will strip the private IP 10.0.0.1 and set the upstream header to X-Forwarded-For: 5.6.7.8.

The weight subsystem allows administrators to adjust the suspicion level of a request using WEIGH rules without taking immediate action (like ALLOW or DENY).

When a request matches a rule with action: WEIGH, Anubis adjusts the request's weight by the value specified in the weight.adjust field. This weight is cumulative.

After all rules are processed, if the request hasn't matched an ALLOW, CHALLENGE, or DENY rule, Anubis evaluates the final weight against defined thresholds to determine the appropriate action. If no thresholds match, the request is allowed through by default.

- name: gitea-session-token
  action: WEIGH
  expression:
    all:
      - '"Cookie" in headers'
      - 'headers["Cookie"].contains("i_love_gitea=")'
  weight:
    adjust: -5

Anubis uses Google's Common Expression Language (CEL) to allow administrators to define complex security rules that match against multiple aspects of a request simultaneously. CEL expressions are sandboxed and designed for high-performance evaluation.

An expression acts like an if statement or a SQL WHERE clause. If an expression evaluates to true, the rule's action is taken. If it evaluates to false, Anubis proceeds to the next rule. If an expression raises an error, Anubis will return an error.

Warning: CEL is an advanced feature. Incorrectly written expressions can cause rule evaluation to fail.

- name: allow-api-requests
  action: ALLOW
  expression:
    all:
      - '"Accept" in headers'
      - 'headers["Accept"] == "application/json"'
      - 'path.startsWith("/api/")'

Anubis is designed to act as a filter proxy. When integrating with Nginx, you should split your Nginx configuration into two distinct roles:

1. TLS Termination: Nginx handles incoming HTTPS traffic on ports 80/443 and converts it to HTTP.
2. HTTP Routing: Nginx routes the filtered traffic from Anubis to your backend services (via a Unix Socket or another TCP port).

Traffic Flow: User Traffic $\rightarrow$ Nginx (TLS Termination) $\rightarrow$ Anubis (Traffic Filtering) $\rightarrow$ Nginx (HTTP Routing/Upstream) $\rightarrow$ Backend.

Ensure your TLS certificates are correctly configured at the Nginx TLS termination level.

If you need to protect multiple backend services with Anubis, you have two primary architectural options:

1. Multiple Anubis Instances: Start a separate Anubis container for each service. Anubis is lightweight, typically requiring less than 32 MB of RAM on average.
2. Two-Tier Routing: Use one Caddy instance for TLS termination and a second Caddy instance to handle the actual routing to various services via Anubis.