repository·master·Indexed 42 days ago
https://github.com/andybalholm/redwoodAn internet content-filtering program that intercepts and filters web traffic using categories, rule lists, and access control lists (ACLs). It supports URL matching, regular expressions, content phrase scanning, image hashing, and SSLBump for HTTPS filtering. Redwood can also function as a standalone URL classification service providing JSON data on categories and scores.
Redwood uses a Go implementation of Starlark that enables optional features including set, lambda, recursion, and reassigning global variables.
It provides the following built-in modules:
json, math, time (standard Starlark REPL modules)base64, bsoup, csv, hash, html, http, re, and yaml (from github.com/qri-io/starlib)Redwood uses Starlark (a Python-based scripting language) to allow users to extend filtering logic via external script files.
Redwood loads script files specified via the starlark-script configuration directive. It collects the functions defined within these files and executes them by name during request processing.
Key behaviors:
starlark-go implementation.# Example of a function that Redwood would call automatically
def ssl_bump(session):
# logic here
passRedwood classifies sites into categories. Each HTTP message is assigned a score in each category based on rule lists.
Each category is assigned one of three actions:
allow: The category's score is used to determine if a page is allowed.block: If the score for any block category is higher than the highest score for any allow category, the page is blocked (provided it exceeds the global threshold).ignore: The score does not affect blocking decisions.acl: The category is ignored for top-scoring calculations but remains available for Access Control Lists (ACLs) if the score is greater than zero.Categories are stored in subdirectories within the directory specified by the categories config key. Each subdirectory (named after the category) must contain a category.conf file.
category.conf fields:
description: A user-visible name (defaults to the directory name).action: allow, block, ignore, or acl (defaults to ignore).invisible: true: If set, blocked pages for this category return an invisible image instead of a block page.# Example category.conf for a category named 'mechanical'
description: Auto Repair
action: allowRule lists define how scores are calculated for a category. Files must have a .list extension and be UTF-8 encoded.
napaonline.com 200 matches www.napaonline.com.ip: followed by an address or range (e.g., ip:10.1.10.0/24).h (hostname), d (base domain), p (path), or q (query)./t[iy]re/ 75.< >. Matches are case-insensitive and ignore non-alphanumeric characters.<phrase> weight [max_weight].phrase-scan ACL action is active.dhash. Rules start with % followed by a 32-character hash.%<hash> [threshold].hash-image ACL action is active.default rule: Sets the weight for any rule in the file that doesn't specify one. If no default is present, the weight is zero.<grease> 10 and <grease paint> -10)..urllist files: For very large lists of URL matches. These use a compact format and must start with a score <value> line. They only support URL matching rules.# Example rule-list file
napaonline.com 200
www.napaonline.com/catalog/ 50
default 150
carquest.com
autozone.com
/t[iy]re/ 75
<grease gun> 25
<oil filter> 25 100
%909841dcf4d4c000ff7f00fe30820000 100ACLs allow fine-grained control by assigning tags to requests based on attributes and then performing actions based on those tags.
Use the acl directive in an ACL file:
acl <tag-name> <attribute> <values>
Available Attributes:
connect-port: (Request only) Destination port of a CONNECT request.content-type: (Response only) Media type (supports wildcards like image/*).fingerprint: (CONNECT only) Client JA4 fingerprint.header-name: Checks for presence of a header.header-value: Case-insensitive match of a header's entire value.http-status: (Response only) HTTP status code (supports multiples of 100).method: HTTP method (e.g., GET, POST).referer: The Referer header (supports URL matching rules).referer-category: A category where the Referer has a positive score.server-ip: The server's IP (matches literal IPs in the URL; no DNS lookup).time: Current time (supports days SMTWHFA and 24h ranges).url: The requested URL (supports URL matching rules).user-agent: Case-insensitive regex match against the User-Agent.user-ip: The user's IP (supports CIDR or ranges).user-name: Username from proxy authentication.Actions are selected based on matching tags. Earlier lines in the ACL file take precedence.
Common Actions:
allow: Proceed with request.block: Return 403 with standard block page.block-invisible: Return 403 with a 1-pixel image.disable-proxy-headers: Do not add X-Forwarded-For or Via headers.hash-image: (Response only) Calculate and compare image hashes.ignore-category: Drop the highest-scoring category and re-evaluate.log-content: Save page content to content-log-dir.phrase-scan: (Response only) Run phrase scanning on content.require-auth: (Request only) Require Proxy-Authorization.ssl-bump: (CONNECT only) Activate HTTPS filtering.# Assign tag 'no-web' to a specific IP
acl no-web user-ip 192.168.1.25
# Block all requests with the 'no-web' tag
block no-web
# Assign 'images' tag to image content types
acl images content-type image/*
# Perform image hashing for the 'images' tag
hash-image imagesRedwood supports HTTP basic proxy authentication.
Specify a file using the --password-file directive. Each line format:
<username> <password> [optional-items]
Optional Items:
iPad) and network (CIDR or domain).Example Password File Entry:
ipad-user mySecurePassword 7500 iPad myvzw.com (Authenticates an iPad on the myvzw.com network on port 7500).
Use --authenticator <program> to run an external program. The program is invoked with the username and password as command-line arguments. An exit status of zero indicates successful authentication.
Use the ip-to-user option to pre-authenticate specific LAN IP addresses. The mapping file format is:
<ip-address> <username>
192.168.1.66 joe_pcTo use Starlark scripts, you must specify the path to your .star files in the Redwood configuration file (redwood.conf) using the starlark-script directive.
starlark-script /etc/redwood/safesearch.starSSLBump allows Redwood to perform Man-in-the-Middle filtering of HTTPS traffic. This requires the ssl-bump ACL action.
Requirements:
tls-cert and tls-key options.trusted-root option.Configuration:
tls-cert: Path to the PEM certificate.tls-key: Path to the PEM private key.trusted-root: Path to additional trusted root certificates.block-obsolete-ssl: Option to block SSL versions older than SSLv3.ACL Example:
acl https-traffic method CONNECT followed by ssl-bump https-traffic (Note: ssl-bump is specifically for CONNECT requests).
Redwood can automatically modify URL parameters and remove specific HTML elements from pages.
Configured via the query-changes directive. Each line contains a URL pattern (matching or regex) followed by the desired query parameters. Existing parameters with the same names will be replaced.
# Force safe search on Google
/www\.google\.[^/]+/search/ safe=vssConfigured via a dedicated file. Each line contains a URL pattern and a CSS selector for elements to be deleted.
# Delete Bing ad sidebar
bing.com div.sb_adsNv2
# Delete specific forum elements if category score exceeds 50
talk.newagtalk.com/forums 50 td.messagecellbody > ulRedwood's main configuration file is located at /etc/redwood/redwood.conf by default. You can specify a different path using the -c command line switch.
Configuration options can be defined in the file using key = value or key value syntax. Comments start with #. Values can be enclosed in double quotes and support standard backslash escapes. You can include additional configuration files using the include directive.
Common configuration keys include:
http-proxy: Port to listen for connections.blockpage: Path to the HTML template for blocked pages.static-files-dir: Directory for static files served by the internal web server.cgi-bin: Directory for CGI scripts.categories: Directory containing category information.acls: Path to the Access Control List configuration.threshold: Minimum score from a blocked category required to block a page.content-pruning: Path to the content pruning configuration.query-changes: Path to the URL query modification configuration.access-log: Path to the access log file.# Example configuration
http-proxy :8000
blockpage "/etc/redwood/block.html"
static-files-dir /etc/redwood/static
cgi-bin /etc/redwood/cgi
categories /etc/redwood/categories
acls /etc/redwood/acls.conf
threshold 275
content-pruning /etc/redwood/pruning.conf
query-changes /etc/redwood/safesearch.conf
access-log /var/log/redwood/access.logQueryDict wraps url.Values to provide dictionary-like access to URL query parameters within Starlark scripts.
Available Methods:
get(key, default): Returns the first value associated with the key, or the default if not found.pop(key, default): Returns the value and removes the key from the query string.items(): Returns an iterator of key-value pairs.len(): Returns the number of query parameters.Modifying a QueryDict via setkey or pop will update the underlying query string.
# Example usage in a Starlark script
user_id = query.get("id", "0")Redwood calculates a score for a page within a category based on matched rules and their associated weights.
weight consists of points (points per occurrence) and maxPoints (the maximum points allowed per page for that rule).(points * count) for all matched rules.config.CountOnce is enabled, the score for a rule is simply its points, regardless of how many times it matched.maxPoints is set, the points contributed by a rule are capped at that value.parent_multiplier defined in the child's category.conf.