Unfurl

Query rules match URL query parameters (?key=value) by their key name.

• Basic rule: Match a key and apply a label/data type.
• hover_only: If true, it adds hover text to the existing parameter node without creating a new child node.
• path scoping: Use the path key to restrict a query rule to a specific URL path (e.g., only apply rule for q if the path is /search).
• data_type: url: Setting this allows Unfurl to parse the parameter value as a full URL.

query_rules:
  - key: q
    path: /search
    apply:
      label: "Search: {value}"
      data_type: example.search_query

  - key: redirect_uri
    apply:
      data_type: url

Fragment rules match the URL fragment (the part after #) using regular expressions.

• Use named regex groups (e.g., (?P<name>...)) to create placeholders for the label field.
• If a named group is absent from the match, use the label_single field as a fallback template.
• Note: In YAML, you must escape backslashes in regex (e.g., use \d instead of \d).

fragment_rules:
  - pattern: "^L(?P<start>\\d+)(?:-L(?P<end>\\d+))?$"
    apply:
      label: "Lines {start}-{end}"
      label_single: "Line {start}"
      data_type: example.line_ref

To run Unfurl in a containerized environment using Docker Compose:

1. Clone the repository: git clone https://github.com/obsidianforensics/unfurl
2. Navigate to the directory: cd unfurl
3. Start the services: docker-compose up -d

Unfurl is available as a web service:

1. Web Interface: Visit https://dfir.blog/unfurl, enter a URL, and click 'Unfurl!'.
2. Bookmarklet: You can create a browser bookmark with the following URL to unfurl the current page instantly: javascript:window.location.href='https://dfir.blog/unfurl/?url='+window.location.href;

To use Unfurl on your local machine, install the dfir-unfurl package with all dependencies using pip:

pip install dfir-unfurl[all]

To manually run all unit and integration tests, use the following command from the repository root:

python -m unittest discover -s unfurl/tests

If you are running Unfurl inside a Docker container, use:

docker exec unfurl python -m unittest discover -s unfurl/tests

After installing via pip, you can run a local web interface to visualize URL expansion:

1. Run the command unfurl_app.
2. Open your browser and navigate to localhost:5000/ (the port is editable via the configuration file).
3. Enter the target URL into the form and click 'Unfurl!'.

You can teach Unfurl how to parse specific websites by creating .yaml files in the unfurl/parsers/site_defs/ directory. This allows you to define parsing rules for URL structures without writing Python code. Unfurl automatically loads these files on the next run.

Requirements: You must have PyYAML installed:

pip install pyyaml

name: Example Site
domains:
  - example.com
edge:
  color: "#FF6600"
  title: Example Site
  label: "E"

path_rules:
  - match:
      1: users
    position: 2
    apply:
      label: "User: {value}"
      data_type: example.user_id
      hover: A user profile identifier.

You can expand URLs directly in your terminal using the unfurl command. If the URL contains special characters (like &) that your shell might interpret, wrap the URL in quotes.

Example usage:

unfurl https://twitter.com/_RyanBenson/status/1205161015177961473

You can deploy Unfurl using Docker Compose. By default, the service maps port 5000 on the host to port 5000 in the container.

To provide custom configuration, you must mount a local unfurl.ini file to the container at /unfurl/unfurl.ini with read-only (ro) permissions. The service includes a healthcheck that monitors the endpoint http://localhost:5000/.

services:
  unfurl:
    ports:
      - "5000:5000"
    volumes:
      - ./unfurl.ini:/unfurl/unfurl.ini:ro

The unfurl CLI takes a URL (or other data types like timestamps and encoded protobufs) and expands it into a directed graph, extracting obscured information.

If the provided argument is a file path, the CLI will process the file line-by-line, treating each line as a URL to be unfurled. Results can be output to the console as a text tree or as JSON. You can also save results to a CSV file.

# Unfurl a single URL (default output is a text tree)
unfurl https://example.com

# Unfurl a file containing multiple URLs and save to CSV
unfurl urls.txt -o results.csv

# Unfurl a URL and output JSON format
unfurl https://example.com -t json