DOCSEARCHbyAlgolia
Submit Docs

DOM Distiller Documentation

repository·main·Indexed 20 days ago

https://github.com/chromium/dom-distiller

A tool used to power Reader Mode in Chrome on Android, Desktop, and iOS by extracting core text from web pages and removing non-essential elements. This repository provides utilities for scrawling web page data, extracting features, labeling data for distillability models, and building the distiller using ant. It includes guides for setting up development environments on Ubuntu, Mac OS X, and Vagrant, as well as instructions for running custom Chrome builds with DOM Distiller support.

Tokens
4.7K
Snippets
17
Records
20
Agent score
71%

What's inside DOM Distiller

  1. Verify feature extraction accuracy

    main

    Use the provided sanity check scripts to ensure your JavaScript-based feature extraction matches Chrome's native implementation.

    Check derived features: check_derived_features.py compares features between the JS implementation and Chrome's native implementation (requires Chrome to be running with --distillability-dev).

    • To compare features from the original page: --features <path>
    • To compare features from MHTML archives: --features <path> --from-mhtml

    Check distilled content: check_distilled_mhtml.py compares the distilled content from the original page against the content distilled from the MHTML archive. They should be identical.

    Usage:

    ./check_derived_features.py --features out_dir/feature-derived
./check_distilled_mhtml.py --dir out_dir

  2. Debug DOM Distiller using Chrome Developer Tools

    main

    You can perform interactive debugging of DOM Distiller by running tests within the Chrome browser's Developer Tools.

    1. Prepare the test environment: Run ant extractjs.jstests or ant test to update the test JavaScript.
    2. Open the test page: Open war/test.html in Chrome desktop.
    3. Open Console: Open the Console panel (Ctrl-Shift-J or Cmd-Option-I on Mac).
    4. Execute tests: Call the test runner via the console to run all tests or a filtered subset.

    If JavaScript source maps are enabled, the Sources panel will contain both the extracted JavaScript and the Java source files, allowing you to set breakpoints in the Java code.

    // Run all tests
org.chromium.distiller.JsTestEntry.run()

// Run a specific test using a regular expression filter
org.chromium.distiller.JsTestEntry.runWithFilter('MyTestClass.testSomething')

  3. Simulate mobile distillation from a desktop browser

    main

    To test how DOM Distiller behaves on mobile devices using a desktop Chrome instance, follow this workflow:

    1. Emulate Device: Open Developer Tools, click the mobile device icon, select a Device, and reload the page.
    2. Capture User-Agent: Copy the string from the UA field in the emulation panel.
    3. Launch Chrome with Emulation: Restart Chrome using the captured User-Agent and enabling the distiller.

    Command Template:

    chrome --user-agent="<USER_AGENT_STRING>" --enable-dom-distiller
    1. View Distilled Content: Once the page loads, select Toggle distilled page contents from the menu.
    # Example for Nexus 4 emulation
--user-agent="Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 4 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19"

  4. Label data for distillability models

    main

    To train a distillability model, you must manually label the collected data. Use server.py to host a local web server that allows you to view pages and provide labels. Labels are periodically written to out_dir/archive/.

    Usage:

    1. Run the server pointing to your data directory: ./server.py --data-dir out_dir
    2. Open the address printed in the terminal (e.g., http://0.0.0.0:8081) in your browser to begin labeling.
    ./server.py --data-dir out_dir

  5. Prepare final CSV output for model training

    main

    Once features are extracted and labels are collected, use write_features_csv.py to generate the final .csv files used for training machine learning models.

    For the Distillability Model: Requires a marked label file from the archive. ./write_features_csv.py --marked <path_to_latest_label_file> --features <derived_features_dir> --out <output_prefix>

    For the Long-Article Model: Uses both distilled and non-distilled features. ./write_features_csv.py --distilled <distilled_features_dir> --features <raw_features_dir> --out <output_prefix>

    # Distillability model training data
./write_features_csv.py --marked $(ls -rt out_dir/archive/*|tail -n1) --features out_dir/feature-derived --out labelled

# Long-article model training data
./write_features_csv.py --distilled out_dir/dfeature-derived --features out_dir/feature-derived --out labelled

  6. Run Chrome for desktop with DOM Distiller support

    main

    To run a custom build of Chrome with your local DOM Distiller changes:

    1. Set the required environment variables:

      • CHROME_SRC: Path to your Chromium source.
      • DOM_DISTILLER_DIR: Path to your DOM Distiller checkout.

    2. Use the roll-distiller function (see below) to package and copy files into Chrome.

    3. Run GN to setup ninja build files from $CHROME_SRC:

      gn args out/Debug
    4. Build and run Chrome with the distiller enabled:
   ```bash
   autoninja -C out/Debug chrome && out/Debug/chrome --enable-dom-distiller

    To ensure a clean state, you can add --user-data-dir=/tmp/$(mktemp -d) to the Chrome command line.

    export CHROME_SRC=/path/to/chromium/src
export DOM_DISTILLER_DIR=/path/to/dom-distiller

# Automate packaging
roll-distiller () {
  ( 
    (cd $DOM_DISTILLER_DIR && ant package) && \
    rm -rf $CHROME_SRC/third_party/dom_distiller_js/dist/* && \
    cp -rf $DOM_DISTILLER_DIR/out/package/* $CHROME_SRC/third_party/dom_distiller_js/dist/ && \
    touch $CHROME_SRC/components/resources/dom_distiller_resources.grdp
  )
}

# Build and run
autoninja -C out/Debug chrome && out/Debug/chrome --enable-dom-distiller

  7. Build DOM Distiller using ant

    main

    The project uses ant as the primary build tool. You can list all available targets using ant -p.

    Common Build Targets

    • ant test: Runs all tests.
    • ant test -Dtest.filter=$FILTER_PATTERN: Runs a subset of tests (e.g., *.FilterTest.*:*Foo*-*Bar*).
    • ant gwtc: Compiles .class and .java files to JavaScript. Output: war/domdistiller/domdistiller.nocache.js.
    • ant gwtc.jstests: Creates standalone JavaScript for tests.
    • ant extractjs: Creates standalone JavaScript from ant gwtc output. Output: out/domdistiller.js.
    • ant extractjs.jstests: Creates standalone JavaScript for tests.
    • ant package: Copies main build artifacts (extracted JS and protocol buffer files) into out/package.
    ant test

  8. Scrawl web page data and extract features

    main

    Use get_screenshots.py to generate screenshots of both original and distilled web pages and extract features using extract_features.js. For consistent screenshot sizes and to avoid UI interruptions, it is recommended to run this inside xvfb-run with a specified screen resolution.

    Key Options:

    • --out <dir>: Specifies the output directory.
    • --urls-file <file>: Path to a file containing URLs (one per line).
    • --emulate-mobile: Enables mobile-friendliness emulation.
    • --save-mhtml: Saves a copy of the page in MHTML format (useful for re-extraction).
    • --resume: Resumes a previously interrupted process.
    • --write-index: Exports data for the next stage (labeling/training) after collecting enough data.
    • --load-mhtml: Loads data from MHTML archives instead of live URLs.
    • --skip-distillation: Skips the distillation step (useful when only re-extracting features).
    # Basic usage
./get_screenshots.py --out out_dir --urls-file urls.txt

# Recommended usage with xvfb
xvfb-run -a -s "-screen 0 1600x5000x24" ./get_screenshots.py --out out_dir --urls-file urls.txt

# Resuming an interrupted run
xvfb-run -a -s "-screen 0 1600x5000x24" ./get_screenshots.py --out out_dir --urls-file urls.txt --resume

# Exporting data for training
./get_screenshots.py --out out_dir --urls-file urls.txt --write-index

  9. Recalculate derived features from raw features

    main

    If you have modified how features are derived but do not want to re-extract raw features from the web (which can be slow or produce different results), you can recalculate derived features from existing raw features using calculate_derived_features.py.

    Requirement: You must have previously enabled raw feature writing in get_screenshots.py by uncommenting the writeFeature() line in the source code to populate out_dir/feature.

    Usage: ./calculate_derived_features.py --core <raw_features_dir> --out <output_dir>

    ./calculate_derived_features.py --core out_dir/feature --out out_dir/feature-derived

  10. Set up the DOM Distiller development environment

    main

    Before building, ensure you have installed Google Chrome at its default platform location and installed the git hooks using ./create-hook-symlinks.

    Get the code

    Clone the repository into a directory outside of your main Chromium checkout (i.e., do not put it inside chromium/src):

    git clone https://chromium.googlesource.com/chromium/dom-distiller

    Platform-specific Setup

    Ubuntu/Debian

    Run the dependency installation script from within the dom-distiller folder:

    sudo ./install-build-deps.sh

    Mac OS X

    1. Install JDK 7.
    2. Install Homebrew, then install ant and python via brew.
    3. Install the protocol buffer compiler with Python bindings: brew install protobuf --with-python.
    4. Create a buildtools folder inside your checkout.
    5. Download ChromeDriver, unzip chromedriver_mac32.zip, and place the binary in the buildtools folder.
    6. Install pip via sudo easy_install pip.
    7. Install selenium via pip install --user selenium.

    Note: When running shell commands that normally use xvfb-run, you can omit the xvfb-run prefix on Mac OS X.

    Vagrant (Windows/Red Hat Linux)

    If you are not on Ubuntu or Mac, use Vagrant:

    1. Install Vagrant (v1.7.2+ recommended).
    2. Run vagrant up and vagrant ssh.
    3. Follow the Ubuntu/Debian setup steps inside the VM.

  11. How to use Chrome's Reader Mode

    main

    DOM Distiller powers Reader Mode, which provides a distraction-free viewing experience by extracting core text and stripping non-essential elements.

    Android

    Reader Mode is available on up-to-date versions of Chrome for Android. When visiting a non-mobile-friendly article, tap the "Show simplified view" infobar at the bottom of the screen. You may need to enable this feature in accessibility settings first.

    Desktop

    Reader Mode for desktop Chrome is an experimental feature (as of M80). To enable it:

    1. Navigate to chrome://flags.
    2. Search for enable-reader-mode.
    3. Set the dropdown to "Enabled".
    4. Click "Relaunch Now".
    5. When visiting an article, click the Reader Mode icon that appears in the omnibox.

  12. Install the DOM Distiller developer extension

    main

    To use the developer extension (which supports profiling extraction code and adds a Developer Tools panel to trigger extraction), follow these steps:

    1. Build the extension: Run ant package to generate an unpacked extension in the out/extension folder.
    2. Open Chrome extensions: Navigate to chrome://extensions.
    3. Enable Developer Mode: Toggle the 'Developer mode' switch.
    4. Load the extension: Click 'Load unpacked' and select the out/extension directory.

    Tip: For easier JavaScript debugging, build with the following command to enable pretty-printing: ant package '-Dgwt.custom.args=-style PRETTY'.

    ant package '-Dgwt.custom.args=-style PRETTY'