Apache Nutch Documentation

Apache Nutch provides certain security guarantees but explicitly disclaims others. Understanding these is critical for proper configuration.

Provided Security Properties

• Robust content handling: Nutch aims to handle well-formed fetched content within configured limits. Parsing/normalization should fail safely rather than corrupting the crawl store or host.
• Crawl scope enforcement: Nutch honors the operator's configured URL filters and seeds.
• No execution of fetched content: Fetched pages are treated as data; Nutch does not execute scripts from crawled pages on the server side.

Disclaimed Security Properties (Operator Responsibilities)

• No Sandbox/Curation: Nutch will fetch whatever the operator's scope permits. Reaching internal hosts via the crawler is an inherent risk controlled by your URL filters, not a Nutch defect.
• No nutch-server Authentication: The legacy nutch-server REST API does not have built-in authentication and assumes a trusted environment.
• No Deployment Protection: Nutch does not provide inherent protection for deployments made in untrusted environments.

Apache Nutch is designed to operate in trusted environments (either locally or on a Hadoop cluster). The core security principle is that Nutch is built to fetch and parse content from the open, untrusted web by design. Therefore, the primary security concern is the robust handling of attacker-controllable input (fetched bytes) rather than preventing Nutch from reaching hostile content.

Key Trust Boundaries

1. The Fetch Boundary (Primary): Everything Nutch fetches from the web (pages, redirects, robots.txt, sitemaps, feeds) is considered untrusted and crosses into the parser and crawl store.
2. The Operator/Config Boundary: Seeds, URL filters, plugin configurations, and the nutch-server endpoint are controlled by the operator and are considered trusted.

Deployment Responsibilities

• Operator/Admin: Responsible for managing seeds, URL filters, plugin configuration, the Hadoop/host environment, and securing the nutch-server endpoint.
• Environment Security: The operator must ensure HTTPS and network isolation for any exposed endpoints, as Nutch is not intended to be exposed directly to the internet without fronting authentication.

To run Nutch classes (like indexers or crawlers) within IntelliJ IDEA, configure a new Application run configuration with the following settings:

• Main Class: Enter the fully qualified class name (e.g., org.apache.nutch.indexer.IndexingJob).
• Program Arguments: Add the arguments required by the class. You can obtain these by running the crawl executable for your job (e.g., using full-qualified paths for crawldb and segments plus -deleteGone).
• Working Directory: Set this to your Nutch runtime/local path.
• Classpath: Select Modify options > Modify Classpath and add the config directory for your Working Directory (e.g., runtime/local/conf).
• VM Options: Select Modify options > Add VM Options and include the VM options used when running the crawl executable (e.g., -Xmx4096m -Dhadoop.log.dir=...).

Important: Because the Ant build system is separate from IntelliJ, you must manually trigger a build through ANT to ensure the latest changes are reflected when running.

Nutch relies on Apache Tika artifacts for document parsing and language detection. To upgrade Tika, you must update dependencies in multiple configuration files across the Nutch project and its plugins.

Upgrade Steps

1. Update Core Tika Dependency: Update tika-core in ivy/ivy.xml.
2. Update Tika Parser Dependency: Update tika-parsers-standard-package in src/plugin/parse-tika/ivy.xml. Note: The tika-handler-boilerpipe module (used for tika.extractor=boilerpipe configuration) was moved from tika-core in TIKA-4138.
3. Update Tika Plugin Dependencies: Update the dependencies listed in src/plugin/parse-tika/plugin.xml using the following process:
   • Navigate to the plugin directory: cd src/plugin/parse-tika/
   • Generate the dependency list: ant -f ./build-ivy.xml
   • Extract the formatted XML library lines:

     ls lib | sed 's/^/      <library name="/g' | sed 's/$/"\/>/g'

   • In src/plugin/parse-tika/plugin.xml, replace all lines between the following markers with the output from the command above:
     • Start marker: <!-- dependencies of Tika (tika-parsers-standard-package) -->
     • End marker: <!-- end of dependencies of Tika (tika-parsers-standard-package) -->
4. Handle Overlapping Dependencies (Optional): If there are version mismatches between build/lib and build/plugins/parse-tika/, add duplicated libraries to the exclusions of transitive dependencies in src/plugin/parse-tika/ivy.xml. Ensure the versions in ivy/ivy.xml match Tika's requirements.
5. Clean Local Dependencies: Remove the locally installed dependencies in the plugin directory:

   rm -rf src/plugin/parse-tika/lib/

6. Update Language Identifier Plugin: Repeat steps 2 through 5 for the language-identifier plugin:
   • Navigate to the plugin directory: cd src/plugin/language-identifier/
   • Update tika-langdetect-optimaize in src/plugin/language-identifier/ivy.xml.
   • Regenerate the library list in plugin.xml using:

     ant -f ./build-ivy.xml
     ls lib | sed 's/^/        <library name="/g' | sed 's/$/"\/>/g'

7. Verify Build: Build Nutch and run tests from the project root:

   cd ../../../
   ant clean runtime test