DOCSEARCHbyAlgolia
Submit Docs

dhtcrawler2 Documentation

repository·master·Indexed 24 days ago

https://github.com/btdig/dhtcrawler2

An Erlang-based DHT crawler for high-speed, stable crawling of P2P torrent networks. It indexes torrent data into MongoDB and provides search capabilities via an HTTP interface. The system supports MongoDB text search, Sphinx search integration for advanced text or Chinese character support, and MongoDB Replica Set deployment for primary and secondary database architecture.

Tokens
1.1K
Snippets
3
Records
5
Agent score
31%

What's inside dhtcrawler2

  2. Deploy a MongoDB Replica Set for dhtcrawler2

    master

    To support the dhtcrawler2 architecture, you can deploy a MongoDB Replica Set consisting of a primary database (used by the crawler) and secondary databases (used by the HTTP server for queries).

    Prerequisites:

    • Ensure mongod is included in your system PATH.

    Deployment Steps:

    1. Start the primary node using db-start-primary.bat.
    2. Start the secondary nodes using db-start-slave.bat.
    3. Initialize the primary database using init-primary-db.bat.

    Critical Note: After running the initialization script, verify that the rs.initiate() command was successful within the MongoDB shell to ensure the replica set is correctly configured.

    # Sequence of operations
db-start-primary.bat
db-start-slave.bat
init-primary-db.bat
# Verify rs.initiate() success in mongo shell

  3. Install and run dhtcrawler2

    master

    dhtcrawler2 is an Erlang-based DHT crawler that joins P2P networks to collect torrent information and stores it in a MongoDB database. It provides an HTTP interface for searching torrents by keyword.

    Prerequisites

    • Erlang: Install Erlang R16B or newer.
    • MongoDB: Download and start MongoDB with text search enabled.

    Setup Steps

    1. Start MongoDB with the textSearchEnabled parameter:
      mongod --dbpath your-database-path --setParameter textSearchEnabled=true
    2. Start the Crawler:
      • On Windows, run win_start_crawler.bat.
    3. Start the Hash Reader:
      • On Windows, run win_start_hash.bat.
    4. Start the HTTP Server:
      • On Windows, run win_start_http.bat.
    5. Access the Interface: Wait several minutes, then navigate to http://localhost:8000 in your browser.
    mongod --dbpath your-database-path --setParameter textSearchEnabled=true

  4. Enable Sphinx search integration

    master

    By default, dhtcrawler2 uses MongoDB text search. To switch to Sphinx search (recommended for advanced text searching or Chinese character support), follow these steps:

    1. Install Sphinx

    • Download a Sphinx fork like coreseek (which supports Chinese characters).
    • Add the Sphinx bin directory to your system PATH so the crawler can invoke the indexer tool.

    2. Configure Sphinx

    • Edit etc/csft.conf to add a delta index. Example configuration:
      source delta:xml
{
  type = xmlpipe2
  xmlpipe_command = cat g:/downloads/coreseek-4.1-win32/var/test/delta.xml
}
index delta:xml
{
  source = delta
  path = g:/downloads/coreseek-4.1-win32/var/data/delta
}
    - Use absolute paths for all directories.

### 3. Initialize Sphinx Index
- Run `win_init_sphinx_index.bat` to generate a default `sphinx-builder` config file. Terminate the script once generated.
- Edit `priv/sphinx_builder.config` to specify the `main` and `delta` Sphinx index source file names, index names, and the Sphinx config file. **Note**: These must match the configurations you wrote in `etc/csft.conf`.
- Run `win_init_sphinx_index.bat` again to initialize the index file. Terminate the script once finished. **Warning**: Do not run this initialization step again once it has succeeded.

### 4. Run the Search Pipeline
- Start the Sphinx `searchd` server.
- Run `win_start_sphinx_builder` to start the builder, which reads torrents from MongoDB and builds the Sphinx index.
- Update `priv/hash_reader.config` by setting `search_method` to `sphinx`.
- Update `priv/httpd.config` by setting `search_method` to `sphinx`.

    source delta:xml
{
    type = xmlpipe2
    xmlpipe_command = cat g:/downloads/coreseek-4.1-win32/var/test/delta.xml
}
index delta:xml
{
    source = delta
    path = g:/downloads/coreseek-4.1-win32/var/data/delta
}

  5. Configure dhtcrawler2 settings

    master

    Configuration is handled in two ways:

    1. File-based Configuration: Most settings are located in priv/dhtcrawler.config. This file is automatically generated upon the first run. You typically only need to modify network addresses in this file.
    2. Function Arguments: Other configuration values are passed directly as arguments to Erlang functions.

    Specific component configurations include:

    • priv/hash_reader.config: Controls the hash reader behavior (e.g., search_method).
    • priv/httpd.config: Controls the HTTP server behavior (e.g., search_method).
    • priv/sphinx_builder.config: Used when configuring Sphinx search integration.