DOCSEARCHbyAlgolia
Submit Docs

bitmagnet Documentation

repository·main·Indexed 26 days ago

https://github.com/bitmagnet-io/bitmagnet

A self-hosted BitTorrent indexer and search engine featuring a DHT crawler, content classification, a web UI, and a GraphQL API. It includes native integration with the Servarr stack and a customizable classifier using Common Expression Language (CEL) and YAML-based workflows for content type determination and automated torrent management.

Tokens
12.5K
Snippets
40
Records
80
Agent score
88%

What's inside bitmagnet

  1. Overview of bitmagnet

    main
    bitmagnet is a self-hosted BitTorrent indexer that combines several core functionalities: a DHT crawler, a content classifier, and a torrent search engine. It provides a web UI for interaction, a GraphQL API for programmatic access, and native integration with the Servarr stack.

  2. Overview of bitmagnet API and Web endpoints

    main

    bitmagnet provides several endpoints for user interface access, data querying, and system integration:

    • / : Redirects to the web user interface.
    • /webui : The main web user interface.
    • /graphql : The GraphQL API, which includes the GraphiQL browser interface for interactive querying.
    • /torznab/* : Torznab API, used for integration with compatible applications (e.g., download clients or indexer aggregators).
    • /import : The Import API used to add new content to the library. Refer to the importing guide for specific usage.
    • /metrics : Prometheus-formatted metrics for monitoring.
    • /debug/pprof/* : Go pprof profiling endpoints for performance analysis.
    • /status : A health check and status endpoint for monitoring system availability.

  3. Understand the bitmagnet technology stack

    main

    bitmagnet is built using the following core technologies:

    Storage

    • Postgres: Serves as the primary data store, powering the search engine and the message queue. It utilizes specific Postgres features and extensions.

    Backend (GoLang)

    • Dependency Injection: fx (Uber)
    • HTTP Server: gin
    • Database Migrations: goose
    • ORM: gorm
    • GraphQL Server: gqlgen
    • CLI: urfave/cli
    • Logging: zap (Uber)
    • Regex: rex
    • BitTorrent Utilities: anacrolix/torrent

    Frontend (TypeScript/Angular)

    • Framework: Angular with Angular Material components.
    • Deployment: The web UI is embedded directly into the GoLang binary and served via the Gin web framework.

  4. Configure the Classifier via config.yml or Environment Variables

    main

    The classifier can be customized using a classifier.yml file or via the main application configuration in config.yml and environment variables. Note that the application configuration only exposes a subset of the classifier's properties.

    Common configuration options include:

    • classifier.workflow: Specifies a custom workflow to use.
    • classifier.keywords.<type>: Adds keywords to a specific category (e.g., music).
    • classifier.extensions.<type>: Adds file extensions to a specific category (e.g., audiobook).
    • classifier.flags.delete_content_types: A list of content types to automatically delete.

    Environment variable mappings include:

    • CLASSIFIER_WORKFLOW for classifier.workflow.
    • CLASSIFIER_DELETE_XXX for auto-deleting adult content.
    • TMDB_ENABLED for disabling TMDB API integration.
    classifier:
  # specify a custom workflow to be used:
  workflow: custom
  # add to the core list of music keywords:
  keywords:
    music:
      - my-custom-music-keyword
  # add a file extension to the list of audiobook-related extensions:
  extensions:
    audiobook:
      - abc
  # auto-delete all comics
  flags:
    delete_content_types:
      - comics

    Or via environment variables

    TMDB_ENABLED=false \
  CLASSIFIER_WORKFLOW=custom \
  CLASSIFIER_DELETE_XXX=true \
  bitmagnet worker run --all

  9. Use Actions and Workflows in the Classifier

    main

    A workflow is a list of actions executed on torrents. By default, the default workflow runs, but you can specify a custom one using the classifier.workflow configuration option.

    Common Actions:

    • set_content_type: <type>: Sets the torrent's content type.
    • unmatched: Returns an error indicating no match was found.
    • delete: Deletes the torrent being classified.
    • if_else: Executes an if_action if a condition is met, otherwise executes else_action.
    • find_match: Acts like a try/catch block. It attempts to run a list of actions and catches unmatched errors to proceed to the next check in the list.

    Example: Using find_match to chain content type checks

    find_match:
  - if_else:
      condition: "torrent.baseName.matches(keywords.audiobook)"
      if_action:
        set_content_type: audiobook
      else_action: unmatched
  - if_else:
      condition: "torrent.files.map(f, f.extension in extensions.ebook ? f.size : - f.size).sum() > 0"
      if_action:
        set_content_type: ebook
      else_action: unmatched

  10. Disable TMDB or all API integrations

    main

    You can disable API integrations to save resources or prevent errors.

    • Disable TMDB specifically: Use flags.tmdb_enabled: false in configuration, the tmdb.enabled config option, or the TMDB_ENABLED=false environment variable.
    • Disable all APIs: Use flags.apis_enabled: false in configuration.
    • Disable during reprocess: When running the reprocess command, you can pass the --apisDisabled flag to disable integrations for that specific run.

  11. Run multiple bitmagnet instances

    main

    You can run multiple bitmagnet instances pointing to the same Postgres database. While this works, it increases database load and may decrease application performance.

    An alternative approach is to run multiple instances with separate databases and periodically merge them using the database merge process.