LootScraper

repository·main·Indexed 43 days ago

https://github.com/eikowagenknecht/lootscraper

A tool to automatically find and notify users about free games and in-game loot across platforms like Epic Games, Steam, and Amazon Prime. It provides notifications via a Telegram bot, dedicated Telegram channels, and RSS feeds, as well as a web interface for browsing current and historical offers. Version 2.15.0 supports local installation via Node.js 22+ or deployment as a Docker container.

Tokens
13.8K
Snippets
51
Records
66
Agent score
54%

What's inside LootScraper

  1. Configure the Telegram Bot

    main

    To use Telegram notifications, follow these steps:

    1. Create a Bot: Start a chat with @Botfather on Telegram, send the /create command, and follow the instructions to get an access token.
    2. Get your Chat ID: To receive notifications or errors, you need your user ID or group chat ID. Send the /debug command to the public Telegram LootScraperBot to retrieve it.
    3. Update Config: Add the token and the ID to your configuration file.

    Configuration Keys:

    • DeveloperChatId: The ID where Telegram-related errors will be sent.
  2. Use the LootScraper Telegram bot for customizable notifications

    main

    The Telegram bot is the recommended way to receive personalized notifications for specific platforms. You can manage which platforms you want to hear about directly through the bot interface.

    1. Search for @LootScraperBot in Telegram.
    2. Send /start to initialize the bot.
    3. Use /manage to select your preferred platforms.
    4. Use /help to view all available commands.

    The bot will send messages whenever new free games or loot are detected on your selected platforms. You can also add the bot to your own groups or channels.

  3. Build and run LootScraper as a Docker container

    main

    You can run LootScraper using Docker. You can either build it yourself from the source or use the pre-built image from Docker Hub (eikowagenknecht/lootscraper:main).

    Building from source

    1. Build the image: docker build . -t eikowagenknecht/lootscraper:develop
    2. Create and start the container (mapping the ./data directory for persistence): docker run --volume "${PWD}/data:/data" --name lootscraper eikowagenknecht/lootscraper:develop (Add --detach to run in the background)

    Managing the container

    • Stop: docker stop lootscraper
    • Remove: docker container rm lootscraper
    • View logs: docker logs -f lootscraper

    Debugging To explore the container environment without running the scraper: docker run -it --entrypoint /bin/bash --volume "${PWD}/data:/data" --name lootscraper_debug eikowagenknecht/lootscraper:develop

    # Build
    docker build . -t eikowagenknecht/lootscraper:develop
    
    # Run
    docker run --volume "${PWD}/data:/data" --name lootscraper eikowagenknecht/lootscraper:develop
  4. Run LootScraper locally

    main

    To run LootScraper on your own machine, ensure you have Node.js 22+ installed. Follow these steps:

    1. Download the repository.
    2. Navigate to the directory in your terminal.
    3. Install dependencies using pnpm install.
    4. Install the required Playwright browser: playwright install firefox.

    Development Mode Run the application in development mode using:

    pnpm run dev

    Production Mode To run in production, build the project first, then start it:

    pnpm run build
    pnpm run start
    # Install dependencies
    pnpm install
    # Install browser
    playwright install firefox
    # Run dev
    pnpm run dev
  5. Configure the Discord Bot

    main

    The Discord bot posts offers to dedicated channels with rich embeds and claim buttons.

    1. Discord Application Setup

    • Create a new application in the Discord Developer Portal.
    • Go to the Bot section and click Reset Token to get your bot token.
    • In OAuth2 $\rightarrow$ URL Generator, select the following scopes and permissions:
      • Scopes: bot, applications.commands
      • Permissions: Manage Channels, View Channels, Send Messages, Embed Links
    • Use the generated URL to invite the bot to your server.

    2. Get IDs

    Enable Developer Mode in Discord (User Settings $\rightarrow$ Advanced $\rightarrow$ Developer Mode) to easily copy IDs:

    • Server ID: Right-click your server $\rightarrow$ "Copy Server ID" (used for guildId).
    • Role ID: Right-click an admin role $\rightarrow$ "Copy Role ID" (used for adminRoleId).

    3. Configuration Example

    Add the following to your configuration file:

    actions:
      discordBot: true
    
    discord:
      botToken: "your-bot-token-here"
      guildId: "your-server-id"
      adminRoleId: ""  # Optional
      channelPrefix: ""  # Optional: e.g., "loot-"
      categoryName: "Free Games"  # Category to organize channels under

    4. Slash Commands

    • /status: Show bot stats, feed channels, and upcoming scrapes.
    • /scrapenow: Trigger the scraper (Admin only).
    • /populate: Backfill all channels with current active offers (Admin only).
    actions:
      discordBot: true
    
    discord:
      botToken: "your-bot-token-here"
      guildId: "your-server-id"
      adminRoleId: ""
      channelPrefix: ""
      categoryName: "Free Games"
  6. Browse LootScraper offers via Web Interface

    main

    If you prefer browsing manually without push notifications, you can view current and historical offers on dedicated web pages.

    Current Offers

    View the latest deals for specific platforms or all sources on one page:

    Web Archive

    To see all discovered offers (including expired ones), use the archive versions. Note that these pages can be large and slow to load:

  7. How the BaseScraper pipeline processes offers

    main

    When scrape() is called, the data flows through a standardized pipeline to ensure quality:

    1. Read: readOffers() fetches raw data.
    2. Clean: cleanOffers() trims whitespace from URLs and image URLs.
    3. Deduplicate: deduplicateOffers() removes entries with identical titles.
    4. Categorize: categorizeOffers() identifies if an offer is a DEMO, PRERELEASE, or VALID based on the title.
    5. Filter: filterForValidOffers() removes any offers that were categorized as non-valid (like demos or prereleases) if the scraper is configured to only return valid offers.
  8. Configure LootScraper settings via config.toml

    main

    On the first startup, LootScraper creates a default configuration file at ./data/config.toml. You can edit this file to:

    • Define which sites to visit.
    • Define actions to perform (e.g., sending notifications).
    • Provide API keys.

    After editing the file, you must restart the application for changes to take effect.

  9. Handle application errors and shutdown

    main

    The application is designed to catch unhandled exceptions during the main execution loop. When an error occurs, the system performs the following sequence:

    1. Invokes handleError(error) to process and log the error.
    2. Invokes shutdownApp() to ensure all services (database, etc.) are closed gracefully.
    3. Exits the process with exit code 1.
  10. Configure logging and environment settings

    main

    LootScraper uses a centralized configuration system and a logger that can be controlled via the application configuration.

    • Log Levels: The logging level is determined by the common.logLevel key in your configuration. You can use updateConsoleLevel to change the console output level and addFileTransport to direct logs to a file specified by common.logFile.
    • Debug Mode: If the DEBUG environment variable is set, the application will log its value during initialization.
    • Timezone Handling: The application enforces UTC as the default timezone and en-US as the default locale for consistent date handling across all services.
    // Example of how the application handles environment and config internally:
    // (Note: These are internal initialization steps exposed via the core logic)
    
    // Set log level from config
    updateConsoleLevel(cfg.common.logLevel);
    addFileTransport(cfg.common.logLevel, cfg.common.logFile);
    
    // Debugging via environment variable
    if (process.env.DEBUG) {
      logger.info(`DEBUG env is set to ${process.env.DEBUG}.`);
    }