DOCSEARCHbyAlgolia
Submit Docs

8mb.local Documentation

repository·main·Indexed 21 days ago

https://github.com/jms1717/8mb.local

A self-hosted, GPU-accelerated video compressor designed to shrink video files to specific target sizes. It utilizes NVIDIA NVENC for high-speed hardware encoding with automatic CPU fallback (libx264, libx265, libaom-av1). Features include a FastAPI backend, a SvelteKit SPA frontend, and support for various codecs, containers (MP4, MKV), and target size presets.

Tokens
9.8K
Snippets
29
Records
45
Agent score
75%

What's inside 8mb.local

  1. Understand the Multi-Job Queue System

    main

    The 8mb.local queue system enables multiple users to submit video compression jobs simultaneously. It uses a distributed architecture to manage parallel processing, live progress tracking, and job state management.

    Core Components

    • Redis: Stores job metadata and handles SSE (Server-Sent Events) pub/sub.
    • Celery: Manages the distributed task queue.
    • FastAPI: Provides the REST API and SSE streaming endpoints.

    Job Lifecycle

    1. Submission: A job is submitted via /api/compress.
    2. Metadata Creation: Backend creates metadata in Redis (job:{task_id}) and adds the job to the jobs:active sorted set.
    3. Processing: A Celery worker picks up the job based on available concurrency.
    4. Progress: The worker publishes real-time updates via Redis pub/sub.
    5. Completion: Once finished, the job remains visible in the queue for 1 hour before being cleaned up.

  2. Understand encoder mapping for H.264, HEVC, and AV1

    main

    8mb.local automatically selects between NVIDIA NVENC and CPU-based encoders based on hardware availability. If NVIDIA hardware is detected and passes startup tests, it is used; otherwise, the system falls back to CPU encoding.

    Encoder Selection Logic:

    User-facing choiceNVIDIA EncoderCPU Fallback
    H.264h264_nvenclibx264
    HEVC (H.265)hevc_nvenclibx265
    AV1av1_nvenclibaom-av1

    AV1 Decoding Note: If an AV1 input requires decoding before an NVENC encode, the worker first probes av1_cuvid. If the GPU/driver does not support AV1 decoding, it automatically falls back to the software decoder libdav1d to ensure the job completes.

  3. How to compress a video in 8mb.local

    main
    1. Upload: Drag and drop a video file into the UI.
    2. Target Size: Select a preset (e.g., 8 MB, 25 MB) or enter a custom value.
    3. Advanced Options (Optional):
      • Video Codec: Choose AV1 (best for RTX 40/50), HEVC (H.265), or H.264.
      • Audio Codec: Choose Opus (default) or AAC. Note: MP4 containers will automatically switch to AAC as Opus is not supported in MP4.
      • Speed/Quality: Select NVENC presets from P1 (fastest) to P7 (best quality). Default is P6.
      • Container: Choose MP4 (high compatibility) or MKV.
      • Tune: Select HQ (default), Low Latency, Ultra-Low Latency, or Lossless.
      • Resolution: Set max width/height to downscale while maintaining aspect ratio.
      • Trimming: Specify start/end times (seconds or HH:MM:SS).
    4. Execute: Click Compress. You can monitor real-time progress and FFmpeg logs in the UI. The download will start automatically upon completion.

  4. Install 8mb.local with NVIDIA GPU via Docker

    main

    To use NVIDIA hardware acceleration (NVENC), run the container with the --gpus all flag and set the NVIDIA_DRIVER_CAPABILITIES environment variable to compute,video,utility. This is required to mount the necessary NVENC libraries into the container.

    Ensure you have the NVIDIA Container Toolkit installed on your host.

    docker run -d \
  --name 8mblocal \
  --gpus all \
  -e NVIDIA_DRIVER_CAPABILITIES=compute,video,utility \
  -p 8001:8001 \
  -v ./uploads:/app/uploads \
  -v ./outputs:/app/outputs \
  jms1717/8mblocal:latest

  5. Configure NVIDIA GPU support for 8mb.local

    main

    To use NVIDIA hardware acceleration (NVENC), you must ensure the host has NVIDIA drivers installed and the NVIDIA Container Toolkit is configured. When using Docker or Docker Compose, you must grant the container access to the GPU using the gpus: all flag (or equivalent) and set the necessary environment variables like NVIDIA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES as specified in the project's docker-compose.yml.

    # Example Docker Compose snippet for NVIDIA access
gpus: all
environment:
  - NVIDIA_VISIBLE_DEVICES=all
  - NVIDIA_DRIVER_CAPABILITIES=all

  6. Install NVIDIA Container Toolkit (Debian/Ubuntu)

    main

    To use NVIDIA GPUs within the container, you must install the NVIDIA Container Toolkit on your host system. Follow these steps to configure the repository and install the package:

    # Install (Debian/Ubuntu)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

  7. Manage settings via Settings UI

    main

    The /settings endpoint in the web UI allows you to manage several configurations without restarting the container:

    • Authentication: Enable/disable and manage credentials.
    • Default Presets: Set defaults for target size, codec, quality, and container.
    • Codec Visibility: Enable or disable specific NVIDIA and CPU codecs.
    • Preset Profiles: Create named presets for quick access.
    • Worker Concurrency: Adjust the parallel job limit (Note: changing this via UI may require a container restart depending on implementation, though the README notes WORKER_CONCURRENCY env var requires a restart).
    • Size Buttons: Customize the target size quick-pick buttons.

  8. Install 8mb.local for CPU only

    main

    If you do not have an NVIDIA GPU (e.g., macOS or a machine without the NVIDIA Container Toolkit), run the container without GPU flags. The system will automatically use CPU software encoders (libx264, libx265, etc.) as a fallback.

    docker run -d \
  --name 8mblocal \
  -p 8001:8001 \
  -v ./uploads:/app/uploads \
  -v ./outputs:/app/outputs \
  jms1717/8mblocal:latest

  9. Configure Reverse Proxy for SSE (Server-Sent Events)

    main

    The application uses Server-Sent Events (SSE) for real-time progress updates via the /api/stream/ endpoint. If you use a reverse proxy, you must disable buffering; otherwise, the progress bar will appear stuck at 0% until the job completes.

    Apply the following configurations based on your proxy provider:

    ### Nginx / Nginx Proxy Manager

```nginx
location /api/stream/ {
    proxy_pass http://backend:8001;
    proxy_buffering off;
    proxy_cache off;
    proxy_set_header Connection '';
    chunked_transfer_encoding on;
}

    Traefik

    labels:
  - "traefik.http.middlewares.no-buffer.buffering.maxRequestBodyBytes=0"
  - "traefik.http.middlewares.no-buffer.buffering.maxResponseBodyBytes=0"
  - "traefik.http.routers.8mblocal.middlewares=no-buffer"

    Apache

    <Location /api/stream/>
    ProxyPass http://backend:8001/api/stream/
    ProxyPassReverse http://backend:8001/api/stream/
    SetEnv proxy-sendchunked 1
    SetEnv proxy-interim-response RFC
</Location>

  10. Install 8mb.local via Docker Compose

    main

    You can use Docker Compose for easier management. Use the standard configuration for NVIDIA GPU support or the docker-compose.cpu.yml file for CPU-only environments.

    NVIDIA GPU Configuration:

    services:
  8mblocal:
    image: jms1717/8mblocal:latest
    container_name: 8mblocal
    ports:
      - "8001:8001"
    volumes:
      - ./uploads:/app/uploads
      - ./outputs:/app/outputs
      - ./.env:/app/.env  # optional
    gpus: all
    environment:
      - NVIDIA_DRIVER_CAPABILITIES=compute,video,utility
    restart: unless-stopped

    CPU Only (Build from source):

    docker compose -f docker-compose.cpu.yml up -d --build
    services:
  8mblocal:
    image: jms1717/8mblocal:latest
    container_name: 8mblocal
    ports:
      - "8001:8001"
    volumes:
      - ./uploads:/app/uploads
      - ./outputs:/app/outputs
      - ./.env:/app/.env
    gpus: all
    environment:
      - NVIDIA_DRIVER_CAPABILITIES=compute,video,utility
    restart: unless-stopped

  11. Configure 8mb.local via Environment Variables

    main

    You can configure the application by creating a .env file and mounting it to /app/.env.

    Available Variables:

    • AUTH_ENABLED: Boolean to enable/disable authentication.
    • AUTH_USER: Username for authentication.
    • AUTH_PASS: Password for authentication.
    • FILE_RETENTION_HOURS: How long to keep uploads/ and outputs/ files before cleanup.
    • WORKER_CONCURRENCY: Maximum number of parallel compression jobs (requires container restart).
    • CODEC_H264_NVENC, CODEC_HEVC_NVENC, CODEC_AV1_NVENC: Enable/disable specific NVIDIA hardware codecs.
    • CODEC_LIBX264, CODEC_LIBX265, CODEC_LIBAOM_AV1: Enable/disable specific CPU software codecs.
    • REDIS_URL: Connection string for the Redis broker.
    • BACKEND_HOST: Host for the FastAPI backend.
    • BACKEND_PORT: Port for the FastAPI backend.
    # Authentication
AUTH_ENABLED=false
AUTH_USER=admin
AUTH_PASS=changeme

# File retention
FILE_RETENTION_HOURS=1

# Worker concurrency
WORKER_CONCURRENCY=4

# Codec visibility
CODEC_H264_NVENC=true
CODEC_HEVC_NVENC=true
CODEC_AV1_NVENC=true
CODEC_LIBX264=true
CODEC_LIBX265=true
CODEC_LIBAOM_AV1=true

# Redis / backend
REDIS_URL=redis://127.0.0.1:6379/0
BACKEND_HOST=0.0.0.0
BACKEND_PORT=8001

  12. Configure Worker Concurrency

    main

    You can adjust how many compression jobs run in parallel by modifying the Celery worker concurrency setting in supervisord.conf.

    Warning: Increasing concurrency increases CPU and GPU load. For NVENC (NVIDIA hardware encoding), it is recommended to keep concurrency between 2 and 4 for efficiency.

    [program:worker]
command=celery -A worker.celery_app worker --loglevel=info -n 8mblocal@%%h --concurrency=4