DOCSEARCHbyAlgolia
Submit Docs

SentrySearch

repository·master·Indexed 26 days ago

https://github.com/ssrajadh/sentrysearch

A tool for performing semantic search over video footage using natural language queries. SentrySearch indexes video files into a local ChromaDB vector database and allows users to retrieve trimmed clips of specific events. It supports multiple backends including Gemini, Qwen Cloud, and local Qwen3-VL-Embedding models. Key features include anomalous highlight detection, customizable chunking and preprocessing, and support for .mp4 and .mov formats.

Tokens
16.4K
Snippets
25
Records
113
Agent score
87%

What's inside sentrysearch

  1. Overview of SentrySearch

    master
    SentrySearch is a semantic retrieval tool for video footage. It allows users to find specific events within video files using natural language descriptions or images. Once a match is found, the system automatically crops and saves the relevant video segments. It is part of a workflow that includes SentryMerge for multi-camera tracking and SentryBlur for sensitive area blurring.

  2. How SentrySearch Works

    master

    SentrySearch operates by splitting video footage into overlapping segments. These segments are vectorized using one of three backends:

    • Google Gemini Embedding API
    • Alibaba DashScope (qwen-cloud)
    • Local Qwen3-VL model

    The resulting vectors are stored in a local ChromaDB instance. During a search, your text query (or image) is embedded into the same vector space to find matches. The best matches are then automatically cropped from the original files and saved as independent clips.

  3. Configure SentrySearch requirements and compatibility

    master

    Supported Video Formats

    • .mp4
    • .mov
    • The directory scanner recursively finds these files regardless of folder structure.

    System Requirements

    • Python: 3.11+
    • FFmpeg: Must be on PATH (on macOS, use brew install ffmpeg).
    • Gemini Backend: Requires a Gemini API key.
    • Local Backend: Requires a GPU with CUDA or Apple Metal.

  4. Configure API Keys with sentrysearch init

    master

    Use the init command to configure your environment. This is required if you are using the default Gemini backend. The command will prompt you for your Gemini API Key, write it to a .env file, and run an embedding test to verify the configuration.

    When you can skip this:

    • If you are using the local backend via --backend local.
    • If you have already configured DASHSCOPE_API_KEY in your .env file and are using --backend qwen-cloud.
    sentrysearch init

  5. Search video by text

    master

    Search your indexed footage using natural language queries. The tool will return matches with similarity scores and automatically save the best matching clip.

    Common Options:

    • --results <N>: Number of results to show.
    • --output-dir <DIR>: Directory to save clips.
    • --no-trim: Skip automatic clip trimming (only shows results).
    • --threshold <float>: Adjust the confidence cutoff (default is 0.41).
    • --save-top <N>: Save the top N clips instead of just the best one.
    • --overlay: Overlay Tesla telemetry data (speed, GPS, time) on the clips.
    • --backend <backend> / --model <model>: Override the inferred backend or model.
    $ sentrysearch search "red truck running a stop sign"

  6. Overlay Tesla telemetry data

    master

    When searching, you can overlay speed, GPS, and time data onto the resulting clips if they are sourced from Tesla dashcam files.

    Requirements:

    • Tesla firmware 2025.44.25 or higher.
    • Hardware version HW3+.
    • SEI metadata (available in driving footage, not parked/Sentry mode).
    • For city/road names, geopy must be installed (uses OpenStreetMap Nominatim API).

    Installation: uv tool install ".[tesla]"

    sentrysearch search "car cutting me off" --overlay

  7. Search video with text queries

    master

    Search your indexed footage using natural language queries. SentrySearch returns ranked matches with similarity scores and timestamps.

    Options:

    • --results <N>: Number of results to return.
    • --output-dir <DIR>: Directory to save clips.
    • --no-trim: Skip automatic trimming of clips.
    • --threshold <float>: Confidence cutoff (default: 0.41).
    • --save-top <N>: Save the top N clips instead of just the best match.
    • --dedupe <float>: Cosine similarity ceiling (0–1) to drop near-duplicate chunks. A value of 0.9 is recommended.
    • --rerank: Use a VLM to re-rank the top candidates before trimming.
    • --backend <backend> / --model <model>: Override the auto-detected backend/model.
    • --overlay: (Tesla only) Burn speed, location, and time onto trimmed clips.
    # Save top 5 clips, dropping near-duplicates
sentrysearch search "red truck" --save-top 5 --dedupe 0.9

# Re-rank the top 10 embedding matches with a VLM before trimming
sentrysearch search "pedestrian crossing behind the car" --rerank --results 10

  8. Install and Setup SentrySearch

    master

    To use SentrySearch for natural language video search, you need uv and Python 3.11+.

    1. Clone and install the repository:
    git clone https://github.com/ssrajadh/sentrysearch.git
cd sentrysearch
uv sync
    1. For Tesla telemetry support (enables reverse geocoding for overlays), install the extra dependency:
    uv sync --extra tesla
    1. Configure your Gemini API key: Run the initialization command to prompt for your key, which will be written to a .env file and validated:
    sentrysearch init

    Alternatively, you can set the GEMINI_API_KEY environment variable directly.

    git clone https://github.com/ssrajadh/sentrysearch.git
cd sentrysearch
uv sync
uv sync --extra tesla
sentrysearch init

  9. Configure the Local Backend (No API Key)

    master

    Run indexing and search locally using Qwen3-VL-Embedding models. This is free and private.

    Hardware-specific Installation:

    • Apple Silicon (24 GB+ RAM): uv tool install ".[local]" (uses qwen8b)
    • Apple Silicon (16 GB RAM): uv tool install ".[local]" (uses qwen2b)
    • Apple Silicon (8 GB RAM): uv tool install ".[local]" (uses qwen2b)
    • NVIDIA (18 GB+ VRAM): uv tool install ".[local]" (uses qwen8b)
    • NVIDIA (8–16 GB VRAM): uv tool install ".[local-quantized]" (uses 4-bit qwen8b)

    Important Notes:

    • Requires Python 3.11 or 3.12. For Python 3.13+, use: uv python install 3.12 && uv tool install --python 3.12 ".[local]".
    • Mac users must install FFmpeg: brew install ffmpeg.
    • The --model flag can be used to override the auto-detected model (e.g., --model qwen2b).
    • Embeddings from different backends are not compatible.
    # Example for NVIDIA with limited VRAM
uv tool install ".[local-quantized]"
sentrysearch index /path/to/footage --backend local
sentrysearch search "car running a red light"

  10. Use Local Backend (No API Key)

    master

    The local backend uses Qwen3-VL-Embedding to run indexing and searching entirely on your hardware. It is free and private.

    Hardware Selection & Installation:

    HardwareInstall CommandAuto-detected ModelNotes
    Apple Silicon (RAM $\ge$ 24GB)uv tool install ".[local]"qwen8bMPS float16
    Apple Silicon (16GB RAM)uv tool install ".[local]"qwen2b8B won't fit; 2B is ~6GB
    Apple Silicon (8GB RAM)uv tool install ".[local]"qwen2bHigh load may cause swapping
    NVIDIA (VRAM $\ge$ 18GB)uv tool install ".[local]"qwen8bbf16 (Linux/Win uses CUDA)
    NVIDIA (VRAM 8–16GB)uv tool install ".[local-quantized]"qwen8b4-bit quantization (~6–8GB)

    Important Notes:

    • Python Version: Requires Python 3.11 or 3.12. If using 3.13+, use: uv python install 3.12 && uv tool install --python 3.12 ".[local]".
    • Mac Requirement: Must install FFmpeg via brew install ffmpeg.
    • Model Overrides: Use --model qwen2b or --model qwen8b to manually select a model. Using --model implies --backend local.
    • Compatibility: Different backends/models use incompatible vector spaces. You must re-index if you switch models.
    sentrysearch index /path/to/footage --backend local
sentrysearch search "car running a red light"

  11. Understand Qwen DashScope (Alibaba Cloud) billing

    master

    DashScope billing for qwen3-vl-embedding is based on Tokens per 1,000 inputs and varies by modality:

    • Text Input: ~¥0.0007 / 1k Tokens
    • Image/Video Input: ~¥0.0018 / 1k Tokens

    Note that indexing uses the Video modality, while search and img queries primarily use Text or Image tokens. Total costs depend on the token count, which is influenced by resolution, duration, and sampling settings (e.g., the DASHSCOPE_VIDEO_FPS environment variable).

  12. Index video footage

    master

    Use the index command to process video files into searchable chunks. You can customize chunking, resolution, and frame rate via options.

    Options:

    • --chunk-duration <seconds>: Duration of each chunk (default: 30).
    • --overlap <seconds>: Overlap between consecutive chunks (default: 5).
    • --no-preprocess: Skip downscaling and frame rate reduction (sends raw chunks).
    • --target-resolution <pixels>: Target height for preprocessing (default: 480).
    • --target-fps <fps>: Target frame rate for preprocessing (default: 5).
    • --no-skip-still: Embed all chunks, even those with no visual change.
    • --backend <backend>: Specify the backend (e.g., local).
    $ sentrysearch index /path/to/video/footage