DOCSEARCHbyAlgolia
Submit Docs

Memories.ai Video Datalake Documentation

website·Indexed 34 days ago

https://api-tools.memories.ai/home/overview

API documentation for the Video Datalake, featuring tools for video indexing, semantic and image search, AI-generated summaries, transcriptions, and face entity detection. Includes guides on managing collections, async operations, webhooks, and a JSON-first CLI client.

Tokens
17.2K
Snippets
63
Records
136
Agent score
50%

What's inside Memories.ai

  1. Overview of the Video Agent

    A Video Agent is an LLM agent (such as Claude, Cursor, or a custom LLM app) that uses the Video Datalake as its long-term video memory. It operates the Datalake via an MCP Server, allowing the agent to ingest, index, search, and read specific video moments using natural language without requiring custom glue code. The Datalake converts raw video into queryable derived content, including embeddings, captions, transcripts, entities, and safety events.

  2. Overview of the Video Datalake

    The Video Datalake is a video-understanding engine that transforms raw videos and live streams (via file, URL, or RTMP) into machine-readable derived content, including embeddings, captions, transcripts, face entities, and safety events. This allows developers to search collections using semantic, keyword, or image search and retrieve specific moments in natural language without requiring an agent to download and watch the entire video.

  3. Overview of Video Datalake API capabilities

    The Video Datalake provides a suite of 41 endpoints for video data management, utilizing OmniRetriever for 3072-dim multilingual embeddings and OmniCaptioner for captions and summaries. Key capabilities include:

    • Collections: Support for fixed embeddings, optional safety detection, and face recognition.
    • Ingestion: Uploads via URL, file, or resumable sessions, including live stream attachment.
    • Async Operations: Uses a 202 + Operation model with polling and HMAC-signed webhooks.
    • Derived Content: Generation of captions, transcripts, frames, clips, titles, summaries, speakers, and entities.
    • Search: Support for semantic, keyword, hybrid, and image-based search returning moment lists.
    • Face Library: Management of face libraries and safety-detection events.

  4. Understand Video Datalake pricing and billing

    The Video Datalake uses a pay-as-you-go model against a prepaid balance. Management and read-from-database operations are free; users are billed only for indexing, search, media generation, and storage.

    Key billing rules:

    • Failed requests: Not billed. Requests fail with 402 quota_exceeded when the balance reaches zero.
    • Idempotency: Requests using the same Idempotency-Key (or idempotency_key for uploads) are not double-billed.
    • Cancellations: Ingests cancelled before the index phase starts are not billed for indexing.
    • Data Retention: Stored data is permanently deleted 30 days after the balance reaches zero. Reminders are sent on day 15 and day 29.

  5. Retrieve a person from the Video Datalake

    Use the GET /datalake/v1/persons/{id} endpoint to fetch details about a specific person. The API is hosted at https://api.memories.ai/serve/datalake/v1 and requires an API key in the Authorization header (e.g., Authorization: sk-mai-...).

    Optional query parameters for data expansion:

    • expand=faces: Returns 24-hour signed reference-image links.
    • expand=embedding: Returns the aggregated face vector.

    This operation is free for management and read access.

    curl -X GET "https://api.memories.ai/serve/datalake/v1/persons/{id}?expand=faces" \
     -H "Authorization: sk-mai-your-key-here"

  6. Memories plugin architecture and requirements

    The Memories plugin is a thin client that connects to a hosted MCP server at https://mcp.memories.ai/mcp. There is no local server to run.

    Requirements:

    • A memories.ai account (free tier supported).
    • The host (Claude Code or Codex) acts as the OAuth client to manage tokens and authorization.

  7. Understand Collections and Indexing Defaults

    A collection is the top-level container in the Datalake. It defines two inherited settings for all videos within it: enabled_detectors (the safety-detection allow-list) and face_recognition_enabled (the master switch for persons and face entities).

    Indexing uses the following models by default:

    • OmniRetriever: The video-embedding model (3072-dim, multilingual). This is the only accepted value for embedding_model and defines the vector space for all search.
    • OmniCaptioner: Used to generate AI captions, titles, and summaries.

  8. Migrate from Visual Intelligence API 1.0 to Datalake API

    The previous Visual Intelligence, Visual Search, and Visual Agents (API 1.0) are deprecated and will be sunset on August 13, 2026. Users should migrate to the current Datalake API before this date.

  9. Connect an MCP client to the Memories.ai Video Datalake

    The Memories.ai MCP Server allows MCP-compatible agents to index and search video, manage face libraries, and monitor safety events in the Video Datalake using natural language. The server is hosted by Memories.ai and uses OAuth for authentication via a browser sign-in flow; no API keys are required.

    Server Endpoint: https://mcp.memories.ai/mcp

    Prerequisites:

    • A memories.ai account.
    • An MCP-compatible client (e.g., Claude Desktop, Cursor, Claude Code, Codex).

  10. Retrieve detected face entities from a video

    Use the GET /datalake/v1/videos/{id}/entities endpoint to retrieve a list of detected face entities within a specific video.

    Requirements & Constraints:

    • The collection must have face_recognition_enabled: true for this endpoint to return data; otherwise, the response will always be empty.
    • The type parameter defaults to person.
    • Pricing: $0.001 per call.

    Request Details:

    • Host: https://api.memories.ai/serve/datalake/v1
    • Authentication: Requires an API key in the header: Authorization: sk-mai-...
    curl -X GET "https://api.memories.ai/serve/datalake/v1/videos/{id}/entities" \
     -H "Authorization: sk-mai-..."

  11. Integrate Video Datalake with Agents and CLI tools

    The Video Datalake can be operated without custom glue code through the following interfaces:

    • MCP Server: Connects to Claude, Cursor, or any Model Context Protocol (MCP) client to manage the lake using natural language.
    • CLI: A command-line interface for terminal-based workflows.
    • Claude Code / Codex plugin: Specialized plugins for agent-driven workflows.

  12. Manage account balance and plans

    Users can manage their prepaid balance via the Memories.ai Console at https://console.memories.ai/stripe.

    Plan Options:

    • Pay-as-you-go: No monthly fee; billed per API usage.
    • Starter Plan ($20/month): Bundles 10,000 free simple-API calls, 2 QPS, 1 project, and 2 API keys. Indexing, search, and AI processing are still billed at standard per-API rates.