DOCSEARCHbyAlgolia
Submit Docs

Director Framework Documentation

repository·main·Indexed 23 days ago

https://github.com/video-db/director

A framework for building video agents capable of reasoning through complex tasks like search, editing, and generation. Built on VideoDB's 'video-as-data' infrastructure, it features a Reasoning Engine for orchestrating modular agents, a Python-based backend (v0.1.0), and a Vue.js 3 frontend. It supports multiple LLM providers including OpenAI, Anthropic, and Google AI, and provides a VideoDBTool for high-level video operations.

Tokens
8.3K
Snippets
24
Records
69
Agent score
80%

What's inside Director

  1. Overview of The Director

    main

    The Director is an AI-driven video processing and analysis platform built on top of VideoDB. It uses a modular architecture consisting of specialized AI agents and language models to automate video-related tasks.

    Key capabilities include:

    • Intelligent Agents: Specialized agents for video upload, summarization, chapter creation, search, dubbing, dynamic editing, branding, and publishing.
    • Language Model Integration: Chat-based workflows for intuitive video interaction.
    • Flexible Database Interface: Efficient storage, retrieval, and indexing of video content.

    Developers can use The Director to build AI-powered video editors, automate video workflows, or develop new video intelligence applications.

  2. Director Frontend technology stack

    main

    The Director Frontend is built using the following core technologies:

    • Framework: Vue.js 3
    • Routing: Vue Router
    • Real-time Communication: Socket.io Client
    • Chat UI: @videodb/chat-vue
    • Video Player: @videodb/player-vue
    • Styling: Tailwind CSS
    • Build Tool: Vite

  3. Extend the backend with custom Agents

    main
    To implement custom agent functionality, extend the BaseAgent class located in director/agents/base.py. This allows you to create modular components that perform specific tasks within the reasoning engine's orchestration flow.

  4. Implement a custom database using BaseDB

    main
    The director.db.base.BaseDB class serves as the abstract base class for all database implementations within the Director project. When building a custom database connector or adapter, you must inherit from BaseDB to ensure your implementation adheres to the common interface required by the Director backend. This allows the system to interact with your database using a standardized set of methods regardless of the underlying storage technology.

  6. Understand Director Frontend communication patterns

    main

    The Director Frontend interacts with the backend using two distinct communication methods:

    • Real-time Chat: Uses WebSocket connections via Socket.io for real-time chat functionality.
    • Data Fetching: Uses standard HTTP connections to fetch session information, collections, and other metadata.

  7. Understand the Session message hierarchy

    main

    The director session system uses a structured hierarchy of message types to manage communication between users, agents, and the reasoning engine. All messages inherit from BaseMessage and are categorized into four specific types:

    1. BaseMessage: The foundational class for all messages in a session.
    2. InputMessage: Represents data or instructions sent into the session (e.g., from a user or an external trigger).
    3. OutputMessage: Represents data or responses generated by an agent or the system.
    4. ContextMessage: Represents metadata, state, or environmental information that provides background for the current session state.

  8. Understand the Director architecture

    main

    Director is composed of three primary layers that work together to provide a conversational media experience:

    1. Backend Reasoning Engine: The core logic handler that manages workflows, decision-making, and agent orchestration.
    2. Chat-Based UI: The conversational interface used to interact with your media library.
    3. Video Player & Collection View: Advanced playback tools and media organization interfaces.

    This architecture allows users to interact with complex media tasks (like clipping or searching) using natural language.

  9. How the Reasoning Engine works

    main

    The Reasoning Engine is the core of The Director, acting as the brain that drives intelligent decision-making and dynamic workflows. It enables video agents to process natural language commands and orchestrate complex tasks.

    Core Mechanisms

    • Contextual Understanding: Analyzes user input and maintains context for coherent interactions.
    • Dynamic Agent Orchestration: Identifies and activates the specific agents required to fulfill a user's request.
    • Modular Processing: Breaks complex tasks into smaller, manageable steps, allowing multiple agents to collaborate in real time.

    Capabilities

    • Multi-Agent Coordination: Integrates multiple agents (e.g., summarizing, editing, and searching) to handle end-to-end workflows.
    • Real-Time Updates: Provides live progress and feedback during task execution.
    • Extensibility: Can be adapted with custom logic or connected to external GenAI APIs.

  10. What are Tools in Director

    main

    Tools are the functional building blocks used by Agents. They are modular functions that enable an agent to execute specific actions.

    Tools often act as wrapper functions around existing libraries or APIs. For example, an upload tool might be a wrapper around a videodb upload function, or an index tool might provide a parameterized interface for indexing media.

  11. Implement or use an LLM with BaseLLM

    main
    All LLM implementations in Director must inherit from or follow the interface defined by director.llm.base.BaseLLM. This base class ensures a consistent interface across different model providers, allowing the Director reasoning engine to interact with various LLMs interchangeably.

  12. Handle LLM outputs with LLMResponse

    main
    When an LLM processes an input message, it returns an object of type director.llm.base.LLMResponse. This standardized response object contains the processed output from the model, ensuring that the rest of the Director system can predictably consume the results regardless of the underlying LLM provider.