Pixeltable Documentation

Pixeltable is a unified multimodal backend designed for building AI data applications. It replaces the need to manually glue together separate components like blob storage, vector databases, orchestrators, and edge functions.

Key features include:

• Unified API: A single Python API to store media, run models, index embeddings, and serve endpoints.
• Automated Pipelines: Chunking, embeddings, and agents run automatically via computed columns upon data insertion.
• Built-in Reliability: Includes native support for transactions, caching, retries, and observability.
• Extensibility: Developers can extend the system using @pxt.udf (User Defined Functions), @pxt.uda (User Defined Aggregations), and @pxt.query.

The Prompt Engineering Studio is a sample application designed to demonstrate how Pixeltable can be used for LLM testing and prompt engineering. It provides the following capabilities:

• Model Comparison: Side-by-side comparison of responses from different LLM models.
• Parameter Tuning: Interactive control over model parameters like temperature and top_p.
• Experiment Tracking: Automatic versioning and history of all prompts and their corresponding results.
• Advanced Analytics: Automatic analysis of model outputs using computed columns for metrics like sentiment analysis, readability scoring, and keyword extraction.
• Historical Analysis: Ability to query and review past experiments and analytics.

Pixeltable is a declarative data infrastructure for multimodal AI applications. It allows you to define data processing and AI workflows using computed columns on tables.

Key capabilities include:

• Data Ingestion & Storage: Handles structured data and references multimodal files (images, videos, audio, docs) in place.
• Transformation & Processing: Automatically applies Python User-Defined Functions (UDFs) or built-in operations like chunking and frame extraction.
• AI Model Integration: Integrates inference (embeddings, object detection, LLMs) directly into the data pipeline.
• Indexing & Retrieval: Manages vector indexes for semantic search alongside traditional filtering.
• Incremental Computation: Only recomputes what is necessary when data or code changes.
• Versioning & Lineage: Automatically tracks schema and data changes for reproducibility.

Pixeltable is designed for managing multimodal data and machine learning workflows. Its core capabilities are organized into three primary conceptual pillars:

1. Tables and Data Operations: Creating and managing multimodal data tables, importing datasets, and performing essential operations like filtering, selecting, and ordering data.
2. Computed Columns: Automating data transformations and integrating ML models using columns that compute values dynamically. These support automatic versioning and lineage tracking.
3. Queries & Expressions: Using an expressive query system to perform complex data operations, ranging from simple filters to advanced transformations across images, video, text, and embeddings.

Pixeltable provides several specialized application patterns that you can implement using its core capabilities. Use these categories to guide your development:

Chat Applications

• Multimodal Chat: Interfaces that process images, videos, and documents simultaneously.
• Memory Systems: Systems for managing conversation history and context.
• Tool Integration: Adding external capabilities like web search or API calls to chat workflows.

Vision Applications

• Voxel51 Workflow: Advanced dataset management and curation using Voxel51.
• YOLOX Detection: Real-time object detection systems.

Search Applications

• Audio Search: Semantic search capabilities for audio content.
• Image Search: Visual search with automatic indexing.
• PDF & Web Search: Search systems for documents and websites.

The Multimodal Chat Application demonstrates several key capabilities:

• Multimodal Data Support: Ability to process and analyze documents, videos, and audio files.
• Advanced Processing: Uses Pixeltable's specialized data types and formats.
• Flexible LLM Integration:
  • Default: OpenAI
  • Alternatives: Ollama or LLama.cpp
• Local Processing: Supports local transcription using Whisper or WhisperX.

The system follows a five-phase lifecycle to transform a chart screenshot into actionable trading insights using Pixeltable and Claude Sonnet:

1. Image Capture: The Chrome extension captures the chart and generates a requestId.
2. Initial Processing: The extension sends the image via POST /analyze to the FastAPI backend, which initializes a Pixeltable analysis table.
3. AI Analysis: Pixeltable processes the image through computed columns, sending it to Claude Sonnet for structured analysis.
4. Data Processing & Storage: Pixeltable parses the AI response to extract technical indicators (MACD, RSI, etc.), support/resistance levels, and trade setups, then stores the results.
5. Response Delivery: The backend queries Pixeltable by requestId to retrieve the processed analysis and sends the JSON response back to the extension UI.

Semantic search is performed by calculating the similarity between a text query and visual embeddings (frames or images) stored in a CLIP index. You can use the .similarity() method on a column and order the results by that similarity score.

Example of similarity search across video frames:

# Similarity search across video frames
results = frames_view.order_by(
    frames_view.frame.similarity(query, idx='frame_clip_idx'),
    asc=False
).limit(5).collect()