DOCSEARCHbyAlgolia
Submit Docs

DataStoria

repository·master·Indexed 43 days ago

https://github.com/frankchen021/datastoria

An AI-native management console for ClickHouse clusters featuring natural language query generation, intelligent visualization, and deep diagnostic tools. It supports multi-cluster management, smart query optimization, and integrates with LLM providers including OpenAI, Anthropic, Google Gemini, and GitHub Copilot. DataStoria ensures privacy by executing SQL queries directly from the browser to the ClickHouse server without collecting credentials or results.

Tokens
49K
Snippets
80
Records
227
Agent score
53%

What's inside DataStoria

  1. Overview of DataStoria Core Workflows

    master

    DataStoria is an AI-native ClickHouse console. Its core workflows are organized into six functional areas:

    1. Getting Started

    Covers installation (Source, Docker, or Online), setting up your first connection to ClickHouse, authentication, and basic navigation.

    2. AI Features

    Includes Natural Language Data Exploration (SQL generation), Smart Query Optimization (AI recommendations), Intelligent Visualization (prompt-to-chart), AI-powered error assistance, and AI Model Configuration (API key management).

    3. Query Experience

    Provides a professional SQL environment featuring a SQL Editor (syntax highlighting, auto-completion), SQL Snippets, Error Diagnostics, Query Execution (results, exporting, history), a Query Log Inspector (topology graphs, performance analysis), and a Query Explain tool (Visual AST and pipeline visualization).

    4. Database Management

    Includes a Schema Explorer for navigating databases, tables, columns, and metadata.

    5. Monitoring & Dashboards

    Offers Node Dashboards for individual node health/metrics and Cluster Dashboards for cluster-wide aggregation and health overviews.

    6. Security & Privacy

    Focuses on a Local Execution Model and a No Data Collection Policy to ensure data privacy.

  2. Introduction to DataStoria

    master

    DataStoria is an AI-native, web-based console designed for ClickHouse. It provides a unified interface for querying data, managing multiple clusters, and monitoring performance. The platform is built on three core principles:

    1. Privacy First: All SQL queries execute directly from your browser to your ClickHouse server. Credentials and query results never leave your local machine.
    2. AI-Enhanced Intelligence: Uses AI to convert natural language into optimized SQL, provide query suggestions, and generate visualizations.
    3. Complete Control: Enables management of multiple ClickHouse clusters, schema exploration, and performance monitoring from a single interface.

    Users can either install the software locally or use the hosted version at dataStoria.app.

  3. Overview of DataStoria features

    master

    DataStoria is an AI-native ClickHouse console designed for cluster diagnostics, query generation, and intelligent visualization.

    AI Capabilities

    • Natural Language Exploration: Convert plain English descriptions into optimized ClickHouse queries.
    • Smart Query Optimization: AI-driven schema inspection and SQL validation to recommend performance improvements.
    • Intelligent Visualization: Generate SQL and visualization specs (time series, bar, pie, table) via natural language.
    • Agent SKILLs: Supports skill-based agent architecture (compatible with Official ClickHouse Agent Skills).
    • Model Support: Integrates with OpenAI, Anthropic Claude, Google Gemini, GitHub Copilot, OpenRouter, Groq, Cerebras, Nebius, and more.
    • GitHub Copilot Integration: Use existing Copilot subscriptions via GitHub device flow.

    Query & Monitoring

    • Advanced SQL Editor: Includes syntax highlighting, auto-completion, and formatting.
    • Diagnostics: Smart error diagnostics with one-click AI fixes, visual AST/pipeline views for EXPLAIN, and system log inspection (topology graphs, performance analysis).
    • Cluster Management: Multi-cluster support, multi-node dashboards (real-time metrics, replication status), and a Schema Explorer.

    Privacy & Security

    • Local Execution: SQL queries run directly from your browser to your ClickHouse server.
    • No Data Collection: Credentials and query results never leave your machine.
    • Bring Your Own API Key: Users provide their own LLM API keys to maintain control over data.

  4. Use the DataStoria SQL Editor

    master

    The SQL Editor is the primary interface for writing and executing ClickHouse queries. It is built on the Ace Editor framework and features a terminal-like interface where queries are typed in the lower part of the query tab, while results and requests are displayed above.

    Key features include:

    • Syntax Highlighting: Color-coded SQL keywords, strings, numbers, comments, and functions.
    • Auto-completion: Intelligent suggestions for ClickHouse-specific syntax.
    • Query Snippets: Reusable templates for common SQL operations.
    • Local Storage: Automatic saving of queries that persist across browser sessions.

  5. Use System Log Introspection tools in DataStoria

    master

    DataStoria provides introspection tools to analyze ClickHouse system tables. These tools allow you to monitor query patterns, identify performance issues, debug errors, and track data part operations across your cluster.

    You can access these tools via the side bar icon button in the DataStoria interface.

    Supported System Tables

    • system.ddl_distribution_queue: Monitor distributed DDL operations (CREATE, ALTER, DROP, etc.) across cluster nodes.
    • system.opentelemetry_span_log: Inspect OpenTelemetry span logs, view trace distribution, and follow trace ID links.
    • system.query_log: Analyze query execution logs with filtering, charts, and AI-powered optimization insights.
    • system.query_views_log: Monitor executions of materialized views and live views, including performance metrics and error info.
    • system.part_log: Track part-level operations like merges, mutations, downloads, and removals.
    • system.zookeeper: Browse ZooKeeper data using a tree-table interface to inspect znodes and metadata.
    • system.dashboard: Access dashboard configuration and metadata.

  6. Explore data in the Data Sample Tab

    master

    The Data Sample Tab allows you to inspect actual rows from the table for data exploration and quality verification.

    Features

    • Sample Rows: View formatted data from the table.
    • Configurable Sample Size: Adjust the number of rows displayed.
    • Column Display: View all columns with proper formatting.
    • Sorting & Filtering: Sort by any column or filter rows by specific values.
    • Export: Export the sampled data for external use.

  7. Inspect table structure in the Metadata Tab

    master

    The Metadata Tab displays the detailed schema and configuration of the table.

    Table Information

    Includes the Database Name, Table Name, Engine Type, the full CREATE TABLE Statement, and the Metadata Modification Time.

    Column Information

    For every column, you can view:

    • Column Name and Data Type
    • Default Expression
    • Comment
    • Codec (compression codec)
    • TTL (Time-to-live expression)

    Table Properties

    Provides the Partition Key, Order By key, Primary Key, Sample By expression, and all table-level Settings.

    Engine-Specific Information

    Displays configurations specific to the engine, such as Replication Settings (for ReplicatedMergeTree), Distributed Settings, or Kafka Settings.

  8. Understand Dependency Graph Visualizations and Relationships

    master

    The dependency graph uses nodes and edges to represent your database schema. Understanding the visual language is key to interpreting data lineage:

    • Nodes: Represent tables, materialized views, and other database objects (rendered in a UML class diagram style).
    • Edges (Arrows): Represent the direction of dependency.
    • Distributed vs. Local Tables: The graph shows the connection between a distributed table and its corresponding local table. Edge labels between them display the sharding key (e.g., indicating if it uses random sharding or specific column-based sharding).
    • Materialized View Data Flow:
      • Incoming Edge (labeled Push To): Indicates the source table where data originates.
      • Outgoing Edge (labeled Sink to): Indicates the target table where processed data is written.
    • Node Interaction:
      • Clicking a node opens the Table Details Panel.
      • Hovering highlights connected edges.

  9. EXPLAIN ESTIMATE: Quick performance assessment

    master

    EXPLAIN ESTIMATE provides a simplified, concise summary of what a query will read. It is a lightweight alternative to EXPLAIN PLAN.

    Metrics provided:

    • Data Parts: Number of data parts to be read.
    • Rows: Estimated number of rows to be processed.
    • Marks: Number of marks (index entries) to be read.

    Performance Tip: Smaller values for parts, rows, and marks generally indicate better query performance.

    Limitations: It does not provide the full operator tree, node-level actions, or detailed index breakdowns.

  10. How AI Query Optimization works

    master

    The optimization process follows a structured analysis to provide evidence-based recommendations:

    1. Analyze Query Structure: Examines joins, aggregations, filters, and data access patterns.
    2. Review Schema Context: Considers table structures (including primary keys, PARTITION BY expressions), indexes, and data types.
    3. Collect Evidence: Gathers data on index usage, execution pipelines, and table sizes.
    4. Identify Bottlenecks: Locates inefficient patterns like full table scans, unnecessary joins, or suboptimal aggregations.
    5. Suggest Improvements: Provides specific, actionable recommendations with explanations.

    Note: You can expand the collect_sql_optimization_evidence step to retrieve more detailed diagnostic information.

  11. Automatic Skill loading in DataStoria

    master

    The DataStoria agent automatically detects and loads relevant skills based on the intent of your question. It only loads the skills necessary for the specific task to maintain efficiency.

    Example of automatic loading: If you ask: Visualize the number of commits in 2021 Feb by day in line chart, the agent will automatically load the Visualization skill, while skipping unrelated skills like Optimization.

    Visualize the number of commits in 2021 Feb by day in line chart

  12. Visualize table relationships with the Database Dependency Tab

    master

    The Database Dependency tab provides an interactive visual graph to track data lineage and understand how objects relate within a database.

    Supported Dependencies

    The graph includes relationships for:

    • Distributed tables and their associated local tables.
    • Materialized Views (showing source and target tables).
    • Dictionaries.
    • MySQL table engines and their corresponding MySQL servers.
    • Kafka tables and their corresponding Kafka servers.

    Interaction

    • Navigation: Click nodes to view specific table details.
    • Directionality: View upstream and downstream dependency directions.
    • Details Panel: Access the table's DDL statements and metadata via the details panel.