DAT (Data Ask Tool)

DAT (Data Ask Tool) is an enterprise-grade AI framework designed to bridge the gap between natural language and structured data. It allows business users to interact with databases using natural language instead of writing complex SQL queries.

Unlike a standalone platform, DAT is a framework that provides an Askdata Agent workflow. It focuses on ensuring high-quality and deterministic results by providing precise knowledge (semantic modeling), more computational steps, and extended reasoning time.

Key capabilities include:

• Pluggable SPI Architecture: Flexible extension for databases, LLMs, and embedding models.
• Multi-Database Support: Native support for MySQL, PostgreSQL, and Oracle, with extensibility via SPI.
• Semantic SQL Generation: LLM-based natural language understanding with automatic SQL dialect conversion.
• Semantic Modeling: Definition of Entities (primary/foreign keys), Dimensions (time, category, enumeration), and Measures (aggregations, calculated fields) using YAML.
• Vectorized Retrieval Augmentation (RAG): Semantic matching for SQL question-answer pairs, synonyms, and business knowledge using backends like DuckDB, Weaviate, or PGVector.

DAT is organized into two primary layers: the DAT Language (Creation Layer) and the DAT Engine (Execution Layer).

1. DAT Language (Creation Layer)

This layer is where users define the intelligence of the system using:

• Semantic Model Definitions (YAML): Defining the structure and relationships of the data.
• Data Model Configuration: Configuring how data is accessed.
• Intelligent Agent Configuration: Setting up the behavior of the AI agents.

2. DAT Engine (Execution Layer)

This layer handles the heavy lifting of processing a natural language request:

• Natural Language Understanding (NLU): Parsing the user's intent.
• Semantic SQL Generation: Converting intent into database-specific SQL.
• Data Query Execution: Running the generated SQL against the database.
• LLM Management: Handling calls to Large Language Models.
• Vector Retrieval Augmentation: Enhancing queries with retrieved context.
• SPI Component Management: Using the Factory pattern to manage pluggable components.
• Result Formatting: Preparing the data for the end-user.

┌─────────────────────────────────────────────────────────────┐
│                        DAT Framework                        │
├─────────────────────────────────────────────────────────────┤
│  🎯 DAT Language (创作层)                                    │
│  ├── 📝 语义模型定义 (YAML)                                  │
│  ├── 🗃️ 数据模型配置                                         │
│  └── 🤖 智能代理配置                                         │
├─────────────────────────────────────────────────────────────┤
│  ⚙️ DAT Engine (执行层)                                      │
│  ├── 🔤 自然语言理解   │  📊 语义SQL生成    │  🗄️ 数据查询执行  │
│  ├── 🧠 LLM调用管理    │  🔍 向量检索增强   │  📈 结果格式化    │
│  └── 🔌 SPI组件管理    │  🏭 工厂模式创建   │  ⚡ 缓存优化     │
└─────────────────────────────────────────────────────────────┘

DAT (Data Ask Tool) is designed with a modular architecture where each component has a specific responsibility. This allows developers to pick and choose specific capabilities (like specific LLMs, database adapters, or vector stores) by adding the corresponding Maven dependencies.

Core modules include:

• dat-core: Core interfaces and factory management.
• dat-adapters: Database adapters (e.g., DuckDB, MySQL, Oracle, PostgreSQL).
• dat-llms: LLM integration (e.g., OpenAI, Anthropic, Gemini, Ollama).
• dat-embedders: Embedding model integration (e.g., BGE, Jina, OpenAI).
• dat-rerankers: Reranking model integration (e.g., ONNX, MS-Marco).
• dat-storers: Vector storage backends (e.g., PGVector, Weaviate, Qdrant, Milvus).
• dat-agents: Intelligent agent implementations.
• dat-servers: Server components like MCP or OpenAPI.
• dat-sdk: The primary development kit for Java integration.
• dat-cli: Command-line interface tool.

DAT provides several key capabilities for building natural-language-to-SQL applications:

Architecture & Extensibility

• Pluggable SPI Architecture: Supports flexible extensions for multiple databases, LLMs, and embedding models.
• Factory Pattern: Uses a standardized mechanism for component creation and management.
• Modular Design: Separates responsibilities for easier maintenance.

Database & SQL Support

• Multi-database Support: Native support for MySQL (including connection pooling and dialect conversion), PostgreSQL, and Oracle, with extensibility for others via SPI.
• Intelligent Semantic SQL Generation: Combines LLM-based semantic parsing with automatic SQL dialect conversion and semantic model binding.

Semantic Modeling & Retrieval

• Rich Semantic Modeling: Define Entities (PK/FK relationships), Dimensions (time, categorical, enumeration), and Measures (aggregations, calculated fields) using YAML configuration.
• Vector-enhanced Retrieval: Uses embedding models to perform semantic retrieval on SQL Q&A pairs, synonyms, and business knowledge. Supports backends like DuckDB, Weaviate, and PGVector.

Measures are aggregations performed on columns (e.g., sums, counts). They can be final metrics or building blocks for complex calculations.

Supported Aggregation Types (agg)

sum, min, max, avg, count, count_distinct, median, sum_boolean, none (default).

Configuration Parameters

Handling Non-Additive Dimensions

For metrics like account balances that cannot be summed across certain dimensions, use non_additive_dimension to specify a windowing strategy.

measures:
  - name: account_balance
    expr: balance
    agg: sum
    non_additive_dimension:
      name: "balance_date"
      window_choice: "max"
      window_groupings: ["account_id"]

measures:
  - name: total_amount
    expr: amount
    agg: sum
    agg_time_dimension: "order_date"

  - name: unique_customers
    expr: customer_id
    agg: count_distinct