DOCSEARCHbyAlgolia
Submit Docs

qData Open-Source Data Platform

repository·develop·Indexed 19 days ago

https://github.com/qiantongtech/qdata

An open-source data middle platform for enterprise data governance, ETL processes, and data services. qData provides tools for visual ETL data integration (via DataX or Spark), SQL-based data development (via Quartz or DolphinScheduler), metadata management, data quality auditing, and AI-powered data Q&A using Text2SQL. It supports multiple databases including MySQL, DM8, Oracle, SQL Server, Kingbase8, and Doris.

Tokens
23.6K
Snippets
66
Records
98
Agent score
67%

What's inside qData

  1. Overview of the qData Data Middle Platform

    develop

    qData is an open-source data middle platform designed for enterprise data governance and data development. It provides a centralized foundation for managing the data lifecycle, including access, cleansing, transformation, and asset cataloging.

    Key capabilities include:

    • ETL Data Integration: Visual workflows for data access and transformation.
    • Data Development: SQL-based task development with flexible scheduling.
    • Data Governance: Management of metadata, data models, quality, and assets.
    • API Data Services: Encapsulating data as APIs for system integration.
    • AI-Powered Data Q&A: Natural-language querying via Text2SQL and intelligent analysis.

    Supported databases include MySQL, DM8, Oracle, SQL Server, Kingbase8, and Doris.

  2. Overview of qData Data Mid-Platform

    develop

    qData is an open-source data mid-platform designed for enterprise data governance and data development. It provides a foundation for building ETL platforms, data governance platforms, and data service platforms.

    Core capabilities include:

    • ETL Data Integration: Visual workflows for data access, cleansing, transformation, and output.
    • Data Development: SQL-based data processing and task orchestration.
    • Data Modeling: Support for data standards, warehousing layers, and logical models.
    • Metadata Management: Metadata browsing, versioning, and collection task management.
    • Data Quality: Inspection based on audit rules (completeness, uniqueness, validity, etc.).
    • Data Asset Management: Integrated catalogs with tags and categories for easy discovery.
    • API Data Services: Encapsulating tables or SQL queries into manageable API services.
    • AI Data Querying: Natural language to SQL (Text2SQL) and intelligent chart analysis.

    Supported databases include MySQL, DM8, Oracle, SQL Server, Kingbase8, and Doris.

  5. Understand qData licensing options

    develop

    qData is available in two editions:

    • Open Source Edition: Designed for low-cost entry and initial setup.
    • Professional Edition: Provides deeper functionality and enterprise-grade support/guarantees.

    For details on brand certification for the Open Source edition or consulting for the Professional edition, visit the License Details page.

  6. Understand qData Licensing (Open Source vs. Professional)

    develop

    qData is available in two versions:

    • Open Source Edition: Designed for low-cost entry and learning.
    • Professional Edition: Designed for enterprise-grade depth, support, and reliability.

    For details regarding brand authorization for the Open Source edition or consulting for the Professional edition, visit the Licensing Policy page.

  7. Choose an execution engine for ETL and Data Development

    develop

    qData allows you to select different execution engines and schedulers based on your specific workload requirements:

    ETL Execution Engines

    • DataX: Best for lightweight, single-node data synchronization, related data acquisition, and data import tasks.
    • Spark: Best for large-scale offline processing and complex computing tasks.

    Scheduling Methods

    • Quartz: Suitable for lightweight, local periodic scheduling of data development and metadata collection tasks.
    • DolphinScheduler: Suitable for centralized task orchestration, complex dependency management, and visual task orchestration.

  8. Supported Databases and Execution Engines

    develop

    qData supports a variety of databases and execution engines depending on the task requirements:

    Supported Databases:

    • MySQL
    • DM8 (Dameng)
    • Oracle
    • SQL Server
    • Kingbase8
    • Doris

    Execution Engines for ETL & Data Development:

    • DataX: Best for lightweight data synchronization, single-machine operation, and data import/association tasks.
    • Spark: Best for large-scale offline processing and complex distributed computing tasks.

    Schedulers:

    • Quartz: Suitable for lightweight, single-machine deployment and simple periodic tasks.
    • DolphinScheduler: Suitable for centralized task orchestration, dependency management, and complex scheduling governance.

  9. Core Modules and Features of qData

    develop

    The qData platform is composed of several modular functional areas:

    • Data Integration (ETL): Visual configuration of data access, cleaning, transformation, and output using DataX or Spark engines.
    • Data Development: SQL-based data processing task development with Quartz or DolphinScheduler orchestration.
    • Data Modeling: Supports data standards, warehouse layering, domain partitioning, topic planning, logical modeling, and standard data elements.
    • Metadata Management: Viewing field structures, version management, metadata comparison, and managing collection tasks.
    • Data Quality: Rule-based data quality inspection (integrity, uniqueness, validity) and processing.
    • Data Assets: Cataloging, tagging, and querying of data assets (tables, fields, etc.).
    • Data Query: Online SQL querying of data sources with support for result export.
    • Data Service: Encapsulating tables or SQL results into API services with online testing and call logging.
    • AI Intelligent Querying: Natural language querying (Text2SQL) and intelligent chart generation.
    • System Management: Management of users, roles, menus, departments, and logs.

  10. Understand the qData technology stack

    develop

    qData uses a frontend-backend separated architecture:

    Backend

    • Framework: Spring Boot
    • Security: Spring Security
    • ORM: MyBatis-Plus
    • Storage: MySQL, PostgreSQL, DM8, KingbaseES
    • Caching/Locking: Redis
    • Messaging: RabbitMQ

    Frontend

    • Framework: Vue 3
    • UI Components: Element UI
    • Build Tool: Vite

    Third-Party Data Tools

    • Orchestration/Scheduling: DolphinScheduler, Quartz
    • Data Sync: DataX
    • Distributed Computing: Spark
    • Data Modeling/Storage: Hive, HBase

  11. Run qData locally from source code

    develop

    Developers can run the qData source code locally for daily development and feature integration testing. In this setup, the qData source code is run locally while dependency components (like databases and queues) are managed via Docker Compose.

    https://community.qdata.tech/docs/deploy/build-from-source.html

  12. Deploy qData via Docker Compose

    develop

    For beginners, feature demos, or test environments, use Docker Compose deployment. This method starts all components—including the scheduler, database, message queue, Spark, Flink, and qData source code—with a single command.

    https://community.qdata.tech/docs/deploy/docker-compose-deployment.html