DOCSEARCHbyAlgolia
Submit Docs

TapData Documentation

website·Indexed 32 days ago

https://docs.tapdata.net

Documentation for TapData, a real-time data integration and ETL platform. Covers data synchronization, task management, and API services across versions 3.x and 4.x, including support for PostgreSQL, Oracle, and MySQL. Includes guides on deploying projects via GitHub Actions, managing Agents, and configuring real-time data hubs in ETL and DaaS modes.

Tokens
417.9K
Snippets
1K
Records
3K
Agent score
52%

What's inside TapData

  1. Overview of TapData Deployment Options

    TapData is available in three primary editions based on deployment and feature needs:

    • TapData Cloud: A cloud-based service for fast deployment and low initial investment.
    • TapData Enterprise: Deployed in local data centers; suitable for high data sensitivity, network isolation, real-time data warehousing, and data migration.
    • TapData Community: An open-source data integration platform providing basic synchronization and transformation, deployable via Docker.

  2. Overview of TapData Cloud real-time data synchronization

    TapData Cloud 2.x integrates database Change Data Capture (CDC) technology with stream computing. This approach opens up the data processing stage during synchronization, allowing users to perform more than simple data replication and implement custom data transformations.

  3. Overview of Real-time Materialized Views in TapData

    Real-time materialized views in TapData allow users to pre-store the results of complex queries across multiple data sources. Unlike traditional views, these are automatically updated whenever the source tables change, ensuring data consistency, timeliness, and significantly faster read performance. This is particularly useful for consolidating fragmented data (e.g., joining orders, users, and products) into a single target collection or table.

  4. Overview of Full Breakpoint Resumption for MongoDB

    Full Breakpoint Resumption is designed for migrating massive datasets (billion-level records) to improve reliability and prevent task failure due to network instability or hardware faults. It works by sharding source data, recording the transmission progress, and caching shards locally. When combined with incremental data, it ensures data accuracy by merging cached full-load data with associated incremental changes before sending them to the target database.

    Prerequisites:

    • This feature currently only supports MongoDB as the source data source.

  5. Overview of TapData data service platform

    TapData is a platform-based data service product providing data replication, transformation, and development capabilities. It is designed to break down data silos and enable fast data delivery through real-time synchronization. The platform supports low-code task creation via a drag-and-drop visual interface to reduce development complexity and shorten project lifecycles.

  6. Overview of Tapdata Cloud Data Replication

    Tapdata Cloud provides data replication functionality to create tasks for 1:1 data copying from a source to a target. It supports both full data synchronization and incremental data replication.

  7. Overview of Data Validation methods

    TapData 2.x provides three primary methods to verify the correctness of synchronized data:

    1. Quick Count (快速count校验): Only compares the row counts of source and target tables. It is extremely fast but does not identify specific differing fields.
    2. Full Field Value Validation (表全字段值校验): Performs a row-by-row comparison of all fields in source and target tables. It identifies all differences but is slower.
    3. Associated Field Value Validation (关联字段值校验): Compares only specific associated fields between source and target tables. It is faster than full field validation.

  8. Overview of Tapdata data flow capabilities

    Tapdata 2.x provides data replication, data development, and data services across a wide range of data sources, including mainstream open-source databases, domestic innovation (Xinchuang) databases, offline files, cloud applications, and lakehouse architectures.

  9. Overview of Tapdata Real-Time Data Platform

    Tapdata is a real-time data platform designed to centralize core enterprise data into a centralized data platform. It provides real-time data to downstream interactive applications, microservices, or interactive analysis via APIs or reverse synchronization. It replaces traditional T+1 data management and complex CDC + Kafka + Flink pipelines with a one-stop real-time synchronization experience.

  10. Overview of TapFlow for Real-time Wide Tables

    TapFlow is a programming framework designed for real-time data replication, processing, and the creation of materialized views. It allows developers to build real-time wide tables by monitoring source table changes via CDC (Change Data Capture) and using Lookup functions to join multiple tables into a single destination document (e.g., in MongoDB). This approach eliminates expensive cross-table joins at query time, reducing latency and improving consistency for high-concurrency APIs.

  11. Overview of TapFlow API development framework

    TapFlow is the API development framework for the TapData real-time data platform. It allows developers to implement real-time replication and development tasks for homogeneous or heterogeneous data sources using APIs and the TapShell command-line tool. It is designed for flexible, automated data integration and streaming processing scenarios, such as building real-time wide tables and real-time materialized views.

  12. Overview of TapData Real-time Data Platform

    TapData is a real-time data platform designed to centralize core enterprise data into a centralized platform. It provides real-time data to downstream interactive applications, microservices, and interactive analysis via APIs or reverse synchronization. It is available in three editions:

    • TapData Cloud: Cloud-based service for rapid deployment and low initial investment.
    • TapData Enterprise: On-premises deployment for sensitive data or network-isolated environments, supporting real-time data warehouses and data exchange.
    • TapData Community: An open-source data integration platform providing basic synchronization and transformation, deployable via Docker.