DOCSEARCHbyAlgolia
Submit Docs

DolphinDB Tutorials (CN)

repository·master·Indexed 18 days ago

https://github.com/dolphindb/tutorials_cn

A collection of tutorials, guides, and examples for DolphinDB, a high-performance distributed database system. Content covers installation, deployment, database design, streaming computation, system administration, and industry applications in finance and IoT. Includes detailed documentation on ACL and security, user/group role management, permission categories, HTTPS configuration, SSO implementation, and the Anomaly Detection Engine for real-time stream monitoring. Note: This repository is no longer maintained as of April 1, 2024 (versions 2.00.12/3.00.0 and later).

Tokens
286.8K
Snippets
673
Records
943
Agent score
62%

What's inside dolphindb-tutorials_cn

  1. Overview of Debezium + Kafka Real-time Sync from MySQL to DolphinDB

    master

    This tutorial describes a real-time data synchronization pipeline that moves data from a MySQL database to DolphinDB using Debezium and Kafka.

    The architecture follows this flow:

    1. MySQL acts as the source.
    2. Debezium (running on Kafka Connect) captures Change Data Capture (CDC) events from MySQL.
    3. Kafka serves as the real-time distributed message broker/storage.
    4. Schema Registry manages the data schemas.
    5. Kafka-DolphinDB Connector consumes data from Kafka and writes it into DolphinDB.

  2. Overview of DolphinDB Low-Code Platform nodes

    master

    The DolphinDB low-code platform on Node-RED abstracts DolphinDB functions into visual nodes. The available nodes are categorized into three main groups:

    • File Operations: Includes nodes like Data Loading to load local files into DolphinDB in-memory tables.
    • Data Cleaning: Provides nodes for Format Conversion, NULL Filling, Data Deduplication, and Data Normalization (e.g., Max-Min normalization).
    • DolphinDB Functional Nodes: Includes advanced features such as Data Ingestion (入库), Downsampling (降采样), Schema-less Writing (无模式写入), Stream Table Creation (流表创建), and Streaming Data Engine (流数据引擎).

  3. Overview of Factor Calculation with DolphinDB

    master

    DolphinDB is a high-performance time-series database that integrates distributed computing, real-time stream computing, and distributed storage. It is designed to handle the full lifecycle of quantitative factor mining, including:

    • Data Handling: Processing various frequencies (tick, snapshot, minute, daily) and scales (GB to PB).
    • Factor Development: Using multiple programming paradigms (functional, imperative, vector, and SQL) and over 1,400 built-in functions, specifically optimized for window operations.
    • Research to Production: Bridging the gap between research and live trading via built-in data replay and streaming incremental calculation engines.
    • Engineering Requirements: Addressing code management, unit testing, parallel computing, memory management, and permission control for multi-user environments.

  4. Overview of DolphinDB function libraries

    master

    DolphinDB provides an extensive library of built-in functions categorized by domain. Key functional areas include:

    • Mathematical & Statistical: Math functions, statistical functions, distribution functions, and hypothesis testing.
    • Data Processing: Data manipulation, window functions, and string functions.
    • Advanced Computing: High-order functions, metaprogramming, distributed computing, and stream computing.
    • System & Logic: Logic functions, time functions, performance monitoring, and user permission management.
    • Specialized: Machine learning functions and scheduled task functions.

  5. Introduction to the DolphinDB AMD Plugin

    master

    The AMD plugin allows DolphinDB to interface with the Huaru AMD (High-speed Market Data) distribution platform via the AMA (AMD Market Data API). It enables consumers to receive transcoded market data directly into DolphinDB.

    Supported Data Sources include:

    • Stock: snapshot, execution, order
    • Index: index
    • Order Queue: orderQueue
    • Fund: fundSnapshot, fundExecution, fundOrder
    • Bond: bondSnapshot, bondOrder, bondExecution

    Compatibility Notes:

    • Supported DolphinDB Server versions: Stable 1.30.X and Latest 2.00.X.
    • The plugin version must match your DolphinDB server version. For example, using AMD Plugin release200.10 requires DolphinDB 2.00.10 server.
    • The plugin supports AMA versions 3.9.8 and 4.0.1. Ensure the plugin files in the bin/ directory match your required AMA version.

  6. Overview of components in the Factor Calculation Platform

    master

    SQL Server

    Acts as the original data source for the architecture.

    dataX

    An offline heterogeneous data synchronization tool. In this platform, the dolphindbWriter plugin is used to import data from SQL Server into DolphinDB.

    DolphinDB

    A high-performance time-series data processing framework. It serves as the primary tool for factor calculation and storage. It allows predefined factor functions to be called from Python using the DolphinDB Python API.

    Celery

    A distributed asynchronous task queue for Python. It is used as the task scheduling framework. It integrates with Redis (acting as both the message broker and result backend) to handle asynchronous requests and manage task execution units (Workers).

  7. Use the ODBC Plugin for Data Synchronization

    master

    The ODBC plugin allows DolphinDB to perform offline data synchronization with external databases. You can use it to:

    • Import data from external databases (e.g., Oracle) into DolphinDB.
    • Export DolphinDB in-memory tables to external databases.
    • Export data from DolphinDB distributed tables by first retrieving them as in-memory tables.

    General Workflow:

    1. Load the plugin using loadPlugin.
    2. Establish a connection using odbc::connect.
    3. Use odbc::query to import data.
    4. Use odbc::append to export data.

  8. Predict Taxi Trip Duration using DolphinDB Machine Learning

    master
    This tutorial demonstrates how to use DolphinDB's high-performance time-series database and analytical capabilities to predict taxi trip durations. The workflow involves using DolphinDB for data preprocessing (including PCA for location information and KMeans clustering), feature engineering, and training an XGBoost model. The solution is designed to handle both batch training on historical data and real-time prediction using the DolphinDB Streaming Data Engine to process continuous streams of ride-hailing order data.

  9. DolphinDB Performance Monitoring Methods

    master

    DolphinDB provides three primary ways to monitor system performance:

    1. Built-in Functions: Use functions like getPerf, getClusterPerf, and getJobStat directly within DolphinDB.
    2. Web Interface: Access performance metrics via the built-in Web UI.
    3. Third-party Systems: Integrate with external monitoring tools like Prometheus and Grafana via APIs.

    For detailed usage of the first two methods, refer to the official User Manual.

  10. Backup and Restore Compatibility and Scenarios

    master

    DolphinDB backup and restore functionality is designed for backing up/restoring entire clusters, databases, tables, or partitions, and for large-scale data migration.

    Supported Versions:

    • 130 series: 1.30.20 and later
    • 200 series: 2.00.8 and later

    Key Improvements (v2.00.8/1.30.20+):

    • Distributed file copying for improved performance.
    • High-level encapsulated functions (backupDB, backupTable, backup) for ease of use.
    • Enhanced backup file verification capabilities.

    Important Usage Notes:

    • Task Submission: For large datasets, it is highly recommended to use submitJob to run backup/restore tasks in the background rather than running them directly in the client. Background task status can be checked using getRecentJobs.
    • Locking Behavior: During backup, DolphinDB applies locks (on the database for backupDB or the table for backupTable). Data remains readable but becomes non-writable during the lock period. To prevent write failures, schedule backups when no write tasks are active.
    • Storage Limitation (v2.00.8): Backup files must be placed in a directory on the same machine where the data node performing the backup task resides.

  11. Compare DolphinDB Python Parser performance with DolphinDB Scripts and Python

    master

    The DolphinDB Python Parser is designed for quantitative finance scenarios, offering a balance between the high performance of DolphinDB Scripts and the ease of use of Python.

    Key performance characteristics:

    • Vs. Python API: The Python Parser reduces network overhead by allowing direct access to data within the DolphinDB database and automatically implements parallel computing for functions like groupby.
    • Vs. DolphinDB Scripts: While DolphinDB Scripts are generally faster for highly optimized tasks, the Python Parser provides significantly better compatibility with common Python syntax, lowering the learning curve.
    • Vs. Standard Python: In factor calculation scenarios, the Python Parser can achieve over a 5x performance improvement compared to standard Python multi-processing frameworks.

    Typical performance ratios (DolphinDB Scripts / Python Parser):

    • Daily K-line (Full market): ~0.515
    • Snapshot Market Data (Regression coefficient): ~0.019
    • Tick Data (Closing trade ratio): ~0.076

  12. Real-time Minute Capital Flow Calculation Scenario

    master

    This tutorial demonstrates a low-latency solution for calculating minute-level capital flows using DolphinDB's streaming data framework. The system processes tick-by-tick transaction data to derive indicators for large and small buy/sell orders within a 1-minute rolling window.

    Data Source Schema

    The input data (SSE 2020 tick data) uses the following structure:

    NameTypeComment
    SecurityIDSYMBOLStock Code
    MarketSYMBOLExchange
    TradeTimeTIMESTAMPTransaction Time
    TradePriceDOUBLETransaction Price
    TradeQtyINTTransaction Quantity
    TradeAmountDOUBLETransaction Amount
    BuyNumINTBuy Order ID
    SellNumINTSell Order ID