tdx2db

repository·main·Indexed 43 days ago

https://github.com/jing2uo/tdx2db

A tool to import TongdaXing (TDX) market data into local databases such as DuckDB or ClickHouse. It supports daily K-lines, 1-minute intraday data, and automatic calculation of forward and backward adjustment factors for quantitative research.

Tokens
3.7K
Snippets
15
Records
18
Agent score
57%

What's inside tdx2db

  1. Understand tdx2db database tables and views

    main

    The database uses a naming convention where raw_ prefixes indicate base data and v_ prefixes indicate processed views.

    Base Tables (raw_)

    • _meta: Schema version and metadata (current: v6.0).
    • raw_kline_daily: Daily K-lines (includes index/sector rise/fall counts).
    • raw_kline_1min: 1-minute K-lines.
    • raw_basic_daily: Stock/ETF previous close, turnover rate, and market cap.
    • raw_adjust_factor: Post-adjustment factors.
    • raw_gbbq: Capital changes.
    • raw_holidays: Holiday calendar.
    • raw_symbol_class: Symbol classification (stock, index, etf, etc.).
    • raw_symbol_name: Online symbol names.
    • raw_tdx_blocks_info: Online sector/concept/industry info.
    • raw_tdx_blocks_member: Sector membership relations.

    Processed Views (v_)

    Views are named using the pattern v_<class>_<fq>, where <class> is stock or etf, and <fq> is the adjustment type: bfq (none), qfq (forward), or hfq (backward).

    • v_stock_{bfq,qfq,hfq}: Stock daily K-lines.
    • v_etf_{bfq,qfq,hfq}: ETF daily K-lines.

    Note: Stock prices are rounded to 2 decimal places; ETF prices are rounded to 3 decimal places.

    -- Query stock with forward adjustment (qfq)
    select * from v_stock_qfq where symbol='sz000001' order by date;
    
    -- Query ETF with backward adjustment (hfq)
    select * from v_etf_hfq where symbol='sh510300' order by date;
  2. Initialize tdx2db with historical data

    main

    To perform a full historical import, you must first download the Shanghai-Shenzhen-Beijing Daily Data Complete Package from TongdaXing (TDX).

    1. Prepare the data files

    Linux / macOS:

    mkdir -p vipdoc
    wget https://data.tdx.com.cn/vipdoc/hsjday.zip && unzip -q hsjday.zip -d vipdoc
    
    # If files are extracted in a nested structure like sh\lday\sh000001.day, rename them:
    # cd vipdoc && for f in *.day; do mv "$f" "${f##*\\}"; done

    Windows PowerShell:

    Invoke-WebRequest -Uri "https://data.tdx.com.cn/vipdoc/hsjday.zip" -OutFile "hsjday.zip"
    Expand-Archive -Path "hsjday.zip" -DestinationPath "vipdoc" -Force

    2. Run the initialization

    Use the init command with the --dburi and --dayfiledir flags to import the data into your chosen database.

    For DuckDB:

    tdx2db init --dburi 'duckdb://./tdx.db' --dayfiledir ./vipdoc

    For ClickHouse:

    tdx2db init --dburi 'clickhouse://localhost' --dayfiledir ./vipdoc

    ClickHouse default connection: user=default, password="", port=9000, http_port=8123, database=default

  3. Perform incremental updates with cron

    main

    After the initial init, use the cron command to update daily data, capital changes, holiday calendars, and online symbol/sector information. This command also recalculates previous close prices and adjustment factors.

    Standard update (Daily data):

    tdx2db cron --dburi 'duckdb://tdx.db'

    Update with 1-minute intraday data: Use the --min flag to download and import 1-minute K-line data.

    tdx2db cron --dburi 'duckdb://tdx.db' --min

    Important notes for 1-minute data:

    • Downloading and importing 1-minute data is time-consuming due to large data volumes.
    • TongdaXing does not provide historical 1-minute data; you must provide it yourself if needed.
    • If the update interval exceeds 30 days, you must manually fill the gap before continuing.
    • Changes in stock codes will not trigger historical record processing.
  4. Install tdx2db via binary

    main

    Download the appropriate compressed package for your platform from the releases page. After extracting, move the binary to your system path to use it globally.

    Note: On Windows, macOS, and Linux arm64, the tool will skip the download and import of 1-minute intraday data by default, but daily data will be processed normally.

    sudo mv tdx2db /usr/local/bin/ && tdx2db -h
  5. Understand the TempDir and VipdocDir hierarchy

    main

    The tdx2db package manages two primary directory locations for temporary and cached data:

    • TempDir: The primary location for temporary files. It defaults to the system's cache directory (via utils.GetCacheDir()).
    • VipdocDir: A subdirectory located within TempDir (specifically TempDir/vipdoc) used for storing VIP documentation data.

    If you change TempDir using OverrideTempDir, VipdocDir is automatically updated to maintain the correct hierarchy.

  6. Configure the temporary directory with --temp

    main

    The --temp flag allows you to override the parent directory used for temporary files. This is useful if your system's default $TMPDIR (like /tmp on many Linux systems) is a tmpfs with limited capacity and you need to use a disk-backed directory instead.

    This is a persistent flag, meaning it can be used with any command.

    tdx2db --temp /path/to/large/disk/tmp init --dburi 'duckdb://./tdx.db' --dayfiledir /path/to/vipdoc/
  7. Reference global flags for tdx2db

    main

    The following flags are available globally for the tdx2db CLI:

    FlagDescription
    --temp <dir>Sets the parent directory for temporary files. If left empty, it uses $TMPDIR.
    -v, --versionPrints the version (matches local build with release).
  8. Troubleshoot database schema version incompatibility

    main

    If you encounter errors regarding database schema versions, it means the version stored in your database file does not match the version required by the current tdx2db binary.

    There are two primary error states:

    1. Incompatible Version: The database has a major version that differs from the current code's model.SchemaMajor.
      • Error message format: 数据库 schema 版本不兼容 (当前库: v%d.x, 需要: v%d.x)
    2. Missing Version: The database does not contain a schema version record.
      • Error message format: 数据库 schema 版本缺失 (需要: v%d.x)

    In both cases, you should visit the official releases page to find migration instructions: https://github.com/jing2uo/tdx2db/releases

  9. Run incremental updates via the Cron function

    main

    The Cron function is used to perform incremental updates of the A-share market data database. It automates the process of connecting to the database, initializing the schema, building a work plan for the current day, and executing the necessary update tasks.

    Workflow steps:

    1. Database Connection: Connects to the database using the provided dbURI.
    2. Schema Management: Initializes the database schema and verifies the schema version.
    3. Work Plan Generation: Uses workflow.BuildWorkPlan to determine which updates are required for the current day.
    4. Task Execution: Executes registered update tasks via a TaskExecutor using the generated plan.

    Parameters:

    • ctx: A context.Context for managing lifecycle and cancellation.
    • dbURI: The connection string for the database (e.g., duckdb://[path]).
    • min: A boolean flag. If true, it triggers a 'min' mode (likely related to minimal data updates or specific task constraints).

    Note: If the generated plan indicates no work is needed (plan.AnyNeeded() is false), the function exits successfully without performing any tasks.

    // Example conceptual usage of the Cron function
    err := cmd.Cron(ctx, "duckdb://path/to/your/db.duckdb", true)
    if err != nil {
        log.Fatalf("Cron failed: %v", err)
    }
  10. Send requests using OnlineClient.Call

    main

    The Call method is the primary way to send a request to the TDX server. It sends a message with a specific msgID and a byte slice body. It handles the packet framing, transmission, and automatic decompression of the response.

    • msgID: A uint16 identifying the specific TDX protocol message type.
    • body: The raw byte payload for the request.

    Returns the decompressed response body as a []byte or an error.

    // Example usage of Call
    // msgID and body would be determined by the specific TDX protocol being used
    response, err := client.Call(0x1234, []byte{0x01, 0x02, 0x03})
    if err != nil {
        // handle error
    }
    // response contains the unzipped data
  11. Initialize and connect an OnlineClient

    main

    To interact with online TDX data, use the OnlineClient type. You must first initialize it with NewOnlineClient() and then establish a connection using either Connect() or ConnectStandard().

    • Connect(): Attempts to connect to the primary TDX market hosts (macHosts).
    • ConnectStandard(): Attempts to connect to standard TDX market hosts (including Shenzhen, Shanghai, and Beijing dual-line main stations).

    Both methods automatically perform a latency check across available hosts and select the fastest available server to connect to.

    package main
    
    import "github.com/jing2uo/jing2uo/tdx2db/tdx"
    
    func main() {
        client := tdx.NewOnlineClient()
        
        // Connect to primary hosts
        err := client.Connect()
        if err != nil {
            panic(err)
        }
        defer client.Close()
    
        // Use the client... 
    }