DOCSEARCHbyAlgolia
Submit Docs

investment_data

repository·main·Indexed 23 days ago

https://github.com/chenditc/investment_data

A project providing validated investment and stock market data formatted for the Qlib quantitative investment platform. It manages data ingestion from sources including Tushare, Akshare, Wind, and Yahoo, utilizing Dolt for versioned storage. The repository includes tools for exporting data to Qlib-compatible archives (qlib_bin.tar.gz), managing daily updates via GitHub Actions, and deploying an Actions Runner Controller (ARC) on nvdev2 environments.

Tokens
6.2K
Snippets
18
Records
35
Agent score
78%

What's inside investment_data

  1. Understand the purpose of final_a_stock_limit

    main

    The final_a_stock_limit table tracks stock prices that hit upper or lower limits during trading. This is critical for two main reasons:

    1. Backtesting Accuracy: You cannot execute trades once a price hits a limit price.
    2. Feature Engineering: Hitting a limit price is a significant signal for momentum in financial modeling.

  2. Understand data source prefixes

    main

    The database tables in Dolt use prefixes to identify the source of the data. This helps in understanding the quality and availability of the information:

    • w: Wind (High-quality static data, available up to 2019)
    • c: Caihui (High-quality static data, available up to 2019)
    • ts: Tushare
    • ak: Akshare
    • yahoo: Qlib Yahoo collector
    • final: Final verified and corrected merged data

  3. Understand Dolt table prefixes and data sources

    main

    The Dolt database uses prefixes to identify the source of the data. Understanding these prefixes helps you identify the quality and availability of the data:

    • w: Wind (high quality static data, available only until 2019)
    • c: Caihui (high quality static data, available only until 2019)
    • ts: Tushare
    • ak: Akshare
    • yahoo: Qlib's yahoo collector
    • baostock: Baostock
    • final: Merged final data with validation and correction applied.

  4. How stock price adjustment works (ts_link_table)

    main

    Because different data sources may have different initial dates for a stock, the project uses a ts_link_table to ensure consistent adjusted prices across sources.

    The Logic: When calculating adjusted prices, the first available date's price is used as the base (adjust factor = 1.0). To merge sources, an adjust_ratio is calculated:

    1. If the stock already exists in final_a_stock_eod_price: adjust_ratio = final_a_stock_eod_price.adjust_price / current_data_source.adjust_price
    2. If the stock is new to final_a_stock_eod_price: adjust_ratio = 1

    Validation Rule: To verify the adjustment factor between sources, ensure: data_source_1.adjust_ratio * data_source_1.adjust_price == final_a_stock_eod_price.adjust_price

  5. Understand the data merge and validation logic

    main

    The system merges Wind (w) and Tushare (ts) data sources using a specific mapping logic to ensure consistency in adjusted close prices (adjclose).

    Merge Logic

    1. Baseline: Wind (w) data is used as the baseline. Other sources are validated against it.
    2. Link Date & Ratio: Because adjclose differs between Wind and Tushare, a "link date" (the maximum first valid data point for each source) is used to calculate a ratio. This ratio is stored in a link_table.
      • adj_ratio = link_adj_close / w_adj_close
    3. Final Dataset: Tushare data is appended to the final dataset, with its adjusted close calculated as: ts_adj_close / ts_adj_ratio.

    Validation Logic

    Validation is performed by concatenating Wind and Tushare data and checking:

    1. Absolute Values: Comparison of high, low, open, close, and volume.
    2. Ratio Consistency: Calculating the adjclose conversion ratio using the link date for each stock.
    3. Baseline Verification: Calculating the Wind data adjclose using the link date's ratio and comparing it against the final dataset.

  6. Understand the data merging and validation logic

    main

    The system uses Wind (w) data as the baseline and other sources (like Tushare) for validation and augmentation.

    Merging Logic

    Because Wind and Tushare have different adjclose (adjusted close) values, a link date is used to calculate a ratio to map Tushare values to the Wind standard.

    • The link date is typically the first valid data point for each source (to account for different listing/delisting dates).
    • The ratio is stored in a link_table as: adj_ratio = link_adj_close / w_adj_close.
    • Tushare data is appended to the final dataset where the adjusted close is calculated as: ts_adj_close / ts_adj_ratio.

    Validation Logic

    Validation is performed by joining Wind and Tushare data and checking:

    1. Absolute values for High, Low, Open, Close, and Volume.
    2. The adjclose conversion ratio calculated via the link date.
    3. Whether the w_adj_close calculated from the ratio matches the original Wind data.

  7. Understand the ARC deployment architecture on nvdev2

    main

    The ARC deployment on nvdev2 utilizes a dedicated Actions Runner Controller scale set named investment-arc. The installation script sets up the following components:

    • ARC Controller: Installed in the arc-systems namespace.
    • Runner Scale Set: Installed in the arc-runners namespace.
    • Persistent Volume Claim (PVC): A 100Gi PVC named investment-data-docker-graph is created.

    Data Persistence Mechanism: The investment-data-docker-graph PVC is mounted into a dind (Docker-in-Docker) sidecar at /var/lib/docker. This configuration ensures that Docker images and specific named volumes, such as dolt_update and dolt-vol, remain warm and available across ephemeral runner pods, reducing setup time for subsequent jobs.

  8. Perform a fail-safe repository rollback

    main

    A full repository revert is a high-risk, fail-closed operation. Follow these steps exactly:

    1. Disable the upload workflow: gh workflow disable upload_release.yml --repo chenditc/investment_data.
    2. Verify the state is disabled_manually using gh workflow view.
    3. Mandatory: Wait for all queued or in-progress jobs using the shared Dolt volume to drain.
    4. Perform the full repository revert.
    5. Re-verify that upload_release.yml is still in the disabled_manually state.

    Do not re-enable publication until validator-backed digest pinning and workflow authority are restored. To roll back the monitor itself (not the repo), use ops/investment-data-project-monitor/deploy.sh rollback.

    gh workflow disable upload_release.yml --repo chenditc/investment_data

# Verify state
gh workflow view upload_release.yml --repo chenditc/investment_data --json state --jq .state

  9. Run daily data updates

    main

    Daily updates require a Tushare token. Obtain one from https://tushare.pro/.

    Update only:

    export TUSHARE=<Token>
bash daily_update.sh

    Update and Export (Full Pipeline): To run the update and immediately export to Qlib format using Docker:

    docker run -v /<some output directory>:/output -it --rm chenditc/investment_data \
  bash -lc 'bash daily_update.sh && bash dump_qlib_bin.sh'
    export TUSHARE=<Token>
bash daily_update.sh

  10. Use the processed investment data for Qlib

    main

    To use the pre-processed investment data in your local Qlib environment, follow these steps:

    1. Download the standardized archive (qlib_bin.tar.gz) and the manifest (qlib_bin.manifest.json) from a GitHub Release.
    2. Validate the integrity of the archive using the provided Python script.
    3. Extract the archive into your Qlib data directory.

    Validation Command:

    python3 qlib/validate_archive.py \
  --archive qlib_bin.tar.gz \
  --manifest qlib_bin.manifest.json \
  --expected-tag "<release-tag>" \
  --require-publishable

    Extraction Command:

    tar -zxvf qlib_bin.tar.gz -C ~/.qlib/qlib_data/cn_data --strip-components=1
    wget "https://github.com/chenditc/investment_data/releases/download/<release-tag>/qlib_bin.tar.gz"
wget "https://github.com/chenditc/investment_data/releases/download/<release-tag>/qlib_bin.manifest.json"
python3 qlib/validate_archive.py \
  --archive qlib_bin.tar.gz \
  --manifest qlib_bin.manifest.json \
  --expected-tag "<release-tag>" \
  --require-publishable
tar -zxvf qlib_bin.tar.gz -C ~/.qlib/qlib_data/cn_data --strip-components=1

  11. Add a new stock index to the dataset

    main

    To add a new stock index, you must update three distinct parts of the pipeline:

    1. Index Weight Download: Modify tushare/dump_index_weight.py to dump index info. If Tushare doesn't support the index, write a new script and add it to daily_update.sh.
    2. Price Download: Modify tushare/dump_index_eod_price.py to include the index price.
    3. Export Script: Modify the Qlib dump script (e.g., qlib/dump_index_weight.py) so the index is dumped and renamed to a .txt file for Qlib usage.

  12. How to use the investment data archives

    main

    To use the pre-processed data for Qlib, download the canonical archive and its provenance manifest from a GitHub release, validate them, and extract the contents to your Qlib directory.

    1. Download the qlib_bin.tar.gz and qlib_bin.manifest.json files for your desired <release-tag>.
    2. Validate the archive using the provided Python script to ensure integrity and correctness.
    3. Extract the archive to your local Qlib data directory.
    # 1. Download assets
wget "https://github.com/chenditc/investment_data/releases/download/<release-tag>/qlib_bin.tar.gz"
wget "https://github.com/chenditc/investment_data/releases/download/<release-tag>/qlib_bin.manifest.json"

# 2. Validate
python3 qlib/validate_archive.py \
  --archive qlib_bin.tar.gz \
  --manifest qlib_bin.manifest.json \
  --expected-tag "<release-tag>" \
  --require-publishable

# 3. Extract to qlib directory
tar -zxvf qlib_bin.tar.gz -C ~/.qlib/qlib_data/cn_data --strip-components=1