Qlib-Server Documentation

Qlib-Server is a centralized data management system for Qlib that enables 'online' mode. It uses a Client/Server framework based on WebSockets to allow bidirectional, asynchronous communication.

Key components of the workflow include:

1. Request Handling: The server listens for client requests via WebSockets. It uses a RequestListener to parse requests and identifies identical requests from different clients to prevent redundant data generation.
2. Task Queuing: Parsed tasks are submitted to a RabbitMQ pipe in a channel named task_queue.
3. Concurrency Control: The server uses Redis to track session IDs of clients requesting the same data and employs Redis_Lock to prevent IO conflicts during simultaneous read/write operations.
4. Data Processing: A DataProcessor consumes tasks from RabbitMQ. It uses qlib.data.Provider to perform calculations for three specific task types: Calendar, Instruments, and Features.
5. Result Delivery: Once processing is complete, results (either raw data or URIs) are published to a message_queue. A RequestResponder then consumes these results and sends them back to the corresponding clients.

To use Qlib in Online mode via Qlib-Server, you must initialize qlib with a specific configuration dictionary. This configuration directs providers (calendar, feature, etc.) to use remote settings and specifies the connection details for the Flask server.

Key configuration notes:

• Set remote: True in the kwargs for calendar_provider and feature_provider to use the server-side cache.
• Set expression_cache, dataset_cache, and calendar_cache to None to ensure the client uses the server_cache (this disables local writing access).
• Provide the flask_server address and flask_port to connect to the running Qlib-Server instance.

import qlib

ONLINE_CONFIG = {
    # data provider config
    "calendar_provider": {"class": "LocalCalendarProvider", "kwargs": {"remote": True}},
    "instrument_provider": "ClientInstrumentProvider",
    "feature_provider": {"class": "LocalFeatureProvider", "kwargs": {"remote": True}},
    "expression_provider": "LocalExpressionProvider",
    "dataset_provider": "ClientDatasetProvider",
    "provider": "ClientProvider",
    # config it in user's own code
    "provider_uri": "127.0.0.1:/",
    # cache
    # Using parameter 'remote' to announce the client is using server_cache, and the writing access will be disabled.
    "expression_cache": None,
    "dataset_cache": None,
    "calendar_cache": None,
    "mount_path": "/data/stock_data/qlib_data",
    "auto_mount": True,  # The nfs is already mounted on our server[auto_mount: False].
    "flask_server": "127.0.0.1",
    "flask_port": 9710,
    "region": "cn",
}

qlib.init(**ONLINE_CONFIG)
# Note: Ensure ONLINE_CONFIG is passed to qlib.init, not client_config as in the original snippet if client_config is undefined

You can deploy Qlib-Server in Azure using the azure_manager.py script. This requires an Azure account and the azure-cli installed.

1. Configure Azure Credentials

Create an azure_conf.yaml file with your subscription and account details:

sub_id: Your Subscription ID
username: azure user name
password: azure password
# The resource group where the VM is located
resource_group: Resource group name

2. Execute Deployment

Run the create_qlib_cs_vm command from the scripts directory. You must provide the server name, client names, admin credentials, and the path to your configuration file.

git clone https://github.com/microsoft/qlib-server
cd qlib-server/scripts
python azure_manager.py create_qlib_cs_vm \
    --qlib_server_name test_server01 \
    --qlib_client_names test_client01 \
    --admin_username test_user \
    --ssh_key_value ~/.ssh/id_rsa.pub \
    --size standard_NV6_Promo\
    --conf_path azure_conf.yaml

You can initialize Qlib in online mode by passing a YAML configuration file to qlib.init_from_yaml_conf(). This method is useful for managing complex provider and cache settings externally.

Configuration Schema:

calendar_provider: 
    class: LocalCalendarProvider
    kwargs: 
        remote: True
feature_provider:
    class: LocalFeatureProvider
    kwargs: 
        remote: True
expression_provider: LocalExpressionProvider
instrument_provider: ClientInstrumentProvider
dataset_provider: ClientDatasetProvider
provider: ClientProvider
expression_cache: null
dataset_cache: null
calendar_cache: null

provider_uri: 127.0.0.1:/  # NFS server path (host:data_dir)
mount_path: /data/stock_data/qlib_data
auto_mount: True         # Automatically mount provider_uri to mount_path
flask_server: 127.0.0.1  # Data service host/ip
flask_port: 9710         # Data service port

Windows Note: When using Windows, the mount_path must be a non-existent path that is not a root path. Use drive letters like H or i. Avoid paths like C or C:/user/name.

import qlib
qlib.init_from_yaml_conf("qlib_clinet_config.yaml")
from qlib.data import D
ins = D.list_instruments(D.instruments("all"), as_list=True)

RabbitMQ acts as a task queue to separate request handling from data generation. It does not need to be on the same server as Qlib-Server.

Installation

1. Import the signing key and add the repository to apt sources.
2. Install rabbitmq-server.
3. Enable and start the service using systemctl or service.

Configuration

• Create Admin User: Use rabbitmqctl to create an administrator. By default, the system uses guest/guest.
• Web Management Console: Enable the management plugin to manage queues via a web browser at <your rabbitmq host>:15672.

# Install RabbitMQ
echo 'deb http://www.rabbitmq.com/debian/ testing main' | sudo tee /etc/apt/sources.list.d/rabbitmq.list
wget -O- https://www.rabbitmq.com/rabbitmq-release-signing-key.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get install rabbitmq-server

# Start service (Systemctl)
sudo systemctl enable rabbitmq-server
sudo systemctl start rabbitmq-server

# Create admin user
sudo rabbitmqctl add_user admin <your password>
sudo rabbitmqctl set_user_tags admin administrator
sudo rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"

# Enable web management
sudo rabbitmq-plugins enable rabbitmq_management

To build and run Qlib-Server as a Docker container, ensure Docker is installed, then clone the repository and use the provided Dockerfile in docker_support/.

You must provide several --build-arg values during the build process to configure the data path, RabbitMQ, Redis, and Flask server settings. After building, run the container mapping the Flask port (default 9710).

git clone https://github.com/microsoft/qlib-server
cd qlib-server
sudo docker build -f docker_support/Dockerfile -t qlib-server \
    --build-arg QLIB_DATA=/data/stock_data/qlib_data \
        QUEUE_HOST=rabbitmq_server \
        QUEUE_USER=rabbitmq_user \
        QUEUE_PASS=rebbitmq_password \
        MESSAGE_QUEUE=message_queue \
        TASK_QUEUE=task_queue \
        REDIS_HOST=redis_server \
        REDIS_PORT=6379\
        REDIS_DB=1 \
        FLASK_SERVER_HOST=127.0.0.1 \
        QLIB_CODE=/code
sudo docker run -p 9710:9710 qlib-server