repository·main·Indexed 42 days ago
https://github.com/volcengine/volc-sdk-pythonA Python SDK providing programmatic access to Volcengine cloud services, including Visual Intelligence, NLP, BioOS, and Log Service (TLS). It features tools for text privatization using local differential privacy for LLM fine-tuning, as well as comprehensive log management capabilities including project and topic creation, index definition, and high-throughput log writing via put_logs_v2.
Before using the SDK, ensure the following requirements are met:
Text privatization is designed to mitigate privacy leakage risks when uploading private training data for Large Language Model (LLM) fine-tuning.
It utilizes local differential privacy techniques to perturb training data locally before it is used. Currently, the feature only supports text classification tasks and employs a dynamic privacy budget allocation mechanism based on token contribution to balance utility and privacy.
The TLS (Log Service) provides a ConsumerGroup feature to simplify log consumption. Unlike the standard ConsumeLogs OpenAPI, which requires manual management of Shard IDs, Cursors, and load balancing, ConsumerGroup automatically handles shard allocation, load balancing, and fault tolerance across multiple consumers.
To consume logs, you must implement a custom LogProcessor by inheriting from the LogProcessor class and overriding its process method. This method receives a LogGroupList containing the logs to be processed.
from volcengine.tls.consumer.consumer import LogProcessor
from volcengine.tls.log_pb2 import LogGroupList
class MyLogProcessor(LogProcessor):
def process(self, topic_id: str, shard_id: int, log_group_list: LogGroupList):
# Implement your business logic here
for log_group in log_group_list.log_groups:
for log in log_group.logs:
for content in log.contents:
print(f"{content.key}: {content.value}")To use BioOS, initialize the BioOsService class. You can specify the endpoint and region if the defaults are insufficient. If you have not configured your credentials in $HOME/.volc/config, use set_ak() and set_sk() to provide your Access Key and Secret Key manually.
from __future__ import print_function
from volcengine.bioos.BioOsService import BioOsService
if __name__ == '__main__':
# set endpoint/region here if the default value is unsatisfied
bioos_service = BioOsService(endpoint='endpoint', region='region')
# call below method if you don't set ak and sk in $HOME/.volc/config
bioos_service.set_ak('ak')
bioos_service.set_sk('sk')
params = {}
resp = bioos_service.list_workspaces(params)
print(resp)If you are using a sub-account (RAM user), ensure it has the necessary permissions to access BioOS services.
AccessKeyFullAccess permissions or a higher-level permission like AdministratorAccess. If not, use the main account to grant these permissions to the sub-account.The SDK requires an Access Key ID (AK) and a Secret Access Key (SK) to authenticate requests. You can provide these credentials using one of three methods:
Pass the credentials directly to the service client instance using set_ak() and set_sk().
Set the following environment variables in your shell or system:
VOLC_ACCESSKEYVOLC_SECRETKEYCreate a JSON file at ~/.volc/config (under your HOME directory) with the following structure:
# Method 1 Example
iam_service = IamService()
iam_service.set_ak('Your AK')
iam_service.set_sk('Your SK')The volcengine package requires Python version 2.7 or higher. You can install it directly using pip.
pip install volcengineTo interact with the Volcengine Log Service (TLS), you must first initialize a TLSService client. It is recommended to use environment variables for authentication to avoid hardcoding AccessKeys.
Important: The endpoint environment variable must NOT include the protocol prefix (e.g., use tls-cn-beijing.ivolces.com instead of https://tls-cn-beijing.ivolces.com).
If you are using STS (Security Token Service), provide the temporary access_key_id and access_key_secret, and ensure VOLCENGINE_TOKEN is set. For non-STS usage, the token part should be empty.
import os
from volcengine.tls.TLSService import TLSService
# Ensure endpoint does not contain http:// or https://
endpoint = os.environ["VOLCENGINE_ENDPOINT"]
region = os.environ["VOLCENGINE_REGION"]
access_key_id = os.environ["VOLCENGINE_ACCESS_KEY_ID"]
access_key_secret = os.environ["VOLCENGINE_ACCESS_KEY_SECRET"]
tls_service = TLSService(endpoint, access_key_id, access_key_secret, region)To build and view the BioOS technical documentation locally using Sphinx, follow these steps:
volcengine/bioos/doc directory and generate the HTML files:make html${port} with your desired port number):python -m http.server ${port}http://127.0.0.1:${port} to view the documentation.# Generate HTML
make html
# Start server
python -m http.server ${port}To use the text privatization features, ensure you are using Python 3.7 or higher. Install the SDK using pip, and then install the required machine learning dependencies.
# Install the SDK
pip install --user volcengine
# Or upgrade if already installed
pip install --upgrade volcengine
# Install required dependencies
pip install numpy typing_extensions sentencepiece torch~=1.13 transformers~=4.30Example scripts for visual services are located in the volcengine/example/visual directory. To run a specific example (e.g., the bank card OCR example), execute the script from the repository root using Python.
python volcengine/example/visual/example_bank_card.pyInstall the SDK using pip. You can install it for your user or upgrade an existing installation.
To install for the current user:
pip install --user volcengineTo upgrade an existing installation:
pip install --upgrade volcengine