Alibaba Cloud Log Service (SLS) Python SDK

The aliyun-log-python-sdk is an open-source Python interface for Alibaba Cloud Log Service (SLS) Rest APIs.

Key capabilities include:

• Full API Coverage: Encapsulates all Rest APIs for management, data operations, and consumer groups.
• High-level Consumer Group Support: Advanced classes for managing consumer groups.
• Python Logging Integration: Includes a Handler for the standard Python logging module.
• High-level Operations: Supports automatic pagination, automatic retries for unfinished tasks, copying Project/Logstore configurations, adjusting Shard read/write counts, and resource usage monitoring.
• Data Migration & ETL: Supports Elasticsearch data migration and high-speed data ETL (Extract, Transform, Load) across logstores based on shard or time, supporting concurrent batch or continuous transformations.

MigrationManager automatically creates indices in the target SLS logstore based on Elasticsearch data types.

Core Datatypes

Complex Datatypes

The KV parser supports various configurations to handle different log formats and avoid naming collisions with built-in log fields.

Custom Separators

By default, the parser uses the equals sign (=) to separate keys and values. You can change this using the extract_kv_sep parameter. You can also provide a regular expression to support multiple separators (e.g., both = and :).

Handling Field Name Collisions

If your log keys (e.g., level) conflict with Aliyun Log Service's built-in fields, you can use prefixes or suffixes to differentiate them:

• Use buildin_fields_prefix or buildin_fields_suffix to modify the built-in system fields.
• Use extract_kv_prefix or extract_kv_suffix to modify the extracted KV fields.

Consumer Groups provide a high-level, scalable, and reliable way to consume logs in real-time. While advanced integration patterns (like Splunk integration) are available via tutorials, you can manage consumer groups using these basic interfaces:

• list_consumer_group: List current consumer groups.
• create_consumer_group: Create a new consumer group.
• update_consumer_group: Update settings like delay or consumption order.
• delete_consumer_group: Delete a consumer group.
• get_check_point: Retrieve the current consumption progress (Checkpoint).
• update_check_point: Store and update the consumption progress to ensure reliable processing.

For advanced log consumption (e.g., building a custom consumer), you can manage the underlying storage structure and the position in the log stream.

Shards: Control the scale of your logstore by using list_shards, split_shard (to increase throughput), and merge_shard (to reduce costs/complexity).

Cursors: To track your position in a logstream for continuous reading, use cursor management methods:

• get_cursor: Get a cursor for a specific time.
• get_cursor_time: Get a cursor based on a timestamp.
• get_previous_cursor_time: Retrieve the previous cursor time.
• get_begin_cursor: Get the starting cursor.
• get_end_cursor: Get the latest cursor.

MigrationManager uses the Python logging module to record execution status. These logs are automatically uploaded to a special logstore named internal-es-migration-log within your target SLS project.

To track a specific migration, find the migration ID (e.g., c34d9636f8934cc18b9727263c476b66) printed in the console and search for it in SLS using the tag __migration__.

Key Log Fields:

• total: Total number of tasks.
• finished: Number of completed tasks.
• failed: Number of failed tasks.
• dropped: Number of dropped tasks.
• checkpoint: Indicates the last processed _id and offset for a specific shard, used for resumption.

The aliyun.log.QueuedLogHandler provides a feature to automatically parse JSON or KV formatted strings within log messages into searchable fields in Alibaba Cloud Log Service. This allows you to pass Python dictionaries directly to the logger, and the handler will extract the keys and values as individual log fields instead of treating the entire dictionary as a single string message.

To enable this, you must set the extract_json parameter to True during configuration.

# Example of passing a dictionary that will be parsed into fields
response_data = {'name': '小明', 'score': 100.0}
logger.info(response_data)

If your extracted JSON keys conflict with Alibaba Cloud Log Service built-in field names (e.g., a key named level), you can use prefixes or suffixes to rename them.

• To modify the extracted keys: Use extract_json_prefix or extract_json_suffix.
• To modify the system built-in fields: Use buildin_fields_prefix or buildin_fields_suffix.