Alibaba Cloud Log Service (SLS) Go SDK

This project is the Golang programming interface for Alibaba Cloud Log Service (SLS) APIs. It provides a wrapper around the SLS REST API to help Go developers interact with Log Service more efficiently. The SDK consists of three main components:

1. Base API Wrapper: Core implementation of Log Service REST APIs.
2. Golang Producer Library: Used for batch sending data to Log Service. See the Producer documentation.
3. Golang Consumer Library: Used for consuming data from Log Service. See the Consumer documentation.

Increasing the number of goroutines in the thread pool does not linearly increase performance and can be counterproductive. Excessive goroutines can increase CPU usage due to Garbage Collection (GC) overhead.

Recommendation: Adjust the thread pool goroutine count based on the specific machine's performance, but generally, it is best to stick with the default values provided by the SDK.

The producer combines several types of tags into the final LogGroup.LogTags. They are appended in the following order:

1. Dynamic Tags: Passed via the SendLogListWithTags interface. These participate in the aggregation key.
2. Configured Tags: Defined in producerConfig.LogTags. These are fixed for the producer and do not participate in dynamic tag aggregation.
3. Automatic PackId: If producerConfig.GeneratePackId is enabled, a __pack_id__ is appended. This does not participate in aggregation.

Key Behaviors:

• No Deduplication: Tags are appended directly. If keys overlap, both will be kept.
• No Overwriting: If a key exists in both dynamic and config tags, both are preserved.
• Aggregation: Only dynamic tags are used to determine how logs are grouped into batches.

When using the UpdateLogStoreLogs and DeleteLogStoreLogs APIs, keep the following rules in mind:

Time Range Requirements

Both APIs require a time range defined by From and To (Unix seconds, representing the interval [From, To)).

Selection Methods

You can select records to modify or delete using two methods:

1. Query: Uses search expressions (Note: does not support SQL or SPL).
2. RowID: Uses specific record IDs.

Precedence: If both a Query and a RowID are supplied, the RowID takes precedence.

Update Modes

When updating data via UpdateLogStoreLogs, the Data field must be a JSON-encoded string. You can choose between two modes:

• partial: Preserves existing fields and only updates the specified ones.
• full: Replaces the entire record with the new data.

To use the producer, follow these steps:

1. Configure the Producer: Use producer.GetDefaultProducerConfig() to get a default configuration and set your Endpoint and CredentialsProvider (e.g., using sls.NewStaticCredentialsProvider).
2. Initialize: Call producer.NewProducer(producerConfig) to create the instance.
3. Start: Call producerInstance.Start() to begin the background goroutine that monitors logs and sends them to the LogHub service.

producerConfig := producer.GetDefaultProducerConfig()
producerConfig.Endpoint = os.Getenv("Endpoint")
provider := sls.NewStaticCredentialsProvider(os.Getenv("AccessKeyID"), os.Getenv("AccessKeySecret"), "")
producerConfig.CredentialsProvider = provider

producerInstance, err := producer.NewProducer(producerConfig)
if err != nil {
   panic(err)
}

producerInstance.Start() // Starts the background processing

The producer's throughput is significantly impacted by the number of CPU cores available to the Go runtime. To increase throughput, ensure the application is allowed to use more CPU cores by setting runtime.GOMAXPROCS.

Based on performance testing, increasing CPU cores from 1 to 2 or 4 allows the producer to reach the writing capacity limit of a 10-shard Logstore (approx. 100-160 MB/s raw throughput).

// Example: Setting the number of CPU cores used by the program
runtime.GOMAXPROCS(4)

To ensure no data is lost or re-consumed, use StopAndWait() when receiving OS signals (like SIGINT or SIGTERM). This method waits for all active goroutines to finish their current tasks before exiting.

ch := make(chan os.Signal, 1)
signal.Notify(ch, syscall.SIGTERM, syscall.SIGINT)

// Start consumption in a goroutine or main flow
consumerWorker.Start() 

// Wait for signal
if _, ok := <-ch; ok {
    // Gracefully wait for active tasks to complete
    consumerWorker.StopAndWait() 
}

To use the Aliyun Log Service (SLS) API in your Go project, install the SDK using go get.

go get -u github.com/aliyun/aliyun-log-go-sdk

If you need to regenerate code from the log.proto definition, use the following protoc command. This requires protoc and the gofast plugin to be installed in your environment.

protoc -I=. -I=$GOPATH/src -I=$GOPATH/src/github.com/gogo/protobuf/protobuf --gofast_out=. log.proto

As of version v0.1.124, the StoreViewRoutingChecker and its related APIs have been moved from the root sls package to a dedicated module.

Key Changes:

• Import Path: Change from github.com/aliyun/aliyun-log-go-sdk to github.com/aliyun/aliyun-log-go-sdk/contrib/storeviewrouting.
• Dependencies: The root SDK no longer depends on Prometheus and supports Go 1.19+. However, the new storeviewrouting module requires Go 1.25 and Prometheus v0.311.3.

// Before
import sls "github.com/aliyun/aliyun-log-go-sdk"

checker, err := sls.NewStoreViewRoutingChecker(config)

// After
import "github.com/aliyun/aliyun-log-go-sdk/contrib/storeviewrouting"

checker, err := storeviewrouting.NewStoreViewRoutingChecker(config)

Machine groups are used for Logtail collection. You can create either an IP-based machine group or a User-defined ID machine group.

// Create User-defined ID Machine Group
machineList := []string{"mac-user-defined-id-value_XX"}
var machineGroup = &sls.MachineGroup{
   Name:          groupName,
   MachineIDType: "userdefined",
   MachineIDList: machineList,
   Attribute:     sls.MachinGroupAttribute{},
}
err = util.Client.CreateMachineGroup(ProjectName, machineGroup)

// Create IP-based Machine Group
machineListIP := []string{"192.168.XX.XX"}
var machineGroupIP = &sls.MachineGroup{
   Name:          groupName,
   MachineIDType: "ip",
   MachineIDList: machineListIP,
   Attribute:     sls.MachinGroupAttribute{},
}
err = util.Client.CreateMachineGroup(ProjectName, machineGroupIP)

Unit tests in the aliyun-log-go-sdk are designed to run without network access by mocking all HTTP traffic. They are safe to run in any environment, including CI, and do not require any environment variables.

To run all unit tests, use the standard Go test command:

go test ./...