DOCSEARCHbyAlgolia
Submit Docs

Tencent Cloud Object Storage Service (COS) Go SDK v5

repository·master·Indexed 42 days ago

https://github.com/tencentyun/cos-go-sdk-v5

Official Go SDK for Tencent Cloud Object Storage Service (COS) supporting the V5 XML API. It provides tools to manage buckets and objects, perform batch operations, and handle data processing and content auditing. Features include multiple authentication methods (SecretID/Key, CVM role-based, and STS temporary credentials), automatic request signing via AuthorizationTransport, and DNS scattering for internal hosts.

Tokens
37.2K
Snippets
145
Records
172
Agent score
54%

What's inside Tencent Cloud COS Go SDK

  1. Initialize a COS Client

    master

    To use the SDK, you must create a client using cos.NewClient. This requires a cos.BaseURL containing your bucket's URL and an http.Client configured with an AuthorizationTransport for authentication.

    Important Requirements:

    • The bucket URL must follow the format: https://<bucket>.cos.<region>.myqcloud.com.
    • The bucket name must follow the naming rule: {name}-{appid}.
    • Authentication can be provided via SecretID and SecretKey directly or by using environment variables COS_SECRETID and COS_SECRETKEY.
    package main

import (
	"context"
	"fmt"
	"io/ioutil"
	"net/http"
	"net/url"
	"os"
	"time"

	"github.com/tencentyun/cos-go-sdk-v5"
)

func main() {
	// Replace <bucket> and <region> with your actual information
	u, _ := url.Parse("https://<bucket>.cos.<region>.myqcloud.com")
	b := &cos.BaseURL{BucketURL: u}
	c := cos.NewClient(b, &http.Client{
		Timeout: 100 * time.Second,
		Transport: &cos.AuthorizationTransport{
			SecretID:  os.Getenv("COS_SECRETID"),
			SecretKey: os.Getenv("COS_SECRETKEY"),
		},
	})

	name := "test/hello.txt"
	resp, err := c.Object.Get(context.Background(), name, nil)
	if err != nil {
		panic(err)
	}
	bs, _ := ioutil.ReadAll(resp.Body)
	resp.Body.Close()
	fmt.Printf("%s\n", string(bs))
}

  2. Configure COS credentials via environment variables

    master

    Before running examples or using the SDK, you must export your Tencent Cloud Cloud Object Storage (COS) credentials as environment variables. The SDK expects COS_SECRETID and COS_SECRETKEY to be set in your shell environment.

    export COS_SECRETID=xx
export COS_SECRETKEY=xxx

  4. Configure origin HTTP headers and redirection parameters

    master

    Within a BucketOriginRule, the OriginParameter field controls how the origin interacts with the request:

    • Protocol: The protocol to use (e.g., http or https).
    • FollowRedirection: Boolean to determine if the origin should follow redirects.
    • FollowRedirectionConfiguration: Detailed settings for redirection, such as FollowOriginHeaders and FollowUrlAutoDecode.
    • HttpHeader: A BucketOriginHttpHeader object to manage headers:
      • FollowAllHeaders: Boolean to follow all incoming headers.
      • NewHttpHeaders: A list of OriginHttpHeader (Key/Value) to add new headers.
      • FollowHttpHeaders: A list of headers to carry over.
      • ForbidFollowHeaders: A list of headers to explicitly block from being carried over.

  5. Define BucketLifecycleRule components

    master

    A BucketLifecycleRule defines how objects in a bucket are managed over time. Key components include:

    • Filter: Determines which objects the rule applies to (via Prefix, Tag, or an And logical operator).
    • Transition: Defines when objects move to different storage classes (e.g., moving to a cheaper storage class after $N$ days).
    • Expiration: Defines when objects are permanently deleted.
    • AbortIncompleteMultipartUpload: Defines when to clean up failed or incomplete multipart uploads.
    • NoncurrentVersionTransition / NoncurrentVersionExpiration: Manages lifecycle for non-current versions in versioned buckets.

  6. Configure origin host information and standby hosts

    master

    The BucketOriginInfo struct manages how requests are routed to backend hosts. It supports multiple HostInfos and file-based configurations via FileInfo.

    Host Information (BucketOriginHostInfo)

    • HostName: The primary hostname.
    • Weight: The weight for load balancing.
    • PrivateHost: Configuration for a private host, including a CredentialProvider for authentication.
    • StandbyHostName: A list of BucketOriginStandbyHost objects used for failover. Each standby host requires an Index to identify its priority/order.

    Note on Standby Hosts: When using StandbyHostName, you must specify the Index for each host. The SDK handles the mapping of these indices to the XML format StandbyHostName_{Index}.

    info := &cos.BucketOriginInfo{
    HostInfos: []*cos.BucketOriginHostInfo{
        {
            HostName: "primary.example.com",
            Weight:   100,
            StandbyHostName: []*cos.BucketOriginStandbyHost{
                {Index: 1, HostName: "backup1.example.com"},
                {Index: 2, HostName: "backup2.example.com"},
            },
        },
    },
}

  7. Understand ObjectSelectResponse and its frame-based streaming

    master

    The ObjectSelectResponse does not return a simple data blob; it returns a stream of frames. When you call .Read(), the SDK parses the underlying byte stream into structured frames.

    Frame Types

    • kRecordsFrameType: Contains the actual query result data. The DataFrame.ContentType indicates the format.
    • kProgressFrameType: Provides progress updates via ProgressFrame (BytesScanned, BytesProcessed, BytesReturned).
    • kStatsFrameType: Provides statistics via StatsFrame (BytesScanned, BytesProcessed, BytesReturned).
    • kEndFrameType: Signals the end of the stream. Finish will be set to true.
    • kErrorFrameType: Signals a query error. The ErrorFrame contains the Code and Message.
    • kContinuationFrameType: Indicates a continuation of the previous frame.

    Accessing Data

    To access specific frame data, you must check the Frame field of the ObjectSelectResponse after a read operation. Note that ObjectSelectResponse implements io.Reader, so standard reading patterns apply, but the internal state of the Frame is updated as you consume the stream.

  8. Batch Job Operation Types

    master

    The BatchJobOperation struct supports the following operations:

    • PutObjectCopy: Uses BatchJobOperationCopy to copy objects. Supports TargetKeyPrefix, MetadataDirective, TaggingDirective, and AccessControlGrants.
    • RestoreObject: Uses BatchInitiateRestoreObject to restore objects from archive storage. Requires ExpirationInDays and JobTier.
    • PutObjectTagging: Uses BatchPutObjectTagging to update object tags. Requires TaggingDirective and a TagSet.

  9. Manage multiple retry errors with RetryError

    master

    The RetryError type is used to wrap multiple errors that occurred during a retry loop. It implements the error interface by joining all error messages with a semicolon.

    Crucially, RetryError is designed to work with Go's errors.Is and errors.As by 'tunneling' through to the last error in the sequence. This allows you to check if the final attempt failed with a specific error type or value.

    // Example of how RetryError behaves with errors.Is
// If the last error in the slice matches the target, Is returns true.
var errs []error
errs = append(errs, errors.New("first attempt failed"))
errs = append(errs, errors.New("final attempt failed: connection refused"))

retryErr := &cos.RetryError{Errs: errs}

if errors.Is(retryErr, errors.New("final attempt failed: connection refused")) {
    // This will be true because RetryError.Is() checks the last error
}

  10. How DNS scattering works for internal COS hosts

    master
    To improve connection reliability when accessing COS via internal endpoints, the SDK provides DNSScatterTransport. This transport uses a custom DialContext (DNSScatterDialContext) that performs DNS lookups and then randomly selects an IP address from the returned list to initiate the connection. This helps distribute load and avoid issues with specific IP addresses in a multi-IP DNS response.

  12. How multipart upload and download workers work

    master

    The SDK uses a worker pool pattern for high-performance multipart operations.

    Upload Workers

    UploadWorker consumes *Jobs from a channel and produces *Results to another. Each job contains a chunk of the file, an UploadId, and the necessary ObjectUploadPartOptions. The worker handles file seeking, reading the specific chunk, and calling UploadPart with retries.

    Download Workers

    DownloadWorker consumes *Jobs and produces *Results. Each job specifies a byte range. The worker performs a Get request for that range and uses copyChunkToFileAt to write the data to the target file at the correct offset using WriteAt. This allows multiple workers to write to the same file concurrently without overlapping or needing a central coordinator for file offsets.