Cloud SQL Go Connector

repository·main·Indexed 40 days ago

https://github.com/googlecloudplatform/cloud-sql-go-connector

A native Go library for secure connectivity to Cloud SQL instances using IAM-based authorization and TLS 1.3 encryption. It provides a secure alternative to the Cloud SQL Auth Proxy, supporting MySQL, PostgreSQL, and SQL Server. Key features include IAM DB authentication, DNS-based failover for disaster recovery, and optimized patterns for Cloud Run using WithLazyRefresh().

Tokens
8.5K
Snippets
25
Records
36
Agent score
51%

What's inside Cloud SQL Go Connector

  1. Overview of Cloud SQL Go Connector

    main

    The Cloud SQL Go Connector is a native Go alternative to the Cloud SQL Auth Proxy. It provides secure, identity-based connectivity to Cloud SQL instances without requiring manual SSL certificate management or firewall configuration.

    Key benefits include:

    • IAM Authorization: Uses IAM permissions to control access to Cloud SQL instances.
    • Improved Security: Uses TLS 1.3 encryption and identity verification between the client and the server-side proxy, independent of the database protocol.
    • Convenience: Eliminates the need to manage SSL certificates, firewalls, or IP addresses.
    • IAM DB Authentication: Supports Cloud SQL's automatic IAM DB AuthN feature.
  2. Enable Metrics and Tracing with OpenCensus/OpenTelemetry

    main

    The library supports observability via OpenCensus. It is highly recommended to use the OpenTelemetry bridge to migrate to OpenTelemetry.

    Supported Metrics

    • cloudsqlconn/dial_latency: Dialer latencies (ms)
    • cloudsqlconn/open_connections: Current number of open connections
    • cloudsqlconn/dial_failure_count: Failed dial attempts
    • cloudsqlconn/refresh_success_count: Successful certificate refreshes
    • cloudsqlconn/refresh_failure_count: Failed certificate refreshes
    • cloudsqlconn/bytes_sent: Bytes sent to Cloud SQL
    • cloudsqlconn/bytes_received: Bytes received from Cloud SQL

    Supported Traces

    • cloud.google.com/go/cloudsqlconn.Dial: The dial operation (includes certificate refresh and connection)
    • cloud.google.com/go/cloudsqlconn/internal.InstanceInfo: Retrieving instance metadata
    • cloud.google.com/go/cloudsqlconn/internal.Connect: The connection attempt using the ephemeral certificate
    • SQL Admin API client operations
  3. Implement Lazy Instantiation for Cloud Run services

    main

    When running in Cloud Run, avoid initializing the Connector (Dialer) and sql.DB objects in the global scope during container startup. Instead, define them as global variables (initially nil) and use sync.Once to instantiate them inside your request handlers.

    This pattern provides:

    1. Faster Startup: Reduces cold start latency by deferring initialization until the first request.
    2. Resource Efficiency: Prevents background processes from running when the CPU is throttled.
    3. Connection Reuse: Reuses the same Connector and sql.DB instances for all subsequent requests to that container instance, preventing connection limit exhaustion.
  4. Migrate MySQL connections from Cloud SQL Proxy v1 to the Go Connector

    main

    To migrate MySQL connections, replace the github.com/GoogleCloudPlatform/cloudsql-proxy/proxy/dialers/mysql package with cloud.google.com/go/cloudsqlconn/mysql/mysql.

    Unlike the v1 proxy, the Go Connector uses a driver registration pattern. You must call mysql.RegisterDriver with a custom driver name and any desired cloudsqlconn options (e.g., cloudsqlconn.WithCredentialsFile). This function returns a cleanup function that must be deferred to close the underlying driver when the application is finished. Finally, use sql.Open with the name you provided during registration.

    import (
    	"database/sql"
    	"cloud.google.com/go/cloudsqlconn"
    	"cloud.google.com/go/cloudsqlconn/mysql/mysql"
    )
    
    func connectMySQL() *sql.DB {
    	// Register a driver using whatever name you like.
    	cleanup, err := mysql.RegisterDriver(
    		"cloudsql-mysql",
    		// any desired options go here, for example:
    		cloudsqlconn.WithCredentialsFile("key.json"),
    	)
    	if err != nil {
    		// handle error as necessary
    	}
    	// call cleanup to close the underylying driver when you're done with the
    	// db.
    	defer cleanup()
    
    	db, err := sql.Open(
    		"cloudsql-mysql", // matches the name registered above
    		"myuser:mypass@cloudsql-mysql(project:region:instance)/mydb",
    	)
    	if err != nil {
    		// handle error as necessary
    	}
    	return db
    }
  5. Use DNS for Advanced Disaster Recovery and Failover

    main

    The connector can use DNS to identify instances, enabling automatic failover. You can use Cloud SQL Advanced Disaster Recovery write endpoints or your own managed DNS records.

    Configure DNS Records

    Add a private TXT record to your DNS server.

    • Name: The domain name used by your application (e.g., prod-db.mycompany.example.com).
    • Value: The Cloud SQL instance connection name (e.g., my-project:region:my-instance).

    Warning: Do not use public DNS servers, as this exposes your instance name to the internet.

    Automatic Failover Behavior

    When the connector is configured with cloudsqlconn.WithDNSResolver(), it periodically polls for changes to the DNS record (default every 30 seconds). If the record changes, the connector closes existing connections to the old instance, forcing the connection pool to reconnect to the new instance specified in the DNS record.

    To adjust the polling frequency, use WithFailoverPeriod(d time.Duration). Setting this to 0 disables polling and only checks for changes during new connection attempts.

    package main
    
    import (
    	"database/sql"
    
    	"cloud.google.com/go/cloudsqlconn"
    	"cloud.google.com/go/cloudsqlconn/mysql/mysql"
    )
    
    func connect() {
    	cleanup, err := mysql.RegisterDriver("cloudsql-mysql",
    		cloudsqlconn.WithDNSResolver())
    	if err != nil {
    		// ... handle error
    	}
    	defer cleanup()
    
    	db, err := sql.Open(
    		"cloudsql-mysql",
    		"myuser:mypass@cloudsql-mysql(prod-db.mycompany.example.com)/mydb",
        )
        // ... etc
    }
  6. Configure IAM Authentication prerequisites

    main

    To use IAM-based database authentication with the Cloud SQL Go Connector, you must complete two configuration steps:

    1. Grant Cloud SQL Client role: The Cloud Run service account must have the roles/cloudsql.client role.
      gcloud projects add-iam-policy-binding PROJECT_ID \
          --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" \
          --role="roles/cloudsql.client"
    2. Add service account as a database user: The service account must be registered as a user on the Cloud SQL instance with the type cloud_iam_user.
      gcloud sql users create SERVICE_ACCOUNT_EMAIL \
          --instance=INSTANCE_NAME \
          --type=cloud_iam_user
  7. Enable Debug Logging

    main

    To diagnose problems with background certificate refreshes, you can enable debug logging by providing a logger that implements the debug.ContextLogger interface when initializing the Dialer using cloudsqlconn.WithContextDebugLogger(l).

    import (
        "context"
        "cloud.google.com/go/cloudsqlconn"
    )
    
    type myLogger struct{}
    
    func (l *myLogger) Debugf(ctx context.Context, format string, args ...interface{}) {
        // Log as you like here
    }
    
    func connect() {
        l := &myLogger{}
    
        d, err := NewDialer(
            context.Background(),
            cloudsqlconn.WithContextDebugLogger(l),
        )
        // use dialer as usual...
    }
  8. Use Automatic IAM Database Authentication

    main

    The connector supports automatic IAM database authentication for Postgres and MySQL. This requires your Cloud SQL instance to be configured to allow IAM authentication and the appropriate IAM user/service account to be added to the database.

    To enable this, use the WithIAMAuthN() option when creating a Dialer or the WithDialIAMAuthN DialOption.

    DSN Configuration for IAM

    When using IAM authentication, omit the password field in your DSN. The user field must be formatted as follows:

    • Postgres:
      • IAM User: The user's email address.
      • Service Account: The service account's email without the .gserviceaccount.com suffix.
      • Example: user=test-sa@test-project.iam dbname=mydb sslmode=disable
    • MySQL:
      • IAM User: The user's email address without the @ or domain name (e.g., test-user for test-user@gmail.com).
      • Service Account: The service account's email without the @project-id.iam.gserviceaccount.com suffix.
      • Example: user=test-sa dbname=mydb sslmode=disable
  9. Migrate Postgres connections from Cloud SQL Proxy v1 to the Go Connector

    main

    To migrate Postgres connections, replace the github.com/GoogleCloudPlatform/cloudsql-proxy/proxy/dialers/postgres package with cloud.google.com/go/cloudsqlconn/postgres/pgxv5.

    Use pgxv5.RegisterDriver to register a driver name and configure options like cloudsqlconn.WithIAMAuthN(). You must defer the returned cleanup function. When calling sql.Open, use the registered driver name and a connection string that includes the instance connection name as the host.

    import (
    	"database/sql"
    	"cloud.google.com/go/cloudsqlconn"
    	"cloud.google.com/go/cloudsqlconn/postgres/pgxv5"
    )
    
    func connectPostgres() *sql.DB {
    	// Register a driver using whatever name you like.
    	cleanup, err := pgxv5.RegisterDriver(
    		"cloudsql-postgres",
    		// any desired options go here, for example:
    		cloudsqlconn.WithCredentialsFile("key.json"),
    		cloudsqlconn.WithIAMAuthN(),
    	)
    	if err != nil {
    		// handle error as necessary
    	}
    	// call cleanup to close the underylying driver when you're done with the
    	// db.
    	defer cleanup()
    	db, err := sql.Open(
    		"cloudsql-postgres", // matches the name registered above
    		"host=project:region:instance user=postgres password=password dbname=postgres sslmode=disable",
    	)
    	if err != nil {
    		// handle error as necessary
    	}
    	return db
    }
  10. Build and push Docker images to Artifact Registry

    main

    Before deploying to Cloud Run, enable the Artifact Registry API, create a repository, and authenticate Docker.

    1. Enable API:
      gcloud services enable artifactregistry.googleapis.com
    2. Create Repository:
      gcloud artifacts repositories create REPO_NAME \
        --repository-format=docker \
        --location=REGION
    3. Authenticate Docker:
      gcloud auth configure-docker REGION-docker.pkg.dev
    4. Build and Push:
      docker build -t REGION-docker.pkg.dev/PROJECT_ID/REPO_NAME/IMAGE_NAME mysql
      docker push REGION-docker.pkg.dev/PROJECT_ID/REPO_NAME/IMAGE_NAME