cloudlist

Cloudlist is an asset discovery tool that gathers information from multiple providers (e.g., AWS, GCP, DigitalOcean) using their specific APIs.

• Providers: Each provider implements the schema.Provider interface. The core logic resides in the Resources() method, which returns a list of discovered assets.
• Resources: A resource represents a single cloud unit (like an EC2 instance or a DNS record) belonging to an organization. Every resource must provide either an IP address or a DNS name.

Providers return a schema.Resource structure, which includes metadata such as whether the asset is Public or Private, the Provider name, the resource ID, and various IP/DNS fields.

// Resource is a cloud resource belonging to the organization
type Resource struct {
	Public       bool   `json:"public"` 
	Provider     string `json:"provider"` 
	ID           string `json:"id,omitempty"` 
	PublicIPv4    string `json:"public_ipv4,omitempty"` 
	PublicIPv6    string `json:"public_ipv6,omitempty"` 
	PrivateIpv4   string `json:"private_ipv4,omitempty"` 
	PrivateIpv6   string `json:"private_ipv6,omitempty"` 
	DNSName       string `json:"dns_name,omitempty"` 
}

Cloudlist offers two ways to discover GCP assets. Choose based on your scope and speed requirements:

1. Organization-Level Asset API: Best for comprehensive organization-wide audits. It uses the Cloud Asset Inventory API to cover 220+ resources across the entire organization. It is slower (~45s) but provides much broader coverage.
2. Individual Service APIs: Best for fast, project-specific scans. It uses individual GCP service APIs to discover 67 resources per project. It is faster (~23s) but limited to accessible projects.

Extended Metadata: The Organization-Level approach supports an optional 'Extended Metadata' feature. If you provide additional service-specific permissions, Cloudlist fetches more detailed data. If permissions are missing, it gracefully degrades to basic Asset API data.

| Feature | Organization-Level Asset API | Individual Service APIs |
|---------|----------------------------|------------------------|
| **Scope** | All projects in organization | Accessible projects only |
| **Speed** | Slower (~45s) | Faster (~23s) |
| **Coverage** | 220+ resources across org | 67 resources per project |
| **Required Permissions** | Cloud Asset Inventory roles | Individual service permissions |
| **Best For** | Comprehensive org audits | Fast project-specific scans |
| **Extended Metadata** | Optional with additional permissions | Always available |

In CI/CD environments, use a minimal-permission service account that is only allowed to impersonate the powerful target service account. This limits the blast radius if the CI key is compromised.

1. Create a minimal service account for CI.
2. Grant that minimal account the roles/iam.serviceAccountTokenCreator role on the target service account.
3. Generate a key for the minimal account and provide it to the CI environment via source_credentials.

- provider: gcp
  id: ci-discovery
  use_short_lived_credentials: true
  service_account_email: "powerful-sa@project.iam.gserviceaccount.com"
  source_credentials: "minimal-ci-sa.json"
  token_lifetime: "3600s"

# Create minimal CI service account
gcloud iam service-accounts create minimal-ci-sa \
  --display-name="Minimal CI Service Account"

# Grant impersonation permission
gcloud iam service-accounts add-iam-policy-binding \
  powerful-sa@project.iam.gserviceaccount.com \
  --member="serviceAccount:minimal-ci-sa@project.iam.gserviceaccount.com" \
  --role="roles/iam.serviceAccountTokenCreator"

# Create key for CI
gcloud iam service-accounts keys create minimal-ci-sa.json \
  --iam-account=minimal-ci-sa@project.iam.gserviceaccount.com

To enable short-lived credentials, set use_short_lived_credentials: true and provide a service_account_email.

Configuration Options:

• use_short_lived_credentials (bool): Enable short-lived token generation (default: false).
• service_account_email (string, required): Target service account to impersonate.
• source_credentials (string, optional): Path to source credentials file (uses ADC if not provided).
• token_lifetime (string, optional): Token lifetime (e.g., "3600s" or "1h"). Range: 1s to 3600s. Default: "3600s".
• project_ids (list, optional): Limits enumeration to specific projects.
• exclude_project_ids (list, optional): Excludes specific projects from organization-wide discovery (mutually exclusive with project_ids).

Requirements:

• Source credentials need roles/iam.serviceAccountTokenCreator on the target service account.
• Target service account needs appropriate viewer roles (Compute, DNS, Storage, etc.).
• Service Account Credentials API must be enabled.

- provider: gcp
  id: dev-discovery
  use_short_lived_credentials: true
  service_account_email: "cloudlist@project.iam.gserviceaccount.com"

To use the Individual Service API approach, you must create a service account and grant it project-level permissions for each specific service you wish to scan.

1. Create Service Account

gcloud iam service-accounts create cloudlist-sa \
    --display-name="CloudList Individual Services" \
    --description="Service account for individual service API discovery"

SA_EMAIL="cloudlist-sa@YOUR-PROJECT-ID.iam.gserviceaccount.com"
PROJECT_ID="YOUR-PROJECT-ID"

2. Grant Project-Level Permissions

Grant the service account the specific viewer roles required for the services you want to discover. For example:

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:$SA_EMAIL" \
    --role="roles/compute.viewer"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:$SA_EMAIL" \
    --role="roles/dns.reader"
# ... repeat for other services

3. Configuration

Unlike the Organization-Level approach, do not provide an organization_id. The absence of this key triggers the Individual Service API mode.

- provider: gcp
  id: project-discovery
  # No organization_id = uses individual service APIs
  gcp_service_account_key: |
    {
      "type": "service_account",
      "project_id": "your-project-id",
      ...
    }

For GKE, you can use Workload Identity to achieve a 'Zero Secrets' setup where no credentials exist in your configuration files. This involves binding a Kubernetes Service Account (KSA) to a GCP Service Account.

- provider: gcp
  id: workload-discovery
  use_short_lived_credentials: true
  service_account_email: "cloudlist@project.iam.gserviceaccount.com"

# Enable workload identity on cluster
gcloud container clusters update CLUSTER_NAME \
  --workload-pool=PROJECT_ID.svc.id.goog

# Bind Kubernetes service account to GCP service account
gcloud iam service-accounts add-iam-policy-binding \
  cloudlist@PROJECT_ID.iam.gserviceaccount.com \
  --role=roles/iam.workloadIdentityUser \
  --member="serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"

# Configure workload identity on deployment
kubectl annotate serviceaccount KSA_NAME \
  iam.gke.io/gcp-service-account=cloudlist@PROJECT_ID.iam.gserviceaccount.com

To perform organization-wide discovery, you must create a service account and grant it permissions at the organization level.

1. Create Service Account

gcloud iam service-accounts create asset-viewer-sa \
    --display-name="CloudList Asset Viewer" \
    --description="Service account for organization-level asset discovery"

# Note the email for the next steps
SA_EMAIL="asset-viewer-sa@YOUR-PROJECT-ID.iam.gserviceaccount.com"
ORG_ID="YOUR-ORGANIZATION-ID"

2. Grant Organization-Level Permissions

Required Roles:

• roles/cloudasset.viewer (Core Asset API access)
• roles/resourcemanager.viewer (To list projects in the organization)

Optional Roles (for Extended Metadata):

• roles/compute.viewer (Compute instances)
• roles/cloudfunctions.viewer (Cloud Functions)
• roles/storage.objectViewer (Cloud Storage)
• roles/run.viewer (Cloud Run)
• roles/dns.reader (DNS records)
• roles/container.viewer (GKE)

Commands:

# Required
gcloud organizations add-iam-policy-binding $ORG_ID \
    --member="serviceAccount:$SA_EMAIL" \
    --role="roles/cloudasset.viewer"

gcloud organizations add-iam-policy-binding $ORG_ID \
    --member="serviceAccount:$SA_EMAIL" \
    --role="roles/resourcemanager.viewer"

# Optional (Example: Compute)
gcloud organizations add-iam-policy-binding $ORG_ID \
    --member="serviceAccount:$SA_EMAIL" \
    --role="roles/compute.viewer"

3. Generate Key

gcloud iam service-accounts keys create asset-viewer-key.json \
    --iam-account=$SA_EMAIL