DOCSEARCHbyAlgolia
Submit Docs

CloudQuery Documentation

website·Indexed 35 days ago

https://docs.cloudquery.io/docs/intro/

Documentation for the CloudQuery Platform and CLI, a high-performance data integration tool for extracting configuration data from cloud providers and SaaS apps. Includes CLI reference for commands like sync, init, and migrate, core concepts on architecture and type systems, and guides for building, instrumenting, and publishing integrations to the CloudQuery Hub.

Tokens
100K
Snippets
332
Records
693
Agent score
64%

What's inside CloudQuery

  1. Overview of CloudQuery Integration Development

    CloudQuery integrations are modular components: source integrations fetch data from third-party APIs, and destination integrations write that data to a target. The protocol is language-agnostic, utilizing gRPC and Apache Arrow. In the codebase and SDKs, integrations are referred to as plugins.

  2. Overview of CloudQuery Transformations

    CloudQuery supports two types of transformations to modify data as it moves from a source to a destination:

    1. Pre-load transformations: Transformer integrations that modify data 'in flight' between the source and destination. These are used for renaming tables, removing sensitive columns, or filtering rows before they are written to the database.
    2. Post-load transformations: dbt packages that run after data has landed in the destination. These are used to create analytics-ready views, aggregate data, or build reports (e.g., asset inventories or cost analysis).

  3. Overview of Azure Cost Integration

    The Azure Cost integration reads Cost Management FOCUS exports from a designated Azure storage account. Unlike other cost tools, it does not call a cost API per billing period, avoiding Consumption API rate limits.

    Critical Requirement: This integration only supports FOCUS exports delivered as Parquet. It does not support ActualCost, AmortizedCost, PriceSheet metrics, or CSV delivery.

  4. Overview of CloudQuery Platform Notifications

    CloudQuery Platform Notifications route Insights and Policy violations to external tools. The system consists of three primary components:

    1. Notification Rules: Define conditions to match resources and determine if they should be delivered to specific destinations or suppressed.
    2. Notification Destinations: Reusable endpoints such as Slack channels, Microsoft Teams channels, and webhook endpoints.
    3. Message Content: The actual text or JSON body sent to destinations, authored using Liquid templates.

  5. Overview of CloudQuery Platform data access controls

    CloudQuery Platform uses a combination of Data Permissions and Custom Roles to limit the data individual users can see and query. These controls apply across the Asset Inventory, SQL Console, and Reports.

    Important Constraints:

    • This feature only applies to data synced to the CloudQuery Platform. Data synced to external databases must be managed via the external database's own access controls.
    • Custom roles automatically inherit all permissions from the General:Read role.
    • For a custom role to take effect, the user must not have any built-in roles assigned.

  6. Overview of Single Sign-on (SSO) in CloudQuery Platform

    Single Sign-on (SSO) allows users to log into the CloudQuery Platform using company credentials from providers such as Okta, Google, and Microsoft. For providers not explicitly listed in the documentation, users should follow the general patterns established in the existing provider guides and adjust for their specific application.

  7. Overview of CloudQuery Platform Alerts

    Alerts send notifications when a defined SQL query returns one or more rows after a sync. They are designed to feed events into existing incident management tools when infrastructure does not comply with set rules.

    Important: Alerts functionality is deprecated and has been consolidated into Policies.

  8. Overview of CloudQuery Policies

    Policies in CloudQuery Platform are SQL-based detective controls used to continuously evaluate cloud infrastructure at runtime. Unlike IaC scanners that check configurations during deployment, Policies detect issues in existing resources, including those created via the console, third-party tools, or resources that have drifted from their original IaC state. Each row returned by a policy's SQL query is treated as a single violation.

  9. Overview of CloudQuery CLI

    CloudQuery CLI is a self-hosted, open-source tool used to sync cloud infrastructure data into user-managed databases and data warehouses. It provides full control over configuration, scheduling, and data storage, unlike the managed CloudQuery Platform.

  11. Overview of the CloudQuery Kubernetes (K8s) Integration

    The Kubernetes integration syncs cluster resources—such as pods, deployments, services, namespaces, nodes, RBAC roles, and config maps—into the CloudQuery Platform. This allows users to view Kubernetes infrastructure alongside other cloud resources. A prerequisite for setting up a K8s integration is having an existing cloud provider integration (AWS, Azure, or GCP) configured for the account where the cluster is running.

  12. Overview of the CloudQuery Asset Inventory

    The Asset Inventory is a unified view of all cloud resources across connected integrations (e.g., AWS, GCP, Azure). Resources are normalized into a common schema and updated after each sync. The interface consists of a left sidebar for category browsing, a main table for viewing/filtering resources, and a detail panel for inspecting individual resource attributes.