DOCSEARCHbyAlgolia
Submit Docs

Ascend Developer Documentation

website·Indexed 35 days ago

https://docs.ascend.io

Documentation for Ascend, an agentic data engineering platform. It covers building data pipelines using Components, Flows, and Modules, as well as integrating with data sources like Snowflake, BigQuery, and Databricks. Key features include Otto (an AI-powered agent), Intelligence Core for Python and SQL Flex Code, smart schema strategies, and DataOps implementation for the software development lifecycle.

Tokens
192K
Snippets
983
Records
1.6K
Agent score
57%

What's inside Ascend

  1. Overview of Custom Modules

    Custom Modules are reusable groups of derived Components bundled into a single visual unit within a Flow Graph. Unlike template-based Blueprint Modules, Custom Modules provide full programmatic control over Component generation, making them suitable for complex conditional logic, dynamic Component counts, custom configuration processing, and integration with external systems during build time.

  2. Overview of Ascend Component types

    Components are the building blocks of Ascend data pipelines (Flows). They are defined via YAML, Python, or SQL files. The primary types include:

    • Read Components: Starting points that ingest data from external sources (databases, cloud storage) into the data plane.
    • Transforms: Handle processing logic (cleaning, aggregation, enrichment) using SQL or Python.
    • Write Components: Endpoints that export processed data to external systems or storage.
    • Task Components: Execute custom multiline SQL or Python scripts for complex processing tasks.
    • Test Components: Validate data quality and integrity. Tests can also be added to individual components to run automatically during execution.

  3. Overview of Ascend Modules

    Modules are reusable groups of derived Components bundled into a single visual unit within an Ascend Flow Graph. They allow developers to package complex logic into modular building blocks to simplify data pipelines.

  4. Overview of Ascend Modules

    An Ascend Module is a programmatic way to specify a group of related Components, allowing developers to manage complex, repeatable patterns in Flows. Modules provide a layer of abstraction where logic is defined in Python and configured in YAML. They can be instantiated multiple times in Flows and appear in the Flow Graph as either a single collapsed entity or an expanded view showing all internal Components and their Connections.

    Key benefits include:

    • Reusability: Define code once and reuse across Flows.
    • Standardization: Enforce consistent implementation patterns.
    • Encapsulation: Hide implementation details behind a configuration interface.
    • Scalability: Reconfigure Modules as requirements change without significant rework.

  5. Overview of Ascend Data Planes

    Data Planes in Ascend allow users to connect to modern data warehousing and analytics platforms. These connections enable the use of cloud-native compute and storage to handle large-scale data processing and analytics workloads.

  6. Overview of Snowpark Transforms

    Snowpark Transforms allow users to process data directly within Snowflake's native compute engine using Python-like syntax. This leverages Snowflake's performance, scalability, and security while providing the flexibility of Python. Prerequisites include an Ascend Project, Workspace, Read Components (or other data sources), and Snowflake configured as the Data Plane.

  7. Overview of Ascend Automation

    Ascend Automation uses an event-driven trigger-action model to execute Flows on a schedule or in response to specific events. An Automation consists of a trigger (sensor or event) that executes one or more actions.

  8. Overview of Secrets Management in Ascend

    Ascend uses Vaults to securely store and manage secrets. Users can choose between built-in Ascend Managed Vaults or integrating with existing external cloud provider Vault services depending on their infrastructure and security requirements.

  9. Overview of Blueprint Modules

    Blueprint Modules are built-in Ascend solutions used to generate groups of derived Components from templates using Jinja. They are ideal for template-based Component generation, SQL/Python/YAML templates with parameter substitution, and creating reusable patterns across different datasets.

  10. Overview of Ascend Environment Vaults

    Environment Vaults provide secure secret management scoped to specific Environments in an Ascend Instance. While Ascend provides managed vaults by default, users can configure their own cloud provider vaults for greater control and integration with existing infrastructure.

  11. Overview of Otto AI agent in Ascend

    Otto is an AI-powered agent integrated into Ascend to streamline data engineering. It provides capabilities including inline code editing, natural language pipeline building and monitoring, autonomous background agents for routine tasks, and the ability to create custom agents for specific automation needs.

  12. Overview of Ascend Transform Components

    Transforms in Ascend are used for data processing tasks such as business logic application, data cleansing, shaping, and aggregation. They take data from input Components (like Read Components or other Transforms) and produce output for other Components (like Write Components).

    Ascend supports two primary languages for transforms, determined by the file extension:

    • SQL Transforms (.sql): Best for set-based operations, aggregating, filtering, and joining relational data.
    • Python Transforms (.py): Best for complex logic, text processing, custom calculations, or calling external services.