DOCSEARCHbyAlgolia
Submit Docs

Expanso Documentation

website·Indexed 34 days ago

https://docs.expanso.io

Documentation for Expanso Edge, an edge-first data pipeline platform. Includes guides on components (inputs, processors, and outputs), Bloblang transformations, job configuration, and operational management for Expanso Cloud deployments.

Tokens
140.5K
Snippets
744
Records
946
Agent score
64%

What's inside Expanso

  1. Overview of Expanso CLI and Edge Agent

    Expanso provides two primary command-line tools for managing edge computing infrastructure:

    • expanso-cli: The main interface for interacting with the Expanso control plane and edge nodes.
    • expanso-edge: The command-line tool for the edge agent.

    The CLI interacts with either the Control Plane (central management service) or Edge Nodes (individual agents). The target is determined by the active profile endpoint.

  2. Overview of available Input Components

    Expanso provides 52 input components to ingest data from various sources into the pipeline. These include cloud services (AWS, Azure, GCP), databases (MySQL, PostgreSQL, MongoDB, Redis), messaging systems (Kafka, NATS, MQTT), and local system inputs (files, stdin, subprocesses).

  3. Overview of Expanso Edge

    Expanso Edge is a managed SaaS platform for deploying and orchestrating data pipelines at the edge. It allows users to process data where it is generated to reduce bandwidth costs, lower latency, maintain data privacy on-premises, and ensure functionality during network outages. The platform features a visual pipeline builder and over 200 built-in components for connecting to message queues (Kafka, NATS, RabbitMQ), databases, object storage (S3, GCS, Azure Blob), and APIs.

  4. Overview of Expanso Output Components

    Expanso provides 58 output components that allow users to route, store, or transmit processed messages to various destinations. These components range from cloud storage and databases to messaging queues and local system outputs.

  5. Overview of Expanso Getting Started pipeline examples

    For developers new to Expanso, the following basic pipeline examples are available to learn core concepts:

    • Basic Pipeline: Log processing that reads from a file, parses JSON, and outputs to stdout.
    • HTTP to File: Sets up an HTTP server to receive webhooks and write them to files using path templating with timestamps.
    • Data Transformer: Uses Bloblang for field mapping, filtering, data enrichment, and conditional logic.
    • Metrics Collector: Handles time-series data by collecting system metrics periodically and batching them for monitoring services.

  6. Manage Expanso CLI connection profiles

    Profiles in the Expanso CLI store connection settings for different environments (e.g., production, staging, development). They allow users to switch between environments without repeatedly specifying endpoints and API keys. Profiles are stored locally in ~/.expanso/cli-client/profiles/.

  7. Understand Expanso product limitations

    Expanso is a record movement and transformation tool, not a general-purpose orchestrator. Note the following constraints:

    • Not an Orchestrator: It is not a workflow orchestrator, batch query engine, or data warehouse. It does not run arbitrary task graphs or store data for analysis.
    • No Lakehouse Writers: No native components for Apache Iceberg, Delta Lake, or Apache Hudi.
    • No Native Chat/Email Components: Services like Slack, Discord, PagerDuty, or Email must be accessed via http_client.
    • No Native Anthropic (Claude): Claude is not currently available as a model processor.
    • Chunking: The chunker scanner only supports byte-count splitting. For text-aware/paragraph chunking, use a mapping processor.
    • Unproven Features: CDC components (postgres_cdc, mysql_cdc, aws_dynamodb_cdc) and X (Twitter) ingestion are currently unproven in verified runs.

  8. Enforce Security and Compliance policies at the Edge

    Expanso Edge allows organizations to apply security policies and compliance rules (such as GDPR, HIPAA, and PCI-DSS) at the point of data collection. This prevents sensitive data from entering the infrastructure by using automated PII detection, data minimization, and encryption in motion.

  9. Configure the Expanso CLI for Workspace Access

    To connect the CLI to a workspace, navigate to the workspace overview and open the API Access dialog. This dialog provides the specific API endpoint and profile instructions required for configuration. Do not use the general Cloud console URL as the API endpoint.

  10. Understand Expanso System Architecture

    Expanso operates using a decoupled architecture consisting of a central Control Plane and distributed Edge Nodes.

    Expanso Control Plane (cloud.expanso.io) A managed service (or on-prem orchestrator for enterprise) responsible for:

    • Pipeline creation, configuration, and testing.
    • Node deployment and health monitoring.
    • Metrics aggregation and workspace management.
    • User authentication.

    Edge Nodes A lightweight single binary runtime that executes pipelines on your infrastructure (Linux, macOS, Windows). They can be deployed on bare metal, VMs, Docker, Kubernetes, or IoT devices. Edge nodes connect to the workspace via TLS to receive configurations and report telemetry.

  11. Create an Expanso Fleet Overview Dashboard

    Create a dashboard to monitor edge node health and resource utilization using Prometheus queries (PromQL). Recommended panels include:

    • Connected Edge Nodes: Count of nodes reporting metrics (Stat panel).
    • CPU Utilization (%): Percentage of CPU usage across nodes (Time series panel).
    • Memory Usage (MB): Memory consumption converted to MB (Time series panel).
    • Goroutine Count: Monitor for Go runtime leaks (Time series panel).
    • GC Cycles per Minute: Track garbage collection frequency (Time series panel).
    • File Descriptor Usage (%): Monitor for file descriptor exhaustion (Gauge panel).
    # Connected Edge Nodes
count by (node_id, node_hostname) (process_cpu_utilization_ratio{service_name="expanso-edge"})

# CPU Utilization (%)
process_cpu_utilization_ratio * 100

# Memory Usage (MB)
process_memory_usage_bytes / 1024 / 1024

# Goroutine Count
go_goroutines_count_ratio

# GC Cycles per Minute
rate(go_gc_cycles_total[5m]) * 60

# File Descriptor Usage (%)
(process_open_file_descriptors_ratio / process_max_file_descriptors_ratio) * 100

  12. Expanso API Overview and Base URL

    The Expanso API provides a REST interface for managing jobs, nodes, and executions across edge computing infrastructure. All endpoints are versioned under /api/v1/ and return JSON responses.

    Endpoint Types:

    • Orchestrator: The central management service. Querying this returns data across all nodes.
    • Edge Nodes: Individual agents. When running in "Local Mode," these expose the same API for direct management, enabling local development, troubleshooting, or air-gapped deployments.

    Base URL Format: https://YOUR_WORKSPACE_ID.us1.cloud.expanso.io:9010/api/v1

    Replace YOUR_WORKSPACE_ID with your specific workspace ID found in the Expanso Cloud Console under Your Workspace → Endpoint.