DOCSEARCHbyAlgolia
Submit Docs

PyAirbyte

repository·main·Indexed 43 days ago

https://github.com/airbytehq/pyairbyte

A Python library that allows developers to use Airbyte connectors directly within Python applications for data extraction and testing without requiring a full Airbyte orchestration platform. It supports declarative, Python-based, and Docker-based sources, and provides built-in SQL caches including DuckDB, BigQuery, Postgres, Snowflake, and MotherDuck.

Tokens
22.3K
Snippets
66
Records
99
Agent score
56%

What's inside PyAirbyte

  1. What is PyAirbyte and how does it differ from Airbyte?

    main

    PyAirbyte is a Python library designed to bring Airbyte connectors into Python workflows.

    Key Distinctions:

    • Airbyte is a full-fledged data integration platform providing orchestration, scheduling, logging, and alerting for production pipelines.
    • PyAirbyte is a utility library. It does not provide orchestration, scheduling, or production management features. It is intended for use within Python code, for testing, or as part of a larger orchestration framework (like Airflow, Dagster, or Snowpark).

  2. Understand the PyAirbyte cache concept

    main
    The PyAirbyte cache acts as a built-in destination implementation. While it functions like a destination by storing extracted data, it is referred to as a 'cache' to distinguish it from the official list of certified Airbyte destinations.

  3. Understand the relationship between PyAirbyte and Airbyte

    main

    PyAirbyte is a Python library designed to allow the use of Airbyte connectors directly within Python environments. It is not a replacement for the full Airbyte platform.

    Key differences:

    • PyAirbyte: Provides a Pythonic interface to connectors. It lacks orchestration, scheduling, logging, and alerting capabilities.
    • Airbyte: A complete data integration platform providing connectors, orchestration, and scheduling for production data pipelines.

  4. Use PyAirbyte for ETL and data orchestration

    main

    PyAirbyte is compatible with data orchestration frameworks such as Airflow, Dagster, and Snowpark.

    You can build traditional ETL pipelines by selecting a cache type that matches your target destination. For example, use SnowflakeCache to land data directly into Snowflake.

  5. Set up the PyAirbyte development environment

    main

    To develop PyAirbyte locally, ensure you have uv installed. Use uv sync --group dev to synchronize the environment with development dependencies.

    • Running Examples: Examples are located in the examples/ folder and can be executed using uv run python examples/<example_file>.
    • Running Tests: Execute unit tests and type checks using uv run pytest.
    # Install dependencies
uv sync --group dev

# Run a specific example
uv run python examples/your_example.py

# Run tests
uv run pytest

  6. Generate and preview API documentation

    main

    PyAirbyte uses pdoc to generate API documentation from docstrings. Documentation is generated into the docs/generated folder.

    • Generate Docs: Run poe docs-generate to create the documentation files.
    • Preview Docs: Run poe docs-preview (or uv run poe docs-preview) to build and open the documentation in a browser immediately.

    Note: The test_docs.py test in pytest will automatically update generated content; these updates must be manually committed for tests to pass.

    # Generate documentation
poe docs-generate

# Build and preview documentation
poe docs-preview

  7. Test the Airbyte Replication MCP Server from source

    main

    The Airbyte MCP server is built using the FastMCP framework. To inspect the server definition and available tools, use the mcp-inspect task.

    To use the development version of the server in an MCP client configuration, set the command to uv and use the --directory flag to point to the PyAirbyte repository. Use run airbyte-mcp as the arguments.

    Warning: The MCP Server does not hot-reload. You must close and re-open your MCP client to incorporate code changes made during a session.

    # Setup and inspect
uv sync --group dev
uv run poe mcp-inspect
    {
  "mcpServers": {
    "airbyte": {
      "command": "uv",
      "args": [
        "--directory=/path/to/repos/PyAirbyte",
        "run",
        "airbyte-mcp"
      ],
      "env": {
        "AIRBYTE_MCP_ENV_FILE": "/path/to/my/.mcp/airbyte_mcp.env",
        "AIRBYTE_CLOUD_MCP_READONLY_MODE": "0",
        "AIRBYTE_CLOUD_MCP_SAFE_MODE": "0"
      }
    }
  }
}

  8. Install PyAirbyte and its connectors

    main

    PyAirbyte allows you to use Airbyte connectors directly in Python. Installation methods vary depending on the type of connector:

    Declarative Sources

    For sources defined in YAML, you can download the YAML file from connectors.airbyte.com and run them directly. These have the fastest download times.

    Python-based Sources

    PyAirbyte attempts to install Python-based connectors automatically.

    • Default Installer: Since version 0.29.0, PyAirbyte uses uv by default for faster installation and better Python version management.
    • Fallback to pip: If you need to use pip instead of uv, set the environment variable AIRBYTE_NO_UV=true.

    Docker-based Sources

    For any source or destination, you can force the use of Docker to ensure dependency stability. This is useful if you want to lock all dependencies within a container.

    • Set docker_image=True to use the default Docker installation.
    • Set docker_image='MY_IMAGE' to use a specific Docker image tag.

  9. Use PyAirbyte for connector development and testing

    main

    PyAirbyte can be used to develop and test Airbyte sources. This includes:

    • Testing existing published connectors.
    • Developing new local connectors.
    • Using any local installation that provides a CLI (PyAirbyte will automatically find connectors by name if they are available on your PATH).

  11. Generate Markdown documentation for the MCP Server

    main

    The repository includes a script to introspect the MCP server and generate a Markdown documentation site under docs/mcp-generated/. This output is compatible with both Docusaurus and pdoc.

    Running poe mcp-docs-md produces one Markdown file per MCP module:

    • index.md: Server overview
    • cloud.md: Tools in airbyte.mcp.cloud
    • local.md: Tools in airbyte.mcp.local
    • registry.md: Tools in airbyte.mcp.registry
    • prompts.md: Prompts in airbyte.mcp.prompts
    • misc.md: Other resources (e.g., server_info)

    Regenerate these files whenever you change MCP tool signatures, descriptions, or schemas.

    uv sync --group dev
poe mcp-docs-md

  12. Test Airbyte MCP tools using Poe tasks

    main

    The easiest way to test specific MCP tools during development is via the mcp-tool-test Poe task, passing the tool name and a JSON string of arguments.

    Available Server Transport Tasks:

    • poe mcp-serve-local: STDIO transport (default)
    • poe mcp-serve-http: HTTP transport on localhost:8000
    • poe mcp-serve-sse: Server-Sent Events transport on localhost:8000
    • poe mcp-inspect: Show all available MCP tools and their schemas
    # Test specific tools
poe mcp-tool-test list_connectors '{}'
poe mcp-tool-test get_config_spec '{"connector_name": "source-pokeapi"}'
poe mcp-tool-test run_sync '{"connector_name": "source-pokeapi", "config": {"pokemon_name": "pikachu"}}'

# Test cloud context tools
poe mcp-tool-test get_default_cloud_context '{}'
poe mcp-tool-test list_deployed_cloud_connections '{}'