DOCSEARCHbyAlgolia
Submit Docs

Ocean Framework

website·Indexed 35 days ago

https://ocean.port.io/

A framework for building and deploying integrations for Port. Ocean provides tools for data synchronization via resync functions, real-time updates through live event processors, and user-initiated operations via integration actions. It includes a CLI for scaffolding, support for .port configuration files, and deployment options via Terraform, ArgoCD, and Helm.

Tokens
28.2K
Snippets
100
Records
176
Agent score
60%

What's inside Ocean

  1. Overview of the Release RC workflow

    The Release RC (Release Candidate) workflow is a GitHub Actions workflow used to automate the creation and publication of pre-release versions of the Ocean framework and its integrations. It handles publishing packages to PyPI, pushing Docker images to the GitHub Container Registry, creating GitHub pre-releases, and tagging the source branch.

  2. Overview of the Ocean extensibility framework

    Ocean is an open-source Python framework designed to simplify the process of connecting 3rd-party tools and infrastructure to Port. It provides a streamlined interface and abstractions for interacting with Port's REST API, allowing developers to focus on the logic for querying external systems rather than the complexities of API communication and data ingestion.

    Ocean consists of two primary components:

    1. Ocean CLI: Used for creating boilerplates for new integrations and aiding development.
    2. Ocean Framework: Provides common functionalities such as authentication, high-throughput ingestion, JQ mapping, and support for various connection and deployment modes.

  3. Overview of Ocean integration event listeners

    The Ocean framework uses event listeners to receive events and resync requests from Port and forward them to the running integration. Event listeners allow an integration to react to:

    1. Configuration updates: The integration uses new configuration data to perform a resync of information from the 3rd-party service.
    2. Resync requests: The integration performs a resync of data from the 3rd-party service to Port based on existing configuration.

    Caveats:

    • Current event listeners do not support multiple instances of the same integration.
    • If a resync event is received while a resync is already active, the current process is aborted and a new one starts. To avoid infinite recurring resyncs, ensure the interval (for polling) or scheduledResyncInterval provides a sufficient buffer (e.g., 2 hours) to allow a full resync to complete.

  4. Overview of Ocean Resource Mapping

    Resource Mapping is an ETL (Extract, Transform, Load) layer built into the Ocean framework. It allows Ocean integrations to parse and map data from 3rd-party services into standard Port entities. The framework uses JQ (JSON processor) to select, modify, and transform fields from API responses.

  5. Overview of Ocean Action Execution

    The Ocean framework enables 'backstream' capabilities, allowing Port users to trigger operations (actions) in third-party services (e.g., restarting resources, provisioning, or rotating credentials). The framework manages the orchestration, queuing, rate limiting, and error handling of these operations. The system consists of three main components: the Executor (integration-specific logic), the Execution Manager (orchestrator and worker distributor), and the Queue System (buffers for sequential or parallel execution).

  6. Overview of Port Ocean integrations

    An Ocean integration is a connector designed to extract data from external systems, transform that data to fit Port's entity model, and synchronize it with the Port catalog. The Ocean framework manages the underlying infrastructure, allowing developers to focus on the specific extraction and transformation logic for the target system.

  7. Understand the Live Event processing flow in Ocean

    When a service sends a live event via webhook to an integration, Ocean processes it through the following lifecycle:

    1. Event Reception: Ocean receives an HTTP POST, creates a WebhookEvent object containing the payload and headers, and adds it to a processing queue.
    2. Processor Selection: Ocean iterates through processors; each processor uses the should_process_event method to determine if it should handle the specific event.
    3. Event Processing: The selected processor authenticates and validates the event, fetches the updated data from the external service, and returns the results in a WebhookEventRawResults object.
    4. Port Update: Ocean applies JQ transformations to map the results to Port entities, performing the necessary updates or deletions.
    Service Event → Live Event Processor → Data Fetching → Port Update

  8. Understand the Ocean integration bootstrap process

    When an integration starts, the create_default_app() function initiates the bootstrap phase. This process involves loading the integration.py file from the integration directory, extracting any custom integration class defined within it, and creating an Ocean instance using the provided configuration.

  9. Access open-source Ocean integrations

    Port provides a catalog of open-source, Ocean-powered integrations that can be used as reference implementations or templates for building custom integrations. These integrations are hosted publicly on GitHub.
    # Explore the integrations repository on GitHub
https://github.com/port-labs/ocean/tree/main/integrations

  10. Understand the .port directory structure for Ocean integrations

    Ocean integrations use a .port directory to store resources required for startup and configuration validation. The directory is created automatically when a new integration is initialized.
    └── my_new_integration/
    ├── .port/
    │   ├── spec.yml
    │   └── resources/
    │       ├── port-app-config.yml
    │       └── blueprints.json
    └── ...

  11. Understand the Ocean framework architecture

    Ocean is an orchestration framework designed to synchronize data between third-party systems and Port. It manages a complete ETL (Extract, Transform, Load) pipeline consisting of five main stages:

    1. Listen: Event listeners monitor for resync requests.
    2. Extract: User-defined code fetches raw data from third-party APIs.
    3. Transform: JQ mappings convert the raw extracted data into Port entities.
    4. Load: The state applier synchronizes these entities with Port.
    5. Monitor: Metrics are tracked throughout the entire process.

    Key capabilities include event management, data extraction, transformation via JQ, state synchronization, and real-time processing of live events from third-party systems.

  12. Access event-specific data using the Event context

    The event variable provides metadata about the current execution of the integration code. It is only accessible when the code is triggered by an event (such as a resync or an HTTP request).
    from port_ocean.context.event import event

# Event metadata
event_type = event.type            # e.g., 'resync', 'start', 'http_request'
event_trigger = event.triggerType    # e.g., 'manual', 'machine', 'request'

# Resync-specific data
resync_config = event.resource_config
app_config = event.port_app_config

# Shared data/cache
# Use event.attributes to pass data between integration parts or cache results
event.attributes['my_cache_key'] = 'some_value'