DOCSEARCHbyAlgolia
Submit Docs

Apache Polaris Documentation

repository·main·Indexed 24 days ago

https://github.com/apache/polaris

Apache Polaris is an open-source, fully-featured catalog for Apache Iceberg that implements the Iceberg REST API to enable multi-engine interoperability. It provides a centralized governance layer supporting Iceberg, Delta Lake, Hudi, and Lance table formats, and works with engines such as Spark, Trino, Flink, Apache Doris, Dremio OSS, and StarRocks. The project includes a Python client, Kubernetes deployment via Helm, and support for OPA and Apache Ranger authorization.

Tokens
168K
Snippets
380
Records
700
Agent score
82%

What's inside Apache Polaris

  1. Overview of the Polaris Spark Plugin

    main

    The Polaris Spark plugin provides a SparkCatalog class that enables Apache Spark to communicate with Polaris REST endpoints. It implements Spark's TableCatalog, ViewCatalog, and SupportsNamespaces interfaces.

    Key Capabilities

    • Multi-format Support: The client manages both Iceberg and Delta tables. It automatically routes Iceberg requests to Iceberg REST endpoints and Delta requests to Generic Table REST endpoints.
    • Compatibility:
      • Spark 3.5: Supports Scala 2.12 and 2.13.
      • Spark 4.0: Supports Scala 2.13.
    • Delta Requirements:
      • For Spark 3.5: Requires delta >= 3.2.1.
      • For Spark 4.0: Requires delta >= 4.2.0.

  2. Overview of Apache Polaris features

    main

    Apache Polaris provides a centralized catalog and governance layer for Iceberg-compatible engines. Key capabilities include:

    • Iceberg REST Catalog: Implements the Iceberg REST specification.
    • Multi-format Support: Supports Iceberg, Delta Lake, Hudi, and Lance table formats.
    • Multi-engine Interoperability: Works with Spark, Trino, Flink, and other Iceberg-compatible engines.
    • Security & Governance: Centralized management of principals, roles, and fine-grained privileges.
    • Management: Centralized catalog, namespace, and table management.
    • Kubernetes-native: Supports horizontal scaling, Ingress, and Gateway API.
    • Observability: Includes production-ready health checks and metrics.

  3. Overview of the Polaris Runtime Service

    main
    The polaris-runtime-service module is the core runtime service implementation of Apache Polaris. It functions as the primary service layer, providing the REST API endpoints and the underlying business logic for the Polaris ecosystem.

  4. What is Apache Polaris?

    main

    Apache Polaris is a cloud-native, open-source catalog designed for Apache Iceberg. It serves as a centralized service to manage data across various engines and cloud environments, ensuring interoperability and security for Iceberg tables.

    Key features include:

    • Apache Iceberg REST Catalog Compliance: Provides a vendor-neutral implementation of the Apache Iceberg REST Catalog specification.
    • Cross-Engine Interoperability: Enables different engines (such as Dremio, Snowflake, Apache Spark, Trino, or Apache Flink) to access the same source of truth.
    • Fine-Grained Access Control: Offers centralized security policies that protect data regardless of the querying engine.
    • Multi-Cloud Support: Allows the catalog to run in environments where your data resides, helping to prevent vendor lock-in.

  5. Use Releasey scripts for automated releases

    main

    The releasey/ directory contains scripts designed to automate the Apache Polaris release process.

    ⚠️ Important Notice: These scripts are optimized for the GitHub Actions environment and are designed to work with specific workflow contexts and environment variables. They are not intended for manual execution by release managers.

  6. Use the Polaris Admin Tool

    main

    The Polaris Admin Tool is used to manage the Polaris metastore. It is available as a Docker image (apache/polaris-admin-tool) or as part of the binary distribution.

    Important: Ensure the Admin Tool version matches your Polaris server version. The tool must be configured to connect to the same database used by the Polaris server.

    Available commands:

    • help: Display help information.
    • bootstrap: Bootstrap realms and root principal credentials (idempotent).
    • purge: Remove realms and all associated entities (catalogs, namespaces, tables, views, roles, credentials, and grants).
    • nosql: Sub-commands specific to NoSQL persistence.

  7. Understand the core components of polaris-core

    main

    The polaris-core module provides the foundational building blocks for Apache Polaris. It is organized into four primary functional areas:

    • Entity Management: Defines the core entities used throughout the system via PolarisEntityType. Definitions are located in org.apache.polaris.core.entity.
    • Security & Access Control: Implements a Role-Based Access Control (RBAC) model. Privileges are defined by PolarisPrivilege, and the authorization logic is managed by PolarisAuthorizer (see org.apache.polaris.core.auth).
    • Persistence Logic: Provides a layer for entity storage through the BasePersistence interface. Core logic resides in org.apache.polaris.core.persistence.
    • Object Storage Integration: Offers an interface for interacting with various object storage providers via PolarisStorageIntegration (see org.apache.polaris.core.storage).

  8. Apache Polaris API Specifications Overview

    main

    Apache Polaris provides several OpenAPI specifications that define its management and cataloging capabilities. These specifications are divided into management services and catalog services.

    Management Service

    • polaris-management-service.yml: Defines APIs for managing principals, principal roles, catalogs, and catalog roles.

    Catalog Service

    • polaris-catalog-service.yaml: Defines the Apache Polaris Catalog API, which includes both the Apache Iceberg REST Catalog API and Polaris-native APIs.

    Catalog API Components

    • iceberg-rest-catalog-open-api.yaml: Specification for the Apache Iceberg REST Catalog API.
    • polaris-catalog-apis/ folder contains Polaris-specific APIs:
      • generic-tables-api.yaml: Generic Tables API.
      • notifications-api.yaml: Notifications API.
      • policy-apis.yaml: Policy APIs.
      • oauth-tokens-api.yaml: Internal OAuth Token endpoint (extracted from the Iceberg REST Catalog API).

  9. New features in Apache Polaris 1.0.0-incubating

    main

    The 1.0.0-incubating release introduces several major features:

    • Policy Store: Provides persistence with schema evolution, built-in TMS policies (e.g., Data compaction, Snapshot expiry), and REST CRUD endpoints.
    • Postgres JDBC persistence: Native JDBC backend supporting concurrent changes.
    • Enhanced Runtime: The Polaris server and admin tool now use Quarkus, providing Kubernetes readiness, quick startup, and OIDC integration.
    • Identity Provider (IdP) Support: Polaris can act as its own IdP, delegate to an external IdP, or both.
    • Catalog Federation (experimental): Ability to federate requests to external Iceberg REST or Hadoop Catalogs.
    • Generic Tables (experimental): Support for multiple table formats; the initial Spark 3.5 plugin supports Delta Lake.
    • Event Listener framework (experimental): Allows subscribing to catalog events such as AfterTableCommitedEvent and BeforeViewCommitedEvent.
    • HTTP caching via ETag: The loadTable endpoint supports ETag to reduce bandwidth and latency.
    • Rollback Compaction on Conflicts: Prioritizes writers by reverting compaction commits if conflicts occur.

  10. New features in Apache Polaris 1.5.0

    main

    Version 1.5.0 introduces several new capabilities:

    • Apache Ranger Support: Added support for Apache Ranger as an external authorizer (Beta).
    • Polaris CLI Enhancements:
      • Added summarize subcommand.
      • Added find and tables options.
    • Helm Chart: Added envFrom support.
    • Event Listeners: Support for multiple listeners and granular filtering by event type or category.

  11. Understand the Apache Polaris API Modules

    main

    Apache Polaris provides several API modules that are generated from OpenAPI specifications. These modules are categorized by the specific service or model they support:

    • Management API Models: Use polaris-api-management-model to access the model classes required for interacting with the Polaris Management API.
    • Management API Services: Use polaris-api-management-service to access the service classes for making calls to the Polaris Management API.
    • Iceberg REST API Services: Use polaris-api-iceberg-service to access the service classes for the Polaris Iceberg REST API.
    • Native Catalog REST API Services: Use polaris-api-catalog-service to access the service classes for the Polaris native Catalog REST API.

  12. Reference Polaris configuration categories

    main

    Polaris configuration is organized into several functional groups. All properties listed in the configuration reference are runtime properties, meaning they can be modified without rebuilding the Polaris application.

    Key configuration categories include:

    • Features & Behavior: Control feature flags and behavior changes via polaris.features and polaris.behavior-changes.
    • Authentication & Authorization: Manage security via polaris.authentication, polaris.authorization, polaris.oidc, and polaris.service-identity.
    • Storage & Credentials: Configure storage backends and credential management using polaris.storage, polaris.credential-manager, and polaris.secrets-manager.
    • Persistence: Manage data persistence, caching, and NoSQL/Relational database settings (e.g., polaris.persistence.nosql.mongodb, polaris.persistence.relational.jdbc).
    • Events: Configure event listeners and buffers (e.g., polaris.event-listener).
    • Operational: Manage runtime operations including tasks, rate limiting, metrics, logging, and readiness via polaris.tasks, polaris.rate-limiter, polaris.metrics, and polaris.log.